Optimizing the Geometry Weight for a 2.76 A structure
Hi, I'm refining a 2.76 A structure and need help optimizing the refinement weight on geometry. When I refine at the default refinement weight (wxc = 0.5), phenix seems to generate a much poorer structure (both geometrically AND by Rfree value) relative to lower wxc values which put a stronger weight on geometry. I am trying to pick the best structure to use for my next round of rebuilding. The R-stats and condensed molprobity stats are listed below (at the end of this email). I have a few questions regarding this structure and those stats: 1. What is the relevance of having a LOW difference between Rfree and Rwork? My structure seems to have a higher than expected Rwork value relative to Rfree (aren't they supposed to differ by 0.05 or so, instead of the 0.03 observed in my structure?) Additionally, as I refine at lower wxc values (more weight on geometry) I get an increased Rwork value with an equal or decreased Rfree (relative to the refinement at a higher wxc value). Does that mean my structure is better or worse? 2. How can I judge the output from my refinement? I have looked at Rwork, Rfree, and the molprobity clashscore and overall score values. I included them below, at the end of this email. How do I tell which the best refinement is? Which one would you suggest? I thought the best was wxc = 0.1 since the R-work and Rfree aren't changed much from the start values but the geometry is far better. 3. How do I optimize the weight on geometry in refinement? My PI said that phenix can do it but not for twinned structures (my structure is twinned). Can the new version of phenix do weight optimization for twinned structures? How? If not, what's the best way for me to do it manually? As always, thanks immensely for all your help! -Sam All have the same values for: Start R-work = 0.2002, R-free = 0.2376 *wxc = 0.01 *Final R-work = 0.2104, R-free = 0.2378 Molprobity Clashscore: 19.46 Molprobity Score: 2.29 *wxc = 0.05 *Final R-work = 0.2049, R-free = 0.2377 Molprobity Clashscore: 20.96 Molprobity Score: 2.38 *wxc = 0.1 *Final R-work = 0.2010, R-free = 0.2388 Molprobity Clashscore: 23.50 Molprobity Score: 2.54 *wxc = 0.2 *Final R-work = 0.1969, R-free = 0.2399 Molprobity Clashscore: 27.15 Molprobity Score: 2.79 *wxc = 0.3 *Final R-work = 0.1945, R-free = 0.2411 Molprobity Clashscore: 30.76 Molprobity Score: 2.97 *wxc = 0.4 *Final R-work = 0.1930, R-free = 0.2420 Molprobity Clashscore: 34.54 Molprobity Score: 3.11 *wxc = default (0.5) *Final R-work = 0.1918, R-free = 0.2429 Molprobity Clashscore: 37.73 Molprobity Score: 3.22
On Thu, Apr 1, 2010 at 12:57 PM, Sam Stampfer
1. What is the relevance of having a LOW difference between Rfree and Rwork? My structure seems to have a higher than expected Rwork value relative to Rfree (aren't they supposed to differ by 0.05 or so, instead of the 0.03 observed in my structure?) Additionally, as I refine at lower wxc values (more weight on geometry) I get an increased Rwork value with an equal or decreased Rfree (relative to the refinement at a higher wxc value). Does that mean my structure is better or worse?
When the spread between R-work and R-free widens, this means that you've optimized R-work at the expense of R-free, i.e. you're "overfitting". If they're too close together this can indicate that you've somehow biased your refinement (by improper assignment of R-free flags in the presence of NCS, accidentally switching R-free sets in the middle of refinement, or using a very similar MR search model refined against a different set are the usual reasons). However, 0.03 is (usually) not a small enough difference to worry about - on the contrary, it's very good. Therefore, if a refinement raises R-work and lowers R-free, this is almost always a good thing. Anything that raises R-free relative to the starting value is bad.
2. How can I judge the output from my refinement? I have looked at Rwork, Rfree, and the molprobity clashscore and overall score values. I included them below, at the end of this email. How do I tell which the best refinement is? Which one would you suggest? I thought the best was wxc = 0.1 since the R-work and Rfree aren't changed much from the start values but the geometry is far better.
The first one in the list, with wxc=0.01, R=0.2104, R-free=0.2378 is definitely the best, because all of the statistics that matter are much better than in any other refinement. PS. Use POLYGON in the GUI to get a better idea of how good these statistics are relative to other structures. -Nat
Hi Sam,
When the spread between R-work and R-free widens, this means that you've optimized R-work at the expense of R-free, i.e. you're "overfitting". If they're too close together this can indicate that you've somehow biased your refinement (by improper assignment of R-free flags in the presence of NCS, accidentally switching R-free sets in the middle of refinement, or using a very similar MR search model refined against a different set are the usual reasons).
I would also add to the above list: "improper assignment of R-free flags in the presence of twinning".
Therefore, if a refinement raises R-work and lowers R-free, this is almost always a good thing.
I think it is important "by how much" it lowers or/and rises. Say you have a 2.5A resolution data: I would probably still prefer Rwork/Rfree ~ 20/26% over 23/25%, since the 20/26% result would more likely to give a better map at the cost of "insignificant" Rfree fluctuation.
Anything that raises R-free relative to the starting value is bad.
A possible exception is when you run refinement for the first time (not in your life, but given a model and data). Then Rwork ~ Rfree at the start, and they diverge as refinement progresses (may both go down, one faster than the other, or Rwork may drop and Rfree may increase). Again, what's important here is "by how much" they drop or rise.
2. How can I judge the output from my refinement? I have looked at Rwork, Rfree, and the molprobity clashscore and overall score values. I included them below, at the end of this email. How do I tell which the best refinement is? Which one would you suggest? I thought the best was wxc = 0.1 since the R-work and Rfree aren't changed much from the start values but the geometry is far better.
The first one in the list, with wxc=0.01, R=0.2104, R-free=0.2378 is definitely the best, because all of the statistics that matter are much better than in any other refinement.
PS. Use POLYGON in the GUI to get a better idea of how good these statistics are relative to other structures.
Additionally, look at the local model-to-density fit quality: map correlation reported for per residue (for lower resolutions) or per atom (at higher resolution). It is available in PHENIX GUI and from the command line. Also, the values 0.2377, 0.2388, 0.2399 ... look all the same to me. If you run 100 identical refinement runs where in each refinement the only difference is the random seed, you will get an ensemble of refined structures and the Rfree/Rwork spread can be as large as 1-2% or so (it depends on the resolution). This is because the random seed is involved in target weights calculation and therefore a small change in the random seed may slightly change the weights and this may be enough for the refinement to take another route to another "local minimum". Pavel.
More pertinent to this example is the s.d. of the R-free itself, i.e. sigma(Rfree)/Rfree ~ 1/sqrt(Ntest) lifted from Kleywegt and Brunger, Structure, 15 August 1996, 4:897–904 (but the original analysis from an earlier paper) So for 1000 free reflections and an R-free of 24% the sigma is about 0.75% and exceeds the range of variation that you're seeing, i.e. not a significant fluctuation in R-free. Phil Jeffrey Princeton Pavel Afonine wrote: [snip]
Also, the values 0.2377, 0.2388, 0.2399 ... look all the same to me. If you run 100 identical refinement runs where in each refinement the only difference is the random seed, you will get an ensemble of refined structures and the Rfree/Rwork spread can be as large as 1-2% or so (it depends on the resolution). This is because the random seed is involved in target weights calculation and therefore a small change in the random seed may slightly change the weights and this may be enough for the refinement to take another route to another "local minimum".
That uncertainty is not quite the same thing. What you describe is the uncertainty in the free R due to the small sampling size of the test set. It is the spread of free R's obtained when different test sets are chosen for the same project and circumstances, and is useful when comparing free R's calculated using different test sets, or, heaven forbid, trying to compare free R's from different crystals. In this case the question is "How much confidence do I have that a drop in free R (estimate) from a single test set of X% indicates that the average free R calculated over all test sets actually dropped?" It is possible that a free R estimate that happens by chance to be 0.2% lower than the "true" free R will always be 0.2% lower and its behavior is a reliable predictor. It is also possible that the difference between the free R estimate from a particular test set that was 0.2% lower than the true value becomes 0.2% higher after refinement. In that case the free R estimate would be have much worst reliability as a predictor of true free R improvement than it has of predicting the true free R itself. I don't know that anyone has done the tests to determine the actual behavior. My personal bias is that there probably is some tendency for "reversion to the mean" where a free R estimate that happens to be too good will tend to become less "too good" with refinement and one that is too bad will tend to become less "too bad". If this is true the differences in free R estimate will have higher reliability in predicting changes in the true free R than it has in predicting the value of the true free R itself. If you do believe that the free R estimate is "fuzzy" and we see that the minimum of the free R estimate vrs weight is rather broad, how do you decide the optimal weight? Let's go to the extreme and say you have a function that looks like a square well. Do you choose the weight at the center of the well, assuming that is the best compromise? Or do you choose the the side of the well with the lowest working R and the cleanest difference map? Or do you choose the side with the tightest geometry and the least overfitting? If the improvement in free R estimate is not sufficiently precise to determine the weight, I would like a little more guidance in how the weight should be determined. Dale Tronrud Phil Jeffrey wrote:
More pertinent to this example is the s.d. of the R-free itself, i.e.
sigma(Rfree)/Rfree ~ 1/sqrt(Ntest)
lifted from Kleywegt and Brunger, Structure, 15 August 1996, 4:897–904 (but the original analysis from an earlier paper)
So for 1000 free reflections and an R-free of 24% the sigma is about 0.75% and exceeds the range of variation that you're seeing, i.e. not a significant fluctuation in R-free.
Phil Jeffrey Princeton
Pavel Afonine wrote: [snip]
Also, the values 0.2377, 0.2388, 0.2399 ... look all the same to me. If you run 100 identical refinement runs where in each refinement the only difference is the random seed, you will get an ensemble of refined structures and the Rfree/Rwork spread can be as large as 1-2% or so (it depends on the resolution). This is because the random seed is involved in target weights calculation and therefore a small change in the random seed may slightly change the weights and this may be enough for the refinement to take another route to another "local minimum".
_______________________________________________ phenixbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/phenixbb
Phil Jeffrey wrote:
More pertinent to this example is the s.d. of the R-free itself, i.e.
sigma(Rfree)/Rfree ~ 1/sqrt(Ntest)
I believe "~" here is read "proportional to", not "approximately". Ed PS- Rfree can be seen* as an average of (|Fo-Fc|)/Fo weighted by Fo, so variance should be proportional to sqr(N) like the standard error of the mean. But SEM depends also on the sigma for the individual observations, which I would hope is less than 100%, so sigma(Rfree)/Rfree < 1/sqrt(Ntest)? *derivation: unweighted average: Sum(|Fo-Fc|/Fo)/N = Sum(|Fo-Fc|/Fo)/Sum(1) weighted by Fo: Sum(Fo*|Fo-Fc|/Fo)/Sum(Fo) = Sum(|Fo-Fc|)/Sum(Fo) = R
Edward A. Berry wrote:
sigma(Rfree)/Rfree ~ 1/sqrt(Ntest)
I believe "~" here is read "proportional to", not "approximately".
No, seems it should be "approximately": "Interestingly, the standard deviation of the free R value is approximately given by Rfree/(n)^1/2, where n is the number of reflections in the test set." Free R Value: Cross-Validation in Crystallography - AXEL T. BRUNGER METHODS IN ENZYMOLOGY, VOL. 277
Hi Sam,
3. How do I optimize the weight on geometry in refinement? My PI said that phenix can do it but not for twinned structures (my structure is twinned). Can the new version of phenix do weight optimization for twinned structures? How? If not, what's the best way for me to do it manually?
you can do it manually (by trying different wxc_scale and wxu_scale) or let phenix.refine do it automatically (using optimize_wxc=true and/or optimize_wxu=true). It has nothing to do with twinning. Pavel.
participants (6)
-
det102＠uoxray.uoregon.edu -
Edward A. Berry -
Nathaniel Echols -
Pavel Afonine -
Phil Jeffrey -
Sam Stampfer