That's not the point, at least not the way I read this. Weak reflections with Intensity < 0 get set to |F|=0 by CCP4's Truncate with the TRUNCATE NO set (in script files, it's called something else in CCP4i). That's not the same as a Missing Reflection. Ergo the fact that F=0 means that it is weak, but with some even larger uncertainty than it was measured with, since you don't know how negative I/sigI was. Nevertheless this does contain information as to the ballpark of reflection magnitude. A non-default option to include this data would be desirable (reprocessing it with TRUNCATE YES being even more desirable). Phil Jeffrey Princeton Pavel Afonine wrote:

Hi Joe,

...

However, it appears that PHENIX still throws out many reflections where Fobs==0, which can be a significant fraction in the last shell with anisotropic data. Unfortunately, the exclusion of weak reflections depends on how amplitudes were derived. If using CCP4 Truncate, those weak reflections will be inflated a bit to a non-zero value, and a zero-sigma cutoff will have a significantly different affect. Therefore, I think that the default should be to use reflections with Fobs==0, with SigFobs > 0 as the criterion for non-absent reflections in reflection files without a missing number flag (i.e. CNS format).

When it comes down to refinement, it's impractical to hope to find out where the data come from. Sorry for my ignorance in this question.... But could you please tell us what are the exact benefits of using Fobs=0 in refinement, preferable supported by references where it was systematically studied. I'm sure I'm missing or forgot something, but fail to get it right now at this time of the evening...

Imagine I have a dataset of resolution 26.0-2.3A. Do you really think it would be great to do refinement in resolution say 100.0-0.25A, where all missing Fobs are zeros?

Thanks! Pavel.

_______________________________________________ phenixbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/phenixbb

Hi Ed,

AFAIU, Joe is referring to the situation where negative intensities are converted to FOBS=0.

thanks for making this statement clear. I see now what Joe meant.

Ignoring them would allow refinement to have any FCALC for these reflections when in fact we do have some information about them. I am not aware of any systematic study of this question, but it *seems* rather obvious that ignoring zero reflections is a wrong thing to do *in these circumstances*.

Reference, please -:) "Rather obvious" doesn't count for me -:) Thanks! Pavel.

Hi Ed,

You are essentially asking for proof that all the experimental data must be included while fitting theoretical model to it.

No, of course. I was just wondering how better (and what is "better" in this case) the refinement results will be if one includes the negative (zero) reflections into the process of refinement, and did anyone clearly demonstrate it. That's all I asked. Again, I hardly believe that you will notice the difference in refinement outcome (refiened model) if phenix.refine removes a few zero reflections that are in your PDB file. But I might be wrong and so wrote my previous question. Pavel.

HI Sacha, thanks, it may -:) I tried to read it 6 years ago, so I might need to refresh my memory. And again, it's not too relevant to my questions. All the best! Pavel. On 12/4/09 11:06 PM, Alexandre OURJOUMTSEV wrote:

Hi, Ed and Pavel,

...

No, of course. I was just wondering how better (and what is "better" in this case) the refinement results will be if one includes the negative (zero) reflections into the process of refinement, and did anyone clearly demonstrate it.

Can the work below help your discussion?

Best regards,

Sacha

=============

/Acta Cryst./ (1978). A*34*, 517-525[ doi:10.1107/S0567739478001114 http://dx.doi.org/10.1107/S0567739478001114 ]

On the treatment of negative intensity observations

S. French http://scripts.iucr.org/cgi-bin/citedin?search_on=name&author_name=French,%20S. and K. Wilson http://scripts.iucr.org/cgi-bin/citedin?search_on=name&author_name=Wilson,%20K.

------------------------------------------------------------------------

_______________________________________________ phenixbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/phenixbb

Pavel, Here is an example of the SCALEPACK log-file. My reading of it is that in the highest resolution shell I have 25% of reflections with I/sigma less than zero, thus negative intensities. Again, it doesn't bother *me*, because I use French&Wilson to convert the negative intensities to positive amplitudes and would like to reiterate that neither phenix.refine nor any other program can be blamed for what is essentially user error in assigning these reflections zero amplitude. However, Joe's point, imho, was that *if* they are assigned zero amplitudes significant portion of the data will be excluded if FOBS>0 is required. They are not a few, in the dataset that I pulled this from they are apparently >12000 out of about 100000 overall. Shell I/Sigma in resolution shells: Lower Upper % of of reflections with I / Sigma less than limit limit 0 1 2 3 5 10 20 >20 total 50.00 4.52 0.6 1.8 3.4 5.3 8.8 17.4 33.4 66.3 99.8 4.52 3.59 1.0 2.7 5.6 8.6 14.7 28.1 50.4 49.5 99.9 3.59 3.14 2.8 7.2 13.1 19.5 30.2 48.7 70.8 29.2 99.9 3.14 2.85 6.8 16.4 27.9 37.0 50.4 68.9 86.1 13.8 99.9 2.85 2.65 11.5 26.2 41.4 52.0 65.5 81.5 94.0 5.8 99.8 2.65 2.49 15.4 35.0 52.4 63.4 75.3 89.0 97.0 2.7 99.7 2.49 2.37 17.9 41.1 59.0 69.9 80.8 91.8 97.9 1.6 99.6 2.37 2.26 21.1 46.6 64.9 75.0 84.9 93.6 98.5 1.0 99.5 2.26 2.18 22.6 50.5 70.0 79.8 88.7 96.1 98.9 0.3 99.2 2.18 2.10 25.1 56.1 76.0 84.9 92.4 97.8 99.1 0.1 99.3 All hkl 12.4 28.3 41.2 49.4 59.0 71.1 82.4 17.2 99.7

I was just wondering how better (and what is "better" in this case) the refinement results will be if one includes the negative (zero) reflections into the process of refinement, and did anyone clearly demonstrate it. That's all I asked.

This is indeed the key question - how do you determine if your model is getting "better"? R-factors are no good here, as they will definitely improve upon rejection of weak reflections. I guess what most people would accept is the following: 1. Generate synthetic data (e.g. from molecular dynamics trajectories) and use MLFSOM and your favorite data reduction software to obtain a dataset. 2. Convert negative intensities to zero amplitudes. 3. Refine the model with and without FOBS=0 cutoff. 4. Determine the additional error introduced by the data rejection. My suspicion is that the result will be more complex than a simple yes/no answer. It's possible that rejection of negative intensities plays significant role only in certain resolution domain, but at least the effect should be resolution-dependent (as well as vary from dataset to dataset). But one thing we can be certain about. Except for the unlikely but real scenario you mentioned when data well beyond I/s=1 resolution cutoff is somehow included, it is my expectation that by including the weak reflections the model quality will *not* be decreased. If I am right (and that is an if), then removing the FOBS=0 cutoff does no harm and has the potential to improve models at least on some occasions. Cheers, Ed. -- Edwin Pozharski, PhD, Assistant Professor University of Maryland, Baltimore ---------------------------------------------- When the Way is forgotten duty and justice appear; Then knowledge and wisdom are born along with hypocrisy. When harmonious relationships dissolve then respect and devotion arise; When a nation falls to chaos then loyalty and patriotism are born. ------------------------------ / Lao Tse /

(I got busy and did not follow up earlier.) I think the value of weak reflections should be very obvious, if you think about it correctly, and is why I/sigma cutoffs have been discouraged for many years now. The significance of a reflection is not its overall amplitude, but how far it deviates from the expected value, which is approximately the average value for a resolution bin. Weak reflections are far more useful than average-intensity reflections. If you consider the 2D vector space of a complex number, they are very well defined because the phase is not as important. People get confused about I/sigma significance for two reasons. First, weak reflections have no effect on maps, and many rules-of-thumb were developed from heavy-atom methods. Refinement is different. Second, I/sigma is a useful validity measure for a set (resolution bin) of reflections, because it indicates the significance of the expectation value. If the I/sigma for a shell is small, than all reflections are approximately the same as the expectation value. In my experience, anisotropic data can be poorly behaved when many weak reflections are missing in the low-resolution directions, because there is nothing to prevent the model from refining to non-zero values there. My point is that no matter what the argument for the value of culling reflections, or any other sort of weighting scheme, it should never be applied to the "true" R-free value. In practice, some culling is sensible when scaling, but it should be restricted to rejections based on multiple observations of the same reflection, and not systematic culling such as I/sigma cutoffs. That is why HKL sets the default sigma cutoff to -3. Joe Krahn Randy Read wrote:

Pavel wanted some evidence of whether or not it makes a difference to omit very weak reflections. Here's one relevant paper. Hirshfeld and Rabinovich (Acta Cryst. A29: 510-513, 1973) showed in a numerical experiment that, if you omit weak intensities, there is a systematic error in refined scale and ADP parameters. They used least squares so it's possible that the results would be somewhat different with maximum likelihood targets, but at least here is an objective demonstration that the weak data can have a significant influence.

Regards,

Randy ...