phenix.refine using anomalous concepts for neutron data?
Hi, I'm doing a joint X-ray neutron refinement in Phenix and have run into some behaviour that puzzles me. I had not been paying any attention to the use or non-use of anomalous signal in the neutron dataset, as This is what I get if I let phenix.refine itself decide whether or not to use anomalous data: ================================= Neutron data ================================ F-obs: neutron.mtz:FO,SIGFO,DANO,SIGDANO,ISYM Miller array info: neutron.mtz:FO,SIGFO,DANO,SIGDANO,ISYM Observation type: xray.reconstructed_amplitude Type of data: double, size=16672 Type of sigmas: double, size=16672 Number of Miller indices: 16672 Anomalous flag: True Unit cell: (removed) Space group: P 21 21 21 (No. 19) Systematic absences: 0 Centric reflections: 1258 Resolution range: 29.22 1.89826 Completeness in resolution range: 0.785008 Completeness with d_max=infinity: 0.78486 Bijvoet pairs: 6832 Lone Bijvoet mates: 1750 Anomalous signal: 0.0876 Number of F-obs in resolution range: 16672 Number of F-obs<0 (these reflections will be rejected): 0 Number of F-obs=0 (these reflections will be used in refinement): 0 Refinement resolution range: d_max = 29.2200 d_min = 1.8983 and this is what I get if I forcibly switch off use of anomalous data in the phenix.refine GUI: ================================= Neutron data ================================ F-obs: neutron.mtz:FO,SIGFO,DANO,SIGDANO,ISYM Miller array info: neutron.mtz:FO,SIGFO,DANO,SIGDANO,ISYM Observation type: xray.reconstructed_amplitude Type of data: double, size=16672 Type of sigmas: double, size=16672 Number of Miller indices: 16672 Anomalous flag: True Unit cell: (removed) Space group: P 21 21 21 (No. 19) Systematic absences: 0 Centric reflections: 1258 Resolution range: 29.22 1.89826 Completeness in resolution range: 0.785008 Completeness with d_max=infinity: 0.78486 Bijvoet pairs: 6832 Lone Bijvoet mates: 1750 Anomalous signal: 0.0876 force_anomalous_flag_to_be_equal_to=False Reducing data to non-anomalous array. R-linear = sum(abs(data - mean(data))) / sum(abs(data)) R-square = sum((data - mean(data))**2) / sum(data**2) In these sums single measurements are excluded. Redundancy Mean Mean Min Max Mean R-linear R-square unused: - 29.2234 bin 1: 29.2234 - 4.0863 1 2 1.699 0.0240 0.0011 bin 2: 4.0863 - 3.2448 1 2 1.793 0.0302 0.0022 bin 3: 3.2448 - 2.8350 1 2 1.800 0.0417 0.0041 bin 4: 2.8350 - 2.5760 1 2 1.764 0.0486 0.0054 bin 5: 2.5760 - 2.3914 1 2 1.745 0.0505 0.0051 bin 6: 2.3914 - 2.2505 1 2 1.717 0.0513 0.0051 bin 7: 2.2505 - 2.1378 1 2 1.658 0.0533 0.0056 bin 8: 2.1378 - 2.0448 1 2 1.614 0.0611 0.0074 bin 9: 2.0448 - 1.9661 1 2 1.575 0.0634 0.0075 bin 10: 1.9661 - 1.8983 1 2 1.485 0.0736 0.0097 unused: 1.8983 - Fobs statistics after all cutoffs applied: Miller array info: None Observation type: xray.amplitude Type of data: double, size=9840 Type of sigmas: double, size=9840 Number of Miller indices: 9840 Anomalous flag: False Unit cell: (removed) Space group: P 21 21 21 (No. 19) Systematic absences: 0 Centric reflections: 1258 Resolution range: 29.22 1.89826 Completeness in resolution range: 0.855801 Completeness with d_max=infinity: 0.855503 What's going on here? If I let phenix.refine decide automatically, it looks like it's reading in F+ and F- separately and regarding them as independent in the refinement. If I force "no anomalous" it merges them. However I can't work out whether in either case it has actually read FOBS instead of the Friedel mates. According to the SCALA manual the I+ and I- columns will always be written out even if the keyword ANOMALOUS OFF is used, so this is a potential pitfall for many. In addition, phenix.refine seems to identify the neutron data first as xray.reconstructed_amplitude then as xray.amplitude even though the data are explicitly defined as neutron data in the GUI. I can send more information if required! Thanks Derek ________________________________________________________________________ Derek Logan tel: +46 46 222 1443 Associate Professor mob: +46 76 8585 707 Dept. of Biochemistry and Structural Biology www.cmps.lu.sehttp://www.cmps.lu.se Centre for Molecular Protein Science www.maxlab.lu.se/node/307http://www.maxlab.lu.se/node/307 Lund University, Box 124, 221 00 Lund, Sweden www.saromics.comhttp://www.saromics.com
Hi Derek, did you really collected anomalous neutron data? I doubt. Then how you ended up having anomalous data set for your neutron observations: FO,SIGFO,DANO,SIGDANO,ISYM ? By default phenix.refine will always take anomalous data set if there is a choice anomalous versus non-anomalous. This is because anomalous data is a) richer source of information, and b) less manipulated compared to non-anomalous Fmean=(F+ + F-)/2 or Imean=(I+ + I-)/2. In refinement Bijvoet mates F+ and F- are counted as two reflections, not one. So don't get surprised to see "doubled" number of reflections. It's not really doubled, it's what you actually have. All in all, if you don't have anomalous data make sure your inputs files do not have spurious (anomalous) arrays: phenix.refine can't read your mind to see what you have done in your diffraction experiment - it's beyond the scope of what it's supposed to do! All the best, Pavel On 5/29/13 4:18 AM, Derek Logan wrote:
Hi,
I'm doing a joint X-ray neutron refinement in Phenix and have run into some behaviour that puzzles me. I had not been paying any attention to the use or non-use of anomalous signal in the neutron dataset, as
This is what I get if I let phenix.refine itself decide whether or not to use anomalous data:
================================= Neutron data ================================
F-obs: neutron.mtz:FO,SIGFO,DANO,SIGDANO,ISYM
Miller array info: neutron.mtz:FO,SIGFO,DANO,SIGDANO,ISYM *Observation type: xray.reconstructed_amplitude* Type of data: double, size=16672 Type of sigmas: double, size=16672 Number of Miller indices: 16672 Anomalous flag: True Unit cell: (removed) Space group: P 21 21 21 (No. 19) Systematic absences: 0 Centric reflections: 1258 Resolution range: 29.22 1.89826 Completeness in resolution range: 0.785008 Completeness with d_max=infinity: 0.78486 Bijvoet pairs: 6832 Lone Bijvoet mates: 1750 Anomalous signal: 0.0876
Number of F-obs in resolution range: 16672 Number of F-obs<0 (these reflections will be rejected): 0 Number of F-obs=0 (these reflections will be used in refinement): 0 Refinement resolution range: d_max = 29.2200 d_min = 1.8983
and this is what I get if I forcibly switch off use of anomalous data in the phenix.refine GUI:
================================= Neutron data ================================
F-obs: neutron.mtz:FO,SIGFO,DANO,SIGDANO,ISYM
Miller array info: neutron.mtz:FO,SIGFO,DANO,SIGDANO,ISYM Observation type: xray.reconstructed_amplitude Type of data: double, size=16672 Type of sigmas: double, size=16672 Number of Miller indices: 16672 Anomalous flag: True Unit cell: (removed) Space group: P 21 21 21 (No. 19) Systematic absences: 0 Centric reflections: 1258 Resolution range: 29.22 1.89826 Completeness in resolution range: 0.785008 Completeness with d_max=infinity: 0.78486 Bijvoet pairs: 6832 Lone Bijvoet mates: 1750 Anomalous signal: 0.0876
force_anomalous_flag_to_be_equal_to=False Reducing data to non-anomalous array. R-linear = sum(abs(data - mean(data))) / sum(abs(data)) R-square = sum((data - mean(data))**2) / sum(data**2) In these sums single measurements are excluded. Redundancy Mean Mean Min Max Mean R-linear R-square unused: - 29.2234 bin 1: 29.2234 - 4.0863 1 2 1.699 0.0240 0.0011 bin 2: 4.0863 - 3.2448 1 2 1.793 0.0302 0.0022 bin 3: 3.2448 - 2.8350 1 2 1.800 0.0417 0.0041 bin 4: 2.8350 - 2.5760 1 2 1.764 0.0486 0.0054 bin 5: 2.5760 - 2.3914 1 2 1.745 0.0505 0.0051 bin 6: 2.3914 - 2.2505 1 2 1.717 0.0513 0.0051 bin 7: 2.2505 - 2.1378 1 2 1.658 0.0533 0.0056 bin 8: 2.1378 - 2.0448 1 2 1.614 0.0611 0.0074 bin 9: 2.0448 - 1.9661 1 2 1.575 0.0634 0.0075 bin 10: 1.9661 - 1.8983 1 2 1.485 0.0736 0.0097 unused: 1.8983 -
Fobs statistics after all cutoffs applied:
Miller array info: None *Observation type: xray.amplitude* Type of data: double, size=9840 Type of sigmas: double, size=9840 Number of Miller indices: 9840 Anomalous flag: False Unit cell: (removed) Space group: P 21 21 21 (No. 19) Systematic absences: 0 Centric reflections: 1258 Resolution range: 29.22 1.89826 Completeness in resolution range: 0.855801 Completeness with d_max=infinity: 0.855503
What's going on here? If I let phenix.refine decide automatically, it looks like it's reading in F+ and F- separately and regarding them as independent in the refinement. If I force "no anomalous" it merges them. However I can't work out whether in either case it has actually read FOBS instead of the Friedel mates. According to the SCALA manual the I+ and I- columns will always be written out even if the keyword ANOMALOUS OFF is used, so this is a potential pitfall for many. In addition, phenix.refine seems to identify the neutron data first as xray.reconstructed_amplitude then as xray.amplitude even though the data are explicitly defined as neutron data in the GUI.
I can send more information if required!
Thanks Derek
________________________________________________________________________ Derek Logan tel: +46 46 222 1443 Associate Professor mob: +46 76 8585 707 Dept. of Biochemistry and Structural Biology www.cmps.lu.se http://www.cmps.lu.se Centre for Molecular Protein Science www.maxlab.lu.se/node/307 http://www.maxlab.lu.se/node/307 Lund University, Box 124, 221 00 Lund, Sweden www.saromics.com http://www.saromics.com
_______________________________________________ phenixbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/phenixbb
Hi Pavel, did you really collected anomalous neutron data? I doubt. No, I'm unfortunately not yet developing methods for phasing neutron structures :-) Maybe I should actually have finished the second sentence in my mail, as it looked like I was implying that neutron anomalous signal might be something worth caring about, when what I really meant was "I wonder why phenix.refine is even considering this option". I was sitting in a conference at the time... Then how you ended up having anomalous data set for your neutron observations: FO,SIGFO,DANO,SIGDANO,ISYM ? As I mentioned in my original mail, I've discovered that SCALA always outputs I+ and I- no matter what you do. From the manual: "Normally anomalous scattering is ignored during the scale determination (I+ & I- observations are treated together), but the merged file always contains I+ & I-, even if the ANOMALOUS OFF command is used." Thus if you don't take special precautions, e.g. in TRUNCATE, these will be converted to F+ and F- and (I now know) phenix.refine will use them. However I've now discovered that the keyword ANOMALOUS NO in TRUNCATE will prevent output of the anomalous columns altogether. All in all, if you don't have anomalous data make sure your inputs files do not have spurious (anomalous) arrays: phenix.refine can't read your mind to see what you have done in your diffraction experiment - it's beyond the scope of what it's supposed to do! Fine, I've now learned an important lesson in input/output checking! But could I politely suggest that an improvement in phenix.refine's mind-reading capabilities: if the user mistakenly inputs a file as neutron data with anomalous columns in, then these should be ignored, as the concept is by default incompatible with neutron data? As I've discovered, the default behaviour of SCALA and TRUNCATE is to leave these columns in... Finally, it would also be nice if the neutron maps and not just the X-ray maps were displayed in Coot during refinement and if the summary panel at the end of refinement showed statistics for the neutron data as well as for the X-ray data. But having said that, thanks for a powerful and user-friendly program! Best wishes Derek On 5/29/13 4:18 AM, Derek Logan wrote: Hi, I'm doing a joint X-ray neutron refinement in Phenix and have run into some behaviour that puzzles me. I had not been paying any attention to the use or non-use of anomalous signal in the neutron dataset, as This is what I get if I let phenix.refine itself decide whether or not to use anomalous data: ================================= Neutron data ================================ F-obs: neutron.mtz:FO,SIGFO,DANO,SIGDANO,ISYM Miller array info: neutron.mtz:FO,SIGFO,DANO,SIGDANO,ISYM Observation type: xray.reconstructed_amplitude Type of data: double, size=16672 Type of sigmas: double, size=16672 Number of Miller indices: 16672 Anomalous flag: True Unit cell: (removed) Space group: P 21 21 21 (No. 19) Systematic absences: 0 Centric reflections: 1258 Resolution range: 29.22 1.89826 Completeness in resolution range: 0.785008 Completeness with d_max=infinity: 0.78486 Bijvoet pairs: 6832 Lone Bijvoet mates: 1750 Anomalous signal: 0.0876 Number of F-obs in resolution range: 16672 Number of F-obs<0 (these reflections will be rejected): 0 Number of F-obs=0 (these reflections will be used in refinement): 0 Refinement resolution range: d_max = 29.2200 d_min = 1.8983 and this is what I get if I forcibly switch off use of anomalous data in the phenix.refine GUI: ================================= Neutron data ================================ F-obs: neutron.mtz:FO,SIGFO,DANO,SIGDANO,ISYM Miller array info: neutron.mtz:FO,SIGFO,DANO,SIGDANO,ISYM Observation type: xray.reconstructed_amplitude Type of data: double, size=16672 Type of sigmas: double, size=16672 Number of Miller indices: 16672 Anomalous flag: True Unit cell: (removed) Space group: P 21 21 21 (No. 19) Systematic absences: 0 Centric reflections: 1258 Resolution range: 29.22 1.89826 Completeness in resolution range: 0.785008 Completeness with d_max=infinity: 0.78486 Bijvoet pairs: 6832 Lone Bijvoet mates: 1750 Anomalous signal: 0.0876 force_anomalous_flag_to_be_equal_to=False Reducing data to non-anomalous array. R-linear = sum(abs(data - mean(data))) / sum(abs(data)) R-square = sum((data - mean(data))**2) / sum(data**2) In these sums single measurements are excluded. Redundancy Mean Mean Min Max Mean R-linear R-square unused: - 29.2234 bin 1: 29.2234 - 4.0863 1 2 1.699 0.0240 0.0011 bin 2: 4.0863 - 3.2448 1 2 1.793 0.0302 0.0022 bin 3: 3.2448 - 2.8350 1 2 1.800 0.0417 0.0041 bin 4: 2.8350 - 2.5760 1 2 1.764 0.0486 0.0054 bin 5: 2.5760 - 2.3914 1 2 1.745 0.0505 0.0051 bin 6: 2.3914 - 2.2505 1 2 1.717 0.0513 0.0051 bin 7: 2.2505 - 2.1378 1 2 1.658 0.0533 0.0056 bin 8: 2.1378 - 2.0448 1 2 1.614 0.0611 0.0074 bin 9: 2.0448 - 1.9661 1 2 1.575 0.0634 0.0075 bin 10: 1.9661 - 1.8983 1 2 1.485 0.0736 0.0097 unused: 1.8983 - Fobs statistics after all cutoffs applied: Miller array info: None Observation type: xray.amplitude Type of data: double, size=9840 Type of sigmas: double, size=9840 Number of Miller indices: 9840 Anomalous flag: False Unit cell: (removed) Space group: P 21 21 21 (No. 19) Systematic absences: 0 Centric reflections: 1258 Resolution range: 29.22 1.89826 Completeness in resolution range: 0.855801 Completeness with d_max=infinity: 0.855503 What's going on here? If I let phenix.refine decide automatically, it looks like it's reading in F+ and F- separately and regarding them as independent in the refinement. If I force "no anomalous" it merges them. However I can't work out whether in either case it has actually read FOBS instead of the Friedel mates. According to the SCALA manual the I+ and I- columns will always be written out even if the keyword ANOMALOUS OFF is used, so this is a potential pitfall for many. In addition, phenix.refine seems to identify the neutron data first as xray.reconstructed_amplitude then as xray.amplitude even though the data are explicitly defined as neutron data in the GUI. I can send more information if required! Thanks Derek
Hi Derek,
Then how you ended up having anomalous data set for your neutron observations: FO,SIGFO,DANO,SIGDANO,ISYM ?
As I mentioned in my original mail, I've discovered that SCALA always outputs I+ and I- no matter what you do. From the manual:
"Normally anomalous scattering is ignored during the scale determination (I+ & I- observations are treated together), but the merged file always contains I+ & I-, even if the ANOMALOUS OFF command is used."
Thus if you don't take special precautions, e.g. in TRUNCATE, these will be converted to F+ and F- and (I now know) phenix.refine will use them. However I've now discovered that the keyword ANOMALOUS NO in TRUNCATE will prevent output of the anomalous columns altogether.
oh I see how this may be a problem. Though unfortunately I don't think there is a robust way of catching this in phenix.refine.. phenix.refine strives to do a good refinement job but obviously it can't make sure the user provides meaningful data. Any suggestions are welcome!
All in all, if you don't have anomalous data make sure your inputs files do not have spurious (anomalous) arrays: phenix.refine can't read your mind to see what you have done in your diffraction experiment - it's beyond the scope of what it's supposed to do!
Fine, I've now learned an important lesson in input/output checking! But could I politely suggest that an improvement in phenix.refine's mind-reading capabilities: if the user mistakenly inputs a file as neutron data with anomalous columns in, then these should be ignored, as the concept is by default incompatible with neutron data?
Yes, this is doable: adding to my todo list now.
Finally, it would also be nice if the neutron maps and not just the X-ray maps were displayed in Coot during refinement and if the summary panel at the end of refinement showed statistics for the neutron data as well as for the X-ray data.
Let's see what Nat thinks (he is the main and only GUI developer). Also, the concept of joint XN refinement may change as outlined in slides 28-32 here: https://www.dropbox.com/s/jsb7pks7nr019kx/neutrons-and-news.pdf Pavel
Hi again,
Sorry, in my last mail I meant "... it would also be nice if ... the progress panel during refinement showed statistics for the neutron data as well as for the X-ray data". The neutron statistics are of course shown at the end. I'm a bit tired...
Best wishes
Derek
On 29 May 2013, at 16:29, Pavel Afonine
On Wed, May 29, 2013 at 4:18 AM, Derek Logan
What's going on here? If I let phenix.refine decide automatically, it looks like it's reading in F+ and F- separately and regarding them as independent in the refinement. If I force "no anomalous" it merges them. However I can't work out whether in either case it has actually read FOBS instead of the Friedel mates.
Technically it's reading the merged amplitudes plus the anomalous differences*, and reconstructing the original Friedel pairs from these, hence "reconstructed amplitudes". We generally recommend just using the actual F+/F- (or I+/I-) instead of the reconstructed arrays, but if you have the ISYM column it probably doesn't matter too much. (And I realize that for neutron diffraction it's pointless either way, but we try not to assume too much about what the user wants.) (* By the way, does anyone actually understand why CCP4 programs outputs DANO,SIGDANO as separate columns? What programs actually use this in preference to the Friedel pairs?)
According to the SCALA manual the I+ and I- columns will always be written out even if the keyword ANOMALOUS OFF is used, so this is a potential pitfall for many.
I will add a warning to the GUI about this.
In addition, phenix.refine seems to identify the neutron data first as xray.reconstructed_amplitude then as xray.amplitude even though the data are explicitly defined as neutron data in the GUI.
That's an implementation detail - Ralf wrote all of the code to handle experimental observations with the assumption that it would be X-ray data, which (at least) 99% of the time would be true. It won't actually make any difference to the way scattering is treated as long as you select the neutron scattering table. But perhaps we should add a disclaimer to the log output... -Nat
participants (3)
-
Derek Logan -
Nathaniel Echols -
Pavel Afonine