phenix.refine 1.3 vs. 1.4 map quality ?
Hi folks: About a year ago I refined a 1.6 Å RNA structure with phenix.refine 1.3b-rc6 and then got distracted. I picked it up today and essentially repeated the last round of refinement with 1.4-57 (and also 1.4-159). The maps are subtly different, but consistently slightly worse with 1.4. Here are two examples: http://sage.ucsc.edu/~wgscott/mystuff/old_vs_new.pdf In the second example, this would lead to deletion of one of the octahedrally coordinated waters on a known Mg++ ion. 2OEU is similar, except 2.0 Å resolution, and it has Mn2+ instead of Mg ++. A similar thing happens with the corresponding water in that structure. Has the weighting for the bulk solvent mask or something like that been increased? Thanks. Bill Scott
Hi Bill,
About a year ago I refined a 1.6 Å RNA structure with phenix.refine 1.3b-rc6 and then got distracted.
I'm not sure I got the message... Was anything wrong about the maps? If so, did you let us know (sorry, I don't remember).
I picked it up today and essentially repeated the last round of refinement with 1.4-57 (and also 1.4-159). The maps are subtly different, but consistently slightly worse with 1.4.
Which maps we are looking at? I presume 2mFo-DFc (not 2Fo-Fc): regular? filled? kicked? (see copies of previous emails to the BB explaining these maps at the bottom of this email).
Here are two examples: http://sage.ucsc.edu/~wgscott/mystuff/old_vs_new.pdf
May be because I'm looking at static pictures, but apart from the water #5, the differences seem subtle indeed.
In the second example, this would lead to deletion of one of the octahedrally coordinated waters on a known Mg++ ion.
Do you mean you would delete this water manually because you don't see it in the map, or phenix.refine deletes it?
Has the weighting for the bulk solvent mask or something like that been increased?
I can name two major things: - phenix.refine outputs "filled" maps by default since December 2008 (or may be January 2009?); - a few weeks (a month?) ago we switched to using a faster and more memory efficient bulk solvent mask calculation code. Plus this new code has a minor bug fixed that existed many years in the old one. If you send me 1.6A resolution data and your best model, I will be happy to have a closer look. Pavel. PS> Copies of replies to BB about kicked and filled maps: Kick maps: """ there will be a paper about it in next coming Acta D. This method was introduced about 10-15 years ago by Dusan Turk in his program MAIN is used since that. Here is the copy-paste from the manuscript: "(...) An average kick map (AK map) is computed as following (Gunc(ar et al., 2000; Turk, 2007; Praz(nikar et al., 2009): a large ensemble of structures (several hundreds) is created where the coordinates of each structure from the ensemble are all randomly shaken. The shake amount (rmsd distortion introduced to coordinates) varies from 0 to 1.0 Å. Then for each structure a map is computed ((mFobs-DFmodel)exp(i?model) or (2mFobs-DFmodel)exp(i?model) or any other map, for example a ligand-omit map). Finally, all maps are averaged out to produce one averaged kick map. An AK map is expected to have less or no bias, less noise, enhance existing signal and potentially can clear up some initially bad densities. (...)" """. Filled maps: """ Hi Everyone, I think it's time to review this subject since it is one of the most frequently asked questions. Ironically, I think this is because we tried to make it as clear as possible -:) So, what kind of maps phenix.refine outputs by default? phenix.refine outputs two types of maximum-likelihood weighted maps (or, in other words, sigmaa-weighted maps): 2mFo-DFc and mFo-DFc. Now, as many of you noticed, the MTZ file with map coefficients "_map_coeffs.mtz" contains in fact four maps: 2mFo-DFc and mFo-DFc, and "filled" 2mFo-DFc and mFo-DFc (in fact fo-fc maps should be nearly identical; actually I have to fix it and not write identical fo-fc maps). The first two maps are computed using original Fobs (Fo), and the last two maps are computed using "filled Fobs", that is the original Fobs where missing reflections are "filled" with DFc. It is well known (I can spell a long list or references) that the data incompleteness affects the map quality, and sometimes, certain types of data incompleteness can *severely* distort maps. A possible solution (in order to reduce this negative effect) is to "model" missing Fobs somehow. One possibility is just to put in DFc in place where Fobs is missing, or as suggested by the classics, one can use <Fobs> taken in a resolution bin around a missing reflection. I even tried to use the random numbers and it was also better than doing nothing. Obviously, there is a nearly invisible line between the benefits of "filling in" missing Fobs and introducing bias. Where this line goes - is the subject of a research that to my knowledge is not done yet. Anyway, this is why phenix.refine writes out "regular" and "filled" maps: one is to give you unbiased but eventually lower quality map, and the other one is to give you a better-looking map with a risk of being biased. This way users have more options in exploring their maps (and less reasons for saying that Refmac produces better-appearing maps than phenix.refine -- see below). I have to mention that to my knowledge REFMAC always writes "filled" maps (those with missing Fobs substituted by DFc): - it is mentioned in Maria Turkenburg's thesis: http://www.ysbl.york.ac.uk/~mgwt/thesis-tth/chapter2.html#tth_sEc2.6.5 - and in Refmac docs: http://www.ccp4.ac.uk/html/refmac5/keywords/xray-general.html "Missing Data: For those reflections where the FP are missing, mFo is set equal to dFc. (...)". Correct me if I'm missing something. Please let me know if I wasn't clear in my attempt to explain this. I will update the phenix.refine documentation accordingly. Pavel. """.
Hi Pavel, is it possible to refine individual occupancies of atoms in alternative conformations? Maia Pavel Afonine wrote:
Hi Bill,
About a year ago I refined a 1.6 Å RNA structure with phenix.refine 1.3b-rc6 and then got distracted.
I'm not sure I got the message... Was anything wrong about the maps? If so, did you let us know (sorry, I don't remember).
I picked it up today and essentially repeated the last round of refinement with 1.4-57 (and also 1.4-159). The maps are subtly different, but consistently slightly worse with 1.4.
Which maps we are looking at? I presume 2mFo-DFc (not 2Fo-Fc): regular? filled? kicked? (see copies of previous emails to the BB explaining these maps at the bottom of this email).
Here are two examples: http://sage.ucsc.edu/~wgscott/mystuff/old_vs_new.pdf
May be because I'm looking at static pictures, but apart from the water #5, the differences seem subtle indeed.
In the second example, this would lead to deletion of one of the octahedrally coordinated waters on a known Mg++ ion.
Do you mean you would delete this water manually because you don't see it in the map, or phenix.refine deletes it?
Has the weighting for the bulk solvent mask or something like that been increased?
I can name two major things:
- phenix.refine outputs "filled" maps by default since December 2008 (or may be January 2009?); - a few weeks (a month?) ago we switched to using a faster and more memory efficient bulk solvent mask calculation code. Plus this new code has a minor bug fixed that existed many years in the old one.
If you send me 1.6A resolution data and your best model, I will be happy to have a closer look.
Pavel.
PS> Copies of replies to BB about kicked and filled maps:
Kick maps: """ there will be a paper about it in next coming Acta D. This method was introduced about 10-15 years ago by Dusan Turk in his program MAIN is used since that. Here is the copy-paste from the manuscript:
"(...) An average kick map (AK map) is computed as following (Gunčar et al., 2000; Turk, 2007; Pražnikar et al., 2009): a large ensemble of structures (several hundreds) is created where the coordinates of each structure from the ensemble are all randomly shaken. The shake amount (rmsd distortion introduced to coordinates) varies from 0 to 1.0 Å. Then for each structure a map is computed ((mFobs-DFmodel)exp(iαmodel) or (2mFobs-DFmodel)exp(iαmodel) or any other map, for example a ligand-omit map). Finally, all maps are averaged out to produce one averaged kick map. An AK map is expected to have less or no bias, less noise, enhance existing signal and potentially can clear up some initially bad densities. (...)" """.
Filled maps: """ Hi Everyone,
I think it's time to review this subject since it is one of the most frequently asked questions. Ironically, I think this is because we tried to make it as clear as possible -:)
So, what kind of maps phenix.refine outputs by default? phenix.refine outputs two types of maximum-likelihood weighted maps (or, in other words, sigmaa-weighted maps): 2mFo-DFc and mFo-DFc.
Now, as many of you noticed, the MTZ file with map coefficients "_map_coeffs.mtz" contains in fact four maps: 2mFo-DFc and mFo-DFc, and "filled" 2mFo-DFc and mFo-DFc (in fact fo-fc maps should be nearly identical; actually I have to fix it and not write identical fo-fc maps). The first two maps are computed using original Fobs (Fo), and the last two maps are computed using "filled Fobs", that is the original Fobs where missing reflections are "filled" with DFc. It is well known (I can spell a long list or references) that the data incompleteness affects the map quality, and sometimes, certain types of data incompleteness can *severely* distort maps.
A possible solution (in order to reduce this negative effect) is to "model" missing Fobs somehow. One possibility is just to put in DFc in place where Fobs is missing, or as suggested by the classics, one can use <Fobs> taken in a resolution bin around a missing reflection. I even tried to use the random numbers and it was also better than doing nothing. Obviously, there is a nearly invisible line between the benefits of "filling in" missing Fobs and introducing bias. Where this line goes - is the subject of a research that to my knowledge is not done yet.
Anyway, this is why phenix.refine writes out "regular" and "filled" maps: one is to give you unbiased but eventually lower quality map, and the other one is to give you a better-looking map with a risk of being biased. This way users have more options in exploring their maps (and less reasons for saying that Refmac produces better-appearing maps than phenix.refine -- see below).
I have to mention that to my knowledge REFMAC always writes "filled" maps (those with missing Fobs substituted by DFc):
- it is mentioned in Maria Turkenburg's thesis: http://www.ysbl.york.ac.uk/~mgwt/thesis-tth/chapter2.html#tth_sEc2.6.5
- and in Refmac docs: http://www.ccp4.ac.uk/html/refmac5/keywords/xray-general.html
"Missing Data: For those reflections where the FP are missing, mFo is set equal to dFc. (...)".
Correct me if I'm missing something.
Please let me know if I wasn't clear in my attempt to explain this. I will update the phenix.refine documentation accordingly.
Pavel. """.
------------------------------------------------------------------------
_______________________________________________ phenixbb mailing list [email protected] http://www.phenix-online.org/mailman/listinfo/phenixbb
Hi Maia, occupancies of atoms in alternative conformations are refined in phenix.refine by default. If your PDB file contains properly defined (with altLoc identifiers) atoms in alternative conformations, then their occupancies are the subject of constrained occupancy refinement, where occ(A)+occ(B)=1 and occupancies of all atoms belonging to an occupancy group are equal. For this scenario you don't need to do anything apart from having correctly formatted PDB file with alternative conformations defined in it. You can also ask phenix.refine to refine occupancies of selected atoms "completely freely" - without any constraints or restraints. Limits on occupancy values will not be imposed during minimization, but somewhere in refinement macro-cycle to ensure that the refined occupancies are withing [occ_min, occ_max], where occ_min and occ_max are user-defined values. This topic is in the phenix.refine documentation: http://phenix-online.org/documentation/refinement.htm#anch20 Please let me know if you have any further questions. Pavel. On 8/22/09 2:03 PM, Maia Cherney wrote:
Hi Pavel, is it possible to refine individual occupancies of atoms in alternative conformations?
Maia
Hi Bill, I suspect that the maps in 1.3-rc6 do not use DFc for missing reflections, whereas in 1.4 they do by default. However, in version 1.4 you can make non-filled maps in Coot using the other arrays in the map coefficient file. Could you try this to see if using the unfilled data makes your maps look more like the ones from 1.3b? This would be very interesting to us as we are still trying to assess the impact of filling in missing data. In addition, if you are using experimental phases in the refinement these will be used to make a phase combined map by default. Cheers, Paul On Aug 6, 2009, at 7:43 AM, William G. Scott wrote:
Hi folks:
About a year ago I refined a 1.6 Å RNA structure with phenix.refine 1.3b-rc6 and then got distracted. I picked it up today and essentially repeated the last round of refinement with 1.4-57 (and also 1.4-159). The maps are subtly different, but consistently slightly worse with 1.4.
Here are two examples: http://sage.ucsc.edu/~wgscott/mystuff/old_vs_new.pdf
In the second example, this would lead to deletion of one of the octahedrally coordinated waters on a known Mg++ ion.
2OEU is similar, except 2.0 Å resolution, and it has Mn2+ instead of Mg ++. A similar thing happens with the corresponding water in that structure.
Has the weighting for the bulk solvent mask or something like that been increased?
Thanks.
Bill Scott
_______________________________________________ phenixbb mailing list [email protected] http://www.phenix-online.org/mailman/listinfo/phenixbb
-- Paul Adams Acting Division Director, Physical Biosciences Division, Lawrence Berkeley Lab Adjunct Professor, Department of Bioengineering, U.C. Berkeley Vice President for Technology, the Joint BioEnergy Institute Head, Berkeley Center for Structural Biology Building 64, Room 248 Tel: 1-510-486-4225, Fax: 1-510-486-5909 http://cci.lbl.gov/paul Lawrence Berkeley Laboratory 1 Cyclotron Road BLDG 64R0121 Berkeley, CA 94720, USA. Executive Assistant: Patty Jimenez [ [email protected] ] [ 1-510-486-7963 ] --
Filled and non filled maps? Kicked maps? Is there documentation on what these are ? Thanks FR On Aug 6, 2009, at 10:57 AM, Paul Adams wrote:
Hi Bill,
I suspect that the maps in 1.3-rc6 do not use DFc for missing reflections, whereas in 1.4 they do by default. However, in version 1.4 you can make non-filled maps in Coot using the other arrays in the map coefficient file. Could you try this to see if using the unfilled data makes your maps look more like the ones from 1.3b? This would be very interesting to us as we are still trying to assess the impact of filling in missing data. In addition, if you are using experimental phases in the refinement these will be used to make a phase combined map by default.
Cheers, Paul
On Aug 6, 2009, at 7:43 AM, William G. Scott wrote:
Hi folks:
About a year ago I refined a 1.6 Å RNA structure with phenix.refine 1.3b-rc6 and then got distracted. I picked it up today and essentially repeated the last round of refinement with 1.4-57 (and also 1.4-159). The maps are subtly different, but consistently slightly worse with 1.4.
Here are two examples: http://sage.ucsc.edu/~wgscott/mystuff/old_vs_new.pdf
In the second example, this would lead to deletion of one of the octahedrally coordinated waters on a known Mg++ ion.
2OEU is similar, except 2.0 Å resolution, and it has Mn2+ instead of Mg ++. A similar thing happens with the corresponding water in that structure.
Has the weighting for the bulk solvent mask or something like that been increased?
Thanks.
Bill Scott
_______________________________________________ phenixbb mailing list [email protected] http://www.phenix-online.org/mailman/listinfo/phenixbb
-- Paul Adams Acting Division Director, Physical Biosciences Division, Lawrence Berkeley Lab Adjunct Professor, Department of Bioengineering, U.C. Berkeley Vice President for Technology, the Joint BioEnergy Institute Head, Berkeley Center for Structural Biology
Building 64, Room 248 Tel: 1-510-486-4225, Fax: 1-510-486-5909 http://cci.lbl.gov/paul
Lawrence Berkeley Laboratory 1 Cyclotron Road BLDG 64R0121 Berkeley, CA 94720, USA.
Executive Assistant: Patty Jimenez [ [email protected] ] [ 1-510-486-7963 ] --
_______________________________________________ phenixbb mailing list [email protected] http://www.phenix-online.org/mailman/listinfo/phenixbb
--------------------------------------------- Francis Reyes M.Sc. 215 UCB University of Colorado at Boulder gpg --keyserver pgp.mit.edu --recv-keys 67BA8D5D 8AE2 F2F4 90F7 9640 28BC 686F 78FD 6669 67BA 8D5D
I don't know what a kicked map is, but filling is done with a best estimate for missing reflections. (Treating them as zero is a bad thing.) On Aug 6, 2009, at 10:54 AM, Francis E Reyes wrote:
Filled and non filled maps? Kicked maps?
Is there documentation on what these are ?
Thanks
FR
On Aug 6, 2009, at 10:57 AM, Paul Adams wrote:
Hi Bill,
I suspect that the maps in 1.3-rc6 do not use DFc for missing reflections, whereas in 1.4 they do by default. However, in version 1.4 you can make non-filled maps in Coot using the other arrays in the map coefficient file. Could you try this to see if using the unfilled data makes your maps look more like the ones from 1.3b? This would be very interesting to us as we are still trying to assess the impact of filling in missing data. In addition, if you are using experimental phases in the refinement these will be used to make a phase combined map by default.
Cheers, Paul
On Aug 6, 2009, at 7:43 AM, William G. Scott wrote:
Hi folks:
About a year ago I refined a 1.6 Å RNA structure with phenix.refine 1.3b-rc6 and then got distracted. I picked it up today and essentially repeated the last round of refinement with 1.4-57 (and also 1.4-159). The maps are subtly different, but consistently slightly worse with 1.4.
Here are two examples: http://sage.ucsc.edu/~wgscott/mystuff/old_vs_new.pdf
In the second example, this would lead to deletion of one of the octahedrally coordinated waters on a known Mg++ ion.
2OEU is similar, except 2.0 Å resolution, and it has Mn2+ instead of Mg ++. A similar thing happens with the corresponding water in that structure.
Has the weighting for the bulk solvent mask or something like that been increased?
Thanks.
Bill Scott
_______________________________________________ phenixbb mailing list [email protected] http://www.phenix-online.org/mailman/listinfo/phenixbb
-- Paul Adams Acting Division Director, Physical Biosciences Division, Lawrence Berkeley Lab Adjunct Professor, Department of Bioengineering, U.C. Berkeley Vice President for Technology, the Joint BioEnergy Institute Head, Berkeley Center for Structural Biology
Building 64, Room 248 Tel: 1-510-486-4225, Fax: 1-510-486-5909 http://cci.lbl.gov/paul
Lawrence Berkeley Laboratory 1 Cyclotron Road BLDG 64R0121 Berkeley, CA 94720, USA.
Executive Assistant: Patty Jimenez [ [email protected] ] [ 1-510-486-7963 ] --
_______________________________________________ phenixbb mailing list [email protected] http://www.phenix-online.org/mailman/listinfo/phenixbb
--------------------------------------------- Francis Reyes M.Sc. 215 UCB University of Colorado at Boulder
gpg --keyserver pgp.mit.edu --recv-keys 67BA8D5D
8AE2 F2F4 90F7 9640 28BC 686F 78FD 6669 67BA 8D5D
_______________________________________________ phenixbb mailing list [email protected] http://www.phenix-online.org/mailman/listinfo/phenixbb
Hi Francis,
Filled and non filled maps? Kicked maps?
Is there documentation on what these are ?
*Kick maps:* There will be a paper about it in one of the next coming Acta D. This method was introduced about 10-15 years ago by Dusan Turk in his program MAIN is used since that. It is available in PHENIX too. Here is the copy-paste from the manuscript: "(...) An average kick map (AK map) is computed as following (Gunc(ar et al., 2000; Turk, 2007; Praz(nikar et al., 2009): a large ensemble of structures (several hundreds) is created where the coordinates of each structure from the ensemble are all randomly shaken. The shake amount (rmsd distortion introduced to coordinates) varies from 0 to 1.0 Å. Then for each structure a map is computed ((mFobs-DFmodel)exp(i?model) or (2mFobs-DFmodel)exp(i?model) or any other map, for example a ligand-omit map). Finally, all maps are averaged out to produce one averaged kick map. An AK map is expected to have less or no bias, less noise, enhance existing signal and potentially can clear up some initially bad densities. (...)" *Filled maps:* By default, phenix.refine outputs two types of maximum-likelihood weighted maps (or, in other words, sigmaa-weighted maps): 2mFo-DFc and mFo-DFc. The MTZ file with map coefficients "_map_coeffs.mtz" contains in fact three maps: 2mFo-DFc and mFo-DFc, and "filled" 2mFo-DFc (also, it may contain an anomalous difference map if F+/- are present). The first two maps are computed using original Fobs (Fo), and the last one is computed using "filled Fobs", that is the original Fobs where missing reflections are "filled" with DFc. It is well known (I can spell a long list or references) that the data incompleteness affects the map quality, and sometimes, certain types of data incompleteness can severely distort maps. A possible solution (in order to reduce this negative effect) is to "model" missing Fobs somehow. One possibility is just to put in DFc in place where Fobs is missing, or as suggested by the classics, one can use <Fobs> taken in a resolution bin around a missing reflection. I even tried to use the random numbers and it was also better than doing nothing. Obviously, there is a nearly invisible line between the benefits of "filling in" missing Fobs and introducing bias. Where this line goes - is the subject of a research that to my knowledge is not done yet. Anyway, this is why phenix.refine writes out "regular" and "filled" maps: one is to give you unbiased but eventually lower quality map, and the other one is to give you a better-looking map with a risk of being biased. This way users have more options in exploring their maps. I have to mention that to my knowledge REFMAC always writes "filled" maps (those with missing Fobs substituted by DFc): - it is mentioned in Maria Turkenburg's thesis: http://www.ysbl.york.ac.uk/~mgwt/thesis-tth/chapter2.html#tth_sEc2.6.5 - and in Refmac docs: http://www.ccp4.ac.uk/html/refmac5/keywords/xray-general.html "Missing Data: For those reflections where the FP are missing, mFo is set equal to dFc. (...)". Pavel.
HI Paul: I haven't had a chance yet but will do so today. I think I answered the rest in the other email just cc-ed to you. Thanks. Bill On Aug 6, 2009, at 9:57 AM, Paul Adams wrote:
Hi Bill,
I suspect that the maps in 1.3-rc6 do not use DFc for missing reflections, whereas in 1.4 they do by default. However, in version 1.4 you can make non-filled maps in Coot using the other arrays in the map coefficient file. Could you try this to see if using the unfilled data makes your maps look more like the ones from 1.3b? This would be very interesting to us as we are still trying to assess the impact of filling in missing data. In addition, if you are using experimental phases in the refinement these will be used to make a phase combined map by default.
Cheers, Paul
On Aug 6, 2009, at 7:43 AM, William G. Scott wrote:
Hi folks:
About a year ago I refined a 1.6 Å RNA structure with phenix.refine 1.3b-rc6 and then got distracted. I picked it up today and essentially repeated the last round of refinement with 1.4-57 (and also 1.4-159). The maps are subtly different, but consistently slightly worse with 1.4.
Here are two examples: http://sage.ucsc.edu/~wgscott/mystuff/old_vs_new.pdf
In the second example, this would lead to deletion of one of the octahedrally coordinated waters on a known Mg++ ion.
2OEU is similar, except 2.0 Å resolution, and it has Mn2+ instead of Mg ++. A similar thing happens with the corresponding water in that structure.
Has the weighting for the bulk solvent mask or something like that been increased?
Thanks.
Bill Scott
_______________________________________________ phenixbb mailing list [email protected] http://www.phenix-online.org/mailman/listinfo/phenixbb
-- Paul Adams Acting Division Director, Physical Biosciences Division, Lawrence Berkeley Lab Adjunct Professor, Department of Bioengineering, U.C. Berkeley Vice President for Technology, the Joint BioEnergy Institute Head, Berkeley Center for Structural Biology
Building 64, Room 248 Tel: 1-510-486-4225, Fax: 1-510-486-5909 http://cci.lbl.gov/paul
Lawrence Berkeley Laboratory 1 Cyclotron Road BLDG 64R0121 Berkeley, CA 94720, USA.
Executive Assistant: Patty Jimenez [ [email protected] ] [ 1-510-486-7963 ] --
_______________________________________________ phenixbb mailing list [email protected] http://www.phenix-online.org/mailman/listinfo/phenixbb
participants (5)
-
Francis E Reyes -
Maia Cherney -
Paul Adams -
Pavel Afonine -
William G. Scott