[phenixbb] phenix.map_to_model input mtz file failure --caution on using map_to_model with X-ray data

Edward A. Berry BerryE at upstate.edu
Sun Jun 18 23:11:53 PDT 2017

```On 06/19/2017 01:24 AM, Dale Tronrud wrote:
>  In that case the entire
>> coefficient has to be set to zero.  Maybe that is what is being done here.

Yes- The reflection is omitted entirely in making the map.

Phenix.maps makes map coefficients in which the 2mFo-DFc stuff is
already incorporated; one amplitude and one phase for each reflection.
Looking in that file, when free reflections are excluded both phase and
amplitude are "?" (missing number flag). This is what is given to
real-space-refine, so it has no way to generate a coefficient for that
reflection.
I verified this by using ccp4 fft to make a map with the same coefficients
that I gave RSR, then sfall to calculate structure factors from the map.
Then looked at half a dozen missing reflection, and Fcalc from the map was
precisely zero.

A cleaner way to do it would have been to tell phenix.maps to actually make
the maps (real-space-refine takes either maps or map coefficients),
then I could sfall on precisely the map that real-space refine is using.
But if real-space refine used the map-coeff labels I told it to, it has no

ed

On 06/19/2017 01:24 AM, Dale Tronrud wrote:
> hurt so much?
>
>     Lets assume that Fcalc is correlated with its corresponding Fobs.
> (This is reasonable since your R values are 20 to 30%.)  I presume you
> are refining against some sort of (2Fo-Fc, PhiC) map.  If you put in the
> real Fc you will get amplitudes that are roughly equal to (Fo, PhiC)
> since the Fc roughly cancels out one of the Fo.  For reflections where
> Fc is zeroed out the coefficient will be, roughly, (2Fo, PhiC).  2Fo is
> still pretty strongly correlated to Fo.  Certainly large Fo's will
> produce large 2Fo's and small Fo's will produce small 2Fo's.
>
>     A reflection with twice the amplitude but the same phase will have
> peaks and valleys in exactly the same place, so I wouldn't expect huge
> differences in the location of atoms.
>
>     Now I have to ask exactly how this "set Fcalc to zero" works.  The
> simple-minded thing would be to set Fc to zero in 2Fo-Fc and get 2Fo,
> but I don't think this is the best thing to do.  The logic for
> "(2Fo-Fc,Phic)" is that this is a good estimate of the "true" Fourier
> coefficient given that you know the Fo amplitude and a (Fc,Phic) from a
> model.  If you say, instead, that you know only the Fo amplitude and
> Phic, you find that the centroid of that probability distribution is
> (Fo, Phic).  Of course it is a little odd to say that you have a high
> confidence in Phic but no confidence in Fc...  Since they are so tightly
> coupled I would rather assume I don't know either if I wanted to leave
> the test reflection out of the map calculation.  In that case the entire
> coefficient has to be set to zero.  Maybe that is what is being done here.
>
> Dale Tronrud
>
> On 6/18/2017 7:31 PM, Edward A. Berry wrote:
>> On 06/13/2017 02:15 PM, Pavel Afonine wrote:
>>>> The result is self-explicable and is inline with Tom's reply to Wei.
>>
>> Thanks, Pavel.  Yes, that is convincing.
>> I filled in some of the other results I wanted, and they do not change
>> the conclusion.
>>
>> in case anyone is interested:
>>
>> After shaking, R and Rfree are very high and not significantly different
>> (values below).
>>
>> RS Refinement, against any of the maps generated from the refined
>> structure. greatly reduces both. (So R-free is NOT increasing due to
>> free reflections being zero in the transform of the map). If free
>> reflections are excluded in making the map then R-free is higher than
>> Rwork.
>>
>> Using fill-in resulted in an R-free gap similar to that with excluding
>> reflections.
>> R and R-free were both a little lower with fill-in, but they were lower
>> by about the same amount when the free set was included, so this may be
>> more the effect of filling in the truly missing reflections than filling
>> the free reflections. And could be considered as Fc-Phic bias towards
>> the excellent map from fully refined structure.
>>
>> Refining the original refined model from the pdb by RSR (against its own
>> maps) resulted in R, R-free increasing to the same values obtained
>> starting with the shaken models. This is like the increase seen when you
>> solve a structure with a high-resolution search model and refine against
>> a low-resolution dataset.
>>
>> Making the maps using Fobs and the Fc Phi-c from the shaken model rather
>> than final refined model, which is more relevant for a crystallographer
>> (in case any crystallographer should ignore the advice given already in
>> still gave a significant decrease in both R's, and a similar R-Rfree gap
>> (or lack thereof) in all four cases.
>> Fill-in gave higher R's this time (Fc-Phic bias toward the original
>> shaken model?)
>>
>> Specifically:
>>
>>          model                     R-work   R-free
>> Original fully refined PDB         0.171   0.221 from header
>>
>> After dynamics shaking:            0.3923   0.3917
>>
>> After RS refinement of shaken:
>> Using all reflections              0.2409   0.2434
>> Using all, + Fc for missing        0.2352   0.2409
>> excluding free                     0.2419   0.2731
>>
>> After RS refinement of 1f8t:
>> Using all reflections              0.2421   0.2453
>> Using all, + Fc for missing        0.2425   0.2479
>> excluding free                     0.2453   0.2742
>>
>> After RS refinement of shaken
>> against map made from shaken:
>> Using all reflections              0.2870   0.2915
>> Using all, + Fc for missing        0.3008   0.3034
>> excluding free                     0.2917   0.3236
>>
>> It is surprising to me that making the map with fillin Fc
>> from the refined model vs the bad starting model makes a significant
>> difference, but making the map with Fc=0 doesn't seem to hurt.
>>
>>
>> On 06/13/2017 02:15 PM, Pavel Afonine wrote:
>>> Hi Ed,
>>>
>>> Including free-r reflections into map calculation and then using such
>>> map in real-space refinement of entire model will affect Rfree. Here
>>> is a simple example that illustrates my statement, step-by-step:
>>>
>>> 1) Get data and model from PDB:
>>>
>>> phenix.fetch_pdb 1f8t --mtz
>>>
>>> 2) Compute two 2mFo-DFc maps: one includes all reflections the other
>>> one has no free-r terms:
>>>
>>> phenix.python run.py 1f8t.{pdb,mtz}
>>>
>>> This will create an MTZ file (map_coeffs.mtz) that contains Fourier
>>> map coefficients for both maps.
>>>
>>> 3) Shake model a bit:
>>>
>>> phenix.dynamics 1f8t.pdb number_of_steps=500
>>>
>>> 4) Run real-space refinement using two maps:
>>>
>>> phenix.real_space_refine map_coeffs.mtz 1f8t_shaken.pdb
>>> label="work,PHIwork" ncs_constraints=false output.file_name_prefix=work
>>>
>>> phenix.real_space_refine map_coeffs.mtz 1f8t_shaken.pdb
>>> label="all,PHIall" ncs_constraints=false output.file_name_prefix=all
>>>
>>> 5) Compute R-factors using data and real-space refined models:
>>>
>>> phenix.model_vs_data 1f8t.mtz all_real_space_refined.pdb
>>>       r_work(re-computed)                : 0.2419
>>>       r_free(re-computed)                : 0.2441
>>>
>>> phenix.model_vs_data 1f8t.mtz work_real_space_refined.pdb
>>>       r_work(re-computed)                : 0.2444
>>>       r_free(re-computed)                : 0.2756
>>>
>>> The result is self-explicable and is inline with Tom's reply to Wei.
>>>
>>> All files necessary to reproduce calculations above are here:
>>> http://cci.lbl.gov/~afonine/tmp/
>>>
>>> All the best,
>>> Pavel
>>>
>>>
>>> On 6/8/17 10:05, Tim Gruene wrote:
>>>> Hi Ed,
>>>>
>>>> including the 'free' reflections in the map for modelling does not
>>>> taint the
>>>> value of Rfree. That is a misconception that i s very persistent (as
>>>> prejudice
>>>> usually are). I believe it was Ian Tickle who formulated that when
>>>> you simply
>>>> refine long enough towards convergence, all reflections excluded from
>>>> refinement
>>>> will become independent, i.e. you can assign a new set for Rfree
>>>> every time
>>>> you refine, if you wish so.
>>>>
>>>> This concept is the reason why Rcomplete (the "better" equivalent to
>>>> Rfree for
>>>> small data sets with < 10,000 unique reflections), introduced by Axel
>>>> Brunger,
>>>> works, as we could demonstrate in     doi: 10.1073/pnas.1502136112
>>>>
>>>> So nothing to worry about when including all reflections in map
>>>> calculations.
>>>>
>>>> Cheers,
>>>> Tim
>>>>
>>>> On Thursday, June 8, 2017 12:42:53 PM CEST Edward A. Berry wrote:
>>>>> Hi, Tom,
>>>>> Please forgive what may be a silly question from an outsider who hasn't
>>>>> really kept up with the crystallography literature or even all the
>>>>> Phenix
>>>>> newsletters- What is the evidence that including the free set in
>>>>> real space
>>>>> refinement biases R-free of the resulting model? Is this Rfree also
>>>>> biased
>>>>> when map coefficients use "fill-in" for the excluded free
>>>>> reflections (and
>>>>> is that what phenix.remove_free_from_map does?).
>>>>>
>>>>> My point is that literally excluding the free reflections, as
>>>>> opposed to
>>>>> substituting their values with Fc, will bias the free set toward
>>>>> grossly
>>>>> incorrect values (namely zero) and therefore greatly worsen R-free.
>>>>> Thus if
>>>>> the evidence for bias is that you get worse R-free when you exclude the
>>>>> free set, you have to think about how much of that difference
>>>>> results from
>>>>> bias towards the observed values (when the reflections are included)
>>>>> and
>>>>> how much is from bias towards zero (when the free set is excluded).
>>>>> (Again, I realize this may be all very well understood by the
>>>>> crystallography community and properly taken care of in phenix; I'm
>>>>> just
>>>>> asking for my own information) eab
>>>>>
>>>>> On 06/08/2017 07:28 AM, Terwilliger, Thomas Charles wrote:
>>>>>> ​Hi Wei,
>>>>>>
>>>>>>
>>>>>> I want to give a word of caution about how to use
>>>>>> phenix.map_to_model on
>>>>>> crystallographic data...The bottom line is you should remove the
>>>>>> test set
>>>>>> from your map coefficients before running phenix.map_to model on X-ray
>>>>>> data.  Here is why:
>>>>>>
>>>>>>
>>>>>> phenix.map_to_model uses real-space refinement, which is refinement
>>>>>> against the map. If you supply map coefficients that include your test
>>>>>> reflections, then you will be refining against data that is in your
>>>>>> test
>>>>>> set.   This will make your Rfree invalid when you go back and
>>>>>> refine your
>>>>>> model against the original crystallographic data.
>>>>>>
>>>>>>
>>>>>> To remove the test set from your map coefficients you can use:
>>>>>>
>>>>>>
>>>>>> phenix.remove_free_from_map  map_coeffs=my_map_coeffs.mtz
>>>>>> free_in=my_data_file_with_freeR_flags.mtz
>>>>>> mtz_out=my_map_coeffs_no_free.mtz
>>>>>>
>>>>>>
>>>>>> Also note that phenix.map_to_model uses a fixed map (it does not do
>>>>>> density modification).  Consequently for most crystallographic data at
>>>>>> moderate resolution or higher phenix.autobuild is going to do much
>>>>>> better
>>>>>> than phenix.map_to_model.
>>>>>>
>>>>>>
>>>>>> All the best,
>>>>>>
>>>>>> Tom T
>>>>>>
>>>>>>
>>>>>> --------------------------------------------------------------------------
>>>>>>
>>>>>> --------------------------------------------------------------------------
>>>>>>
>>>>>> --------------------------------------------------------------------------
>>>>>>
>>>>>> --------------------------------------------------------------------------
>>>>>>
>>>>>> --------------------------------------------------------------------------
>>>>>>
>>>>>> --------------------------------------------------------------------------
>>>>>>
>>>>>> --------------------------------------------------------------------------
>>>>>>
>>>>>> --------------------------------------------------------------------------
>>>>>>
>>>>>> --------------------------------------------------------------------------
>>>>>>
>>>>>> --------------------------------------------------------------------------
>>>>>>
>>>>>> --------------------------------------------------------------------------
>>>>>>
>>>>>> --------------------------------------------------------------------------
>>>>>>
>>>>>> --------------------------------------------------------------------------
>>>>>>
>>>>>> ---------------------------- *From:*dingding830106 at 163.com
>>>>>> <dingding830106 at 163.com>  on behalf ofdancingdream at 163.com
>>>>>> <dancingdream at 163.com>  *Sent:* Tuesday, June 6, 2017 9:16 PM
>>>>>> *To:* Terwilliger, Thomas Charles
>>>>>> *Cc:*phenixbb at phenix-online.org
>>>>>> *Subject:* Re:Re: [phenixbb] phenix.map_to_model input mtz file
>>>>>> failure
>>>>>> Dear Thomas,
>>>>>> I use CAD to convert the labels from FDM->FWT, PHIDM->PHFWT, then
>>>>>> submit
>>>>>> this job again (without map_coeffs_labels=... ), and everything
>>>>>> seems ok.
>>>>>> Thank you very much for you help.
>>>>>> Best!
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Wei Ding
>>>>>> P.O.Box 603
>>>>>> The Institute of Physics,Chinese Academy of Sciences
>>>>>> Beijing,China
>>>>>> 100190
>>>>>> Tel: +86-10-82649083
>>>>>>
>>>>>> E-mail:dingwei at iphy.ac.cn  <mailto:wangli at moon.ibp.ac.cn>
>>>>>>
>>>>>> At 2017-06-07 10:32:14, "Terwilliger, Thomas Charles"
>>>> <terwilliger at lanl.gov>  wrote:
>>>>>>       Hi Wei,
>>>>>>
>>>>>>
>>>>>>       I'm sorry for the trouble!
>>>>>>
>>>>>>
>>>>>>       If you supply an MTZ file that has FWT,PHFWT or similar
>>>>>> labels, then
>>>>>>       you can skip the "labels=...." statement and it should run.
>>>>>>
>>>>>>
>>>>>>       Let me know if that does not work!
>>>>>>       All the best,
>>>>>>
>>>>>>       Tom T
>>>>>>
>>>>>>
>>>>>>
>>>>>> ----------------------------------------------------------------------
>>>>>>
>>>>>> ----------------------------------------------------------------------
>>>>>>
>>>>>> ----------------------------------------------------------------------
>>>>>>
>>>>>> ----------------------------------------------------------------------
>>>>>>
>>>>>> ----------------------------------------------------------------------
>>>>>>
>>>>>> ----------------------------------------------------------------------
>>>>>>
>>>>>> ----------------------------------------------------------------------
>>>>>>
>>>>>> ----------------------------------------------------------------------
>>>>>>
>>>>>> ----------------------------------------------------------------------
>>>>>>
>>>>>> ----------------------------------------------------------------------
>>>>>>
>>>>>> ----------------------------------------------------------------------
>>>>>>
>>>>>> ----------------------------------------------------------------------
>>>>>>
>>>>>> ----------------------------------------------------------------------
>>>>>>
>>>>>> ----------------------------------------------------------------------
>>>>>>       ---------- *From:*phenixbb-bounces at phenix-online.org
>>>>>>       <mailto:phenixbb-bounces at phenix-online.org>
>>>>>>       <phenixbb-bounces at phenix-online.org
>>>>>>       <mailto:phenixbb-bounces at phenix-online.org>> on behalf of
>>>>>>       dancingdream at 163.com  <mailto:dancingdream at 163.com>
>>>>>>       <dancingdream at 163.com  <mailto:dancingdream at 163.com>> *Sent:*
>>>>>> Tuesday,
>>>>>>       June 6, 2017 8:19 PM
>>>>>>       *To:*phenixbb at phenix-online.org
>>>>>> <mailto:phenixbb at phenix-online.org>
>>>>>>       *Subject:* [phenixbb] phenix.map_to_model input mtz file failure
>>>>>>       Dear Phenix bb,
>>>>>>       I intend to build a initial model by phenix.map_to_model. And the
>>>>>>       command line is as follows: phenix.map_to_model_1.12rc0-2787
>>>>>>       map_coeffs_file=../rep_dm.mtz map_coeffs_labels="'FP,SIGFP'
>>>>>> 'PHIDM'
>>>>>>       'FOMDM'" seq_file=../resolve.seq  is_crystal=True
>>>>>>       use_sg_symmetry=True  density_select=False
>>>>>> truncate_at_d_min=True
>>>>>>       and the feedback like this:
>>>>>>       Sorry: No initial assignment made for map_coeffs. Labels used:
>>>>>>       FP,SIGFP PHIDM FOMDM. Available labels: ['PHIB', 'FOM',
>>>>>>       'HLA,HLB,HLC,HLD', 'FP,SIGFP', 'PHIDM', 'FOMDM', 'FDM',
>>>>>>       'HLADM,HLBDM,HLCDM,HLDDM'] NOTE: grouped labels like
>>>>>> 'FP,SIGFP' must
>>>>>>       stay together,
>>>>>>       have commas, and have no spaces. If they come from an MTZ file,
>>>>>>       they must be in adjacent columns as well.
>>>>>>       Suggested labels to use:  PHIDM  FOMDM
>>>>>>       I try many other input format of map_coeffs_labels, such as
>>>>>>       map_coeffs_labels="FP,SIGFP PHIDM FOMDM"
>>>>>>       map_coeffs_labels=["FP,SIGFP PHIDM FOMDM"]
>>>>>>       ... ...
>>>>>>       but the result is the same. Dose anyone can tell me how to fix
>>>>>> this
>>>>>>       problem? Thank a lot.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>       --
>>>>>>       Wei Ding
>>>>>>       P.O.Box 603
>>>>>>       The Institute of Physics,Chinese Academy of Sciences
>>>>>>       Beijing,China
>>>>>>       100190
>>>>>>       Tel: +86-10-82649083
>>>>>>       E-mail:dingwei at iphy.ac.cn  <mailto:wangli at moon.ibp.ac.cn>
>>>>>>
>>>>>> _______________________________________________
>>>>>> phenixbb mailing list
>>>>>> phenixbb at phenix-online.org
>>>>>> http://phenix-online.org/mailman/listinfo/phenixbb
>>>>>> Unsubscribe:phenixbb-leave at phenix-online.org
>>>>> _______________________________________________
>>>>> phenixbb mailing list
>>>>> phenixbb at phenix-online.org
>>>>> http://phenix-online.org/mailman/listinfo/phenixbb
>>>>> Unsubscribe:phenixbb-leave at phenix-online.org
>>>>
>>>>
>>>> _______________________________________________
>>>> phenixbb mailing list
>>>> phenixbb at phenix-online.org
>>>> http://phenix-online.org/mailman/listinfo/phenixbb
>>>> Unsubscribe:phenixbb-leave at phenix-online.org
>>>
>> _______________________________________________
>> phenixbb mailing list
>> phenixbb at phenix-online.org
>> http://phenix-online.org/mailman/listinfo/phenixbb
>> Unsubscribe: phenixbb-leave at phenix-online.org
> _______________________________________________
> phenixbb mailing list
> phenixbb at phenix-online.org
> http://phenix-online.org/mailman/listinfo/phenixbb
> Unsubscribe: phenixbb-leave at phenix-online.org
>
```