Thanks Gerard and Clemens (and my apologies for oversimplifying the data cutoff description). My data don’t have F_early or F_late, so the second suggestion seems likely relevant to my issue.
Best wishes
Kevin
From:
Gerard Bricogne <[email protected]>
Date: Tuesday, May 21, 2024 at 8:44 AM
To: Dr. Kevin M Jude <[email protected]>
Cc: Billy Poon <[email protected]>, Clemens Vonrhein <[email protected]>, phenixbb <[email protected]>
Subject: Re: [phenixbb] Extra map coefficients in 1.21.1-5286
Dear Kevin and phenixbb readers,
Clemens Vonrhein has just asked to join the BB but is not at present
able to reply to past messages, so he has asked me to post this message on
his behalf.
Best wishes,
Gerard
---------------------------------------------------------------
Dear Kevin,
> I note that I’ve encountered this issue with two anisotropic
> datasets that were ellipsoidally truncated using staraniso. This
> particular dataset uses a spherical cutoff, but it belongs to a
> student and I ran a cycle of refinement within one of my own
> projects that used staraniso data. May or may not be relevant, I
> just mention it in case an output setting is being created for the
> project that I don’t detect.
Your observations should not be related directly to the use of data coming
out of STARANISO, i.e. subjected to cut-off by an anisotropic surface (note
this it is not strictly correct to describe this cut-off as "ellipsoidal
truncation") instead of a spherical/isotropic one - unless there is some
special handling within phenix.refine when it encounters an input MTZ file
containing an SA_flag column (that describes the signal status [1]). Or
perhaps it looks at the history in the MTZ header (that might contain some
information about the use of STARANISO and scaling)?
There are two alternative explanations for why you now see additional map
coefficients:
* The MTZ file from STARANISO (if run via autoPROC [2]) might contain
additional sets of amplitudes, namely F_early and F_late, that can be
used not only to compute F(early)-F(late) radiation-damage detection maps
[3], but also to compute normal electron-density and difference-density
maps using those observed amplitudes in addition to the default ones used
during refinement.
* Looking at the completeness pattern, it might just be a case of the
latest version now distinguishing the set of amplitudes before any
outlier rejections (within refinement) and after such rejections:
1.21rc1_5058
Col Sort Min Max Num % Mean Mean Resolution Type Column
num order Missing complete abs. Low High label
1 NONE -42 41 0 100.00 -3.3 16.0 52.38 1.96 H H
2 NONE 0 28 0 100.00 10.3 10.3 52.38 1.96 H K
3 NONE 0 27 0 100.00 10.3 10.3 52.38 1.96 H L
4 NONE -94.4 13012.5 269 98.44 85.01 85.49 45.63 1.96 J I-obs
5 NONE 0.2 337.4 269 98.44 4.44 4.44 45.63 1.96 Q SIGI-obs
6 NONE 0.0 1.0 269 98.44 0.05 0.05 45.63 1.96 I R-free-flags
7 NONE 10.2 2509.6 321 98.14 129.94 129.94 36.90 1.96 F F-obs-filtered
8 NONE 1.3 35.8 321 98.14 10.11 10.11 36.90 1.96 Q SIGF-obs-filtered
9 NONE 0.1 2250.2 321 98.14 125.18 125.18 36.90 1.96 F F-model
10 NONE -180.0 180.0 321 98.14 4.35 89.81 36.90 1.96 P PHIF-model
11 NONE 0.0 2971.8 1165 93.23 117.81 117.81 36.90 1.96 F 2FOFCWT
12 NONE -180.0 180.0 1165 93.23 4.65 89.89 36.90 1.96 P PH2FOFCWT
13 NONE 0.0 2971.8 0 100.00 117.93 117.93 52.38 1.96 F 2FOFCWT_filled
14 NONE -180.0 180.0 0 100.00 4.85 90.05 52.38 1.96 P PH2FOFCWT_filled
15 NONE 0.0 738.9 1165 93.23 46.86 46.86 36.90 1.96 F FOFCWT
16 NONE -180.0 180.0 1165 93.23 4.75 90.35 36.90 1.96 P PHFOFCWT
==> 98.14% complete amplitudes, but "only" 93.23% complete
map-coefficients, i.e. maps computed /after/ outlier rejection
1.21.1-5286
Col Sort Min Max Num % Mean Mean Resolution Type Column
num order Missing complete abs. Low High label
1 NONE -42 41 0 100.00 -3.3 16.0 52.38 1.96 H H
2 NONE 0 28 0 100.00 10.3 10.3 52.38 1.96 H K
3 NONE 0 27 0 100.00 10.3 10.3 52.38 1.96 H L
4 NONE -94.4 13012.5 269 98.44 85.01 85.49 45.63 1.96 J I-obs
5 NONE 0.2 337.4 269 98.44 4.44 4.44 45.63 1.96 Q SIGI-obs
6 NONE 0.0 1.0 269 98.44 0.05 0.05 45.63 1.96 I R-free-flags
7 NONE 10.3 2535.0 321 98.14 131.25 131.25 36.90 1.96 F F-obs-filtered
8 NONE 1.3 36.1 321 98.14 10.21 10.21 36.90 1.96 Q SIGF-obs-filtered
9 NONE 0.0 2261.4 321 98.14 126.66 126.66 36.90 1.96 F F-model
10 NONE -180.0 180.0 321 98.14 3.96 89.82 36.90 1.96 P PHIF-model
11 NONE 0.0 3119.6 1165 93.23 119.71 119.71 36.90 1.96 F 2FOFCWT_1
12 NONE -180.0 180.0 1165 93.23 4.33 89.89 36.90 1.96 P PH2FOFCWT_1
13 NONE 0.0 3119.6 0 100.00 119.85 119.85 52.38 1.96 F 2FOFCWT_filled
14 NONE -180.0 180.0 0 100.00 4.44 90.05 52.38 1.96 P PH2FOFCWT_filled
15 NONE 0.0 780.4 1165 93.23 45.28 45.28 36.90 1.96 F FOFCWT_2
16 NONE -180.0 180.0 1165 93.23 4.21 90.36 36.90 1.96 P PHFOFCWT_2
17 NONE 0.0 3119.6 0 100.00 119.74 119.74 52.38 1.96 F 2FOFCWT_4
18 NONE -180.0 180.0 0 100.00 4.60 90.01 52.38 1.96 P PH2FOFCWT_4
19 NONE 0.0 3119.6 321 98.14 119.56 119.56 36.90 1.96 F 2FOFCWT_no_fill
20 NONE -180.0 180.0 321 98.14 4.40 89.94 36.90 1.96 P PH2FOFCWT_no_fill
21 NONE 0.0 780.4 321 98.14 45.68 45.68 36.90 1.96 F FOFCWT_5
22 NONE -180.0 180.0 321 98.14 4.12 90.23 36.90 1.96 P PHFOFCWT_5
==> 98.14% complete amplitudes, resulting in
using all amplitudes before rejection (default = fill):
98.14% complete map coefficients FOFCWT_5
100.00% complete map coefficients 2FOFCWT_4
98.14% complete map coefficients 2FOFCWT_no_fill
using amplitudes after rejection (default = nofill)
93.23% complete map coefficients FOFCWT_2
93.23% complete map coefficients 2FOFCWT_1
100.00% complete map coefficients 2FOFCWT_filled
Does that make sense? It looks more like some rather useful provision of
different ways of handling missing data (DFc completion, outlier rejection,
unmeasured due to module gaps, cusp etc). It might be that anisotropic data
as described by STARANISO, plus an assumption of isotropic diffraction (up
to the highest diffraction limit), triggers the creation of these map
coefficient variants. Does this ring a bell with developers?
[1] https://staraniso.globalphasing.org/test_set_flags_about.html
[2] https://www.globalphasing.com/autoproc/
[3] https://doi.org/10.1107/S2059798324001487
--
On Mon, May 20, 2024 at 03:53:01PM +0000, Dr. Kevin M Jude wrote:
> Hi Billy,
> Here’s a dropbox link<https://www.dropbox.com/scl/fo/16qpqwy79pcm7zgl6j608/ADqFMcQcyUw-Jq6g-bt6LBE?rlkey=abq4uyk7oeavb0ckri9tgioek&dl=0>
to input and output from one of my jobs
>
> I note that I’ve encountered this issue with two anisotropic datasets that were ellipsoidally truncated using staraniso. This particular dataset uses a spherical cutoff, but it belongs to a student and I ran a cycle of refinement within one of my own projects
that used staraniso data. May or may not be relevant, I just mention it in case an output setting is being created for the project that I don’t detect.
>
> Best wishes
> Kevin
>
> From: Billy Poon <[email protected]>
> Date: Saturday, May 18, 2024 at 1:23 PM
> To: Elke De Zitter <[email protected]>
> Cc: Dr. Kevin M Jude <[email protected]>, phenixbb <[email protected]>
> Subject: Re: [phenixbb] Extra map coefficients in 1.21.1-5286
> Dear Kevin and Elke,
>
> Can you provide your input files and settings so that we can reproduce the issue? The settings are available from the "Results" tab with the "Config file" button. Alternatively, the settings are available by selecting the job in your job history and clicking
the "Show details" button. A dialog will pop up and show the location of the job and the config file. I am not able to get the numbered map coefficients with a tutorial dataset with default settings. Thanks!
>
> --
> Billy K. Poon
> Research Scientist, Molecular Biophysics and Integrated Bioimaging
> Lawrence Berkeley National Laboratory
> 1 Cyclotron Road, M/S 33R0345
> Berkeley, CA 94720
> Fax: (510) 486-5909
> Web: https://phenix-online.org
>
>
> On Fri, May 17, 2024 at 6:18 AM Elke De Zitter <[email protected]<mailto:[email protected]>> wrote:
> Dear Kevin,
>
> I am facing the same problem upon running phenix.refine 1.21 from the command line with a map.params input file or via the GUI when the default map behavior is altered (e.g. B-sharpening). In phenix.refine 1.20, the new map coefficient parameters would replace
the default ones. I have the impression that in phenix.refine 1.21 the new map coefficient parameters add up to the defaults, leading to many additional columns. I find it confusing as well since I have to figure out myself which coefficients are stored in
*_1 etc. And, as you pointed out these mtz column labels are not automatically recognized.
>
> It would be nice to have a workaround and get the same output as when using phenix 1.20. I couldn't find one (except for using version 1.20). Suggestions are thus welcome :-)
>
> In case it is relevant, I am running phenix on a Mac. I tried version 1.21.1-5286 and dev-5330.
>
> Kind regards,
> Elke
>
> ________________________________
> From: "Dr. Kevin M Jude" <[email protected]<mailto:[email protected]>>
> To: "phenixbb" <[email protected]<mailto:[email protected]>>
> Sent: Thursday, May 2, 2024 5:52:56 PM
> Subject: [phenixbb] Extra map coefficients in 1.21.1-5286
>
> In my latest installed version of phenix, I find that additional, redundant map coefficients are being added to the output mtz file. I compare an identical refinement originally run in 1.21rc1_5058 and then repeated in 1.21.1-5286. *_filled and *_4 appear
to be redundant, and the final sets of coefficients (*_no_fill and *_5), appear to include the test set reflections, even though I’ve selected Exclude R-free set from map coefficients in the gui. This is a little bit confusing, but primarily an annoyance because
auto-open mtz in coot can’t figure out which coefficients to use.
>
> 1.21rc1_5058
> Col Sort Min Max Num % Mean Mean Resolution Type Column
> num order Missing complete abs. Low High label
>
> 1 NONE -42 41 0 100.00 -3.3 16.0 52.38 1.96 H H
> 2 NONE 0 28 0 100.00 10.3 10.3 52.38 1.96 H K
> 3 NONE 0 27 0 100.00 10.3 10.3 52.38 1.96 H L
> 4 NONE -94.4 13012.5 269 98.44 85.01 85.49 45.63 1.96 J I-obs
> 5 NONE 0.2 337.4 269 98.44 4.44 4.44 45.63 1.96 Q SIGI-obs
> 6 NONE 0.0 1.0 269 98.44 0.05 0.05 45.63 1.96 I R-free-flags
> 7 NONE 10.2 2509.6 321 98.14 129.94 129.94 36.90 1.96 F F-obs-filtered
> 8 NONE 1.3 35.8 321 98.14 10.11 10.11 36.90 1.96 Q SIGF-obs-filtered
> 9 NONE 0.1 2250.2 321 98.14 125.18 125.18 36.90 1.96 F F-model
> 10 NONE -180.0 180.0 321 98.14 4.35 89.81 36.90 1.96 P PHIF-model
> 11 NONE 0.0 2971.8 1165 93.23 117.81 117.81 36.90 1.96 F 2FOFCWT
> 12 NONE -180.0 180.0 1165 93.23 4.65 89.89 36.90 1.96 P PH2FOFCWT
> 13 NONE 0.0 2971.8 0 100.00 117.93 117.93 52.38 1.96 F 2FOFCWT_filled
> 14 NONE -180.0 180.0 0 100.00 4.85 90.05 52.38 1.96 P PH2FOFCWT_filled
> 15 NONE 0.0 738.9 1165 93.23 46.86 46.86 36.90 1.96 F FOFCWT
> 16 NONE -180.0 180.0 1165 93.23 4.75 90.35 36.90 1.96 P PHFOFCWT
>
> 1.21.1-5286
> Col Sort Min Max Num % Mean Mean Resolution Type Column
> num order Missing complete abs. Low High label
>
> 1 NONE -42 41 0 100.00 -3.3 16.0 52.38 1.96 H H
> 2 NONE 0 28 0 100.00 10.3 10.3 52.38 1.96 H K
> 3 NONE 0 27 0 100.00 10.3 10.3 52.38 1.96 H L
> 4 NONE -94.4 13012.5 269 98.44 85.01 85.49 45.63 1.96 J I-obs
> 5 NONE 0.2 337.4 269 98.44 4.44 4.44 45.63 1.96 Q SIGI-obs
> 6 NONE 0.0 1.0 269 98.44 0.05 0.05 45.63 1.96 I R-free-flags
> 7 NONE 10.3 2535.0 321 98.14 131.25 131.25 36.90 1.96 F F-obs-filtered
> 8 NONE 1.3 36.1 321 98.14 10.21 10.21 36.90 1.96 Q SIGF-obs-filtered
> 9 NONE 0.0 2261.4 321 98.14 126.66 126.66 36.90 1.96 F F-model
> 10 NONE -180.0 180.0 321 98.14 3.96 89.82 36.90 1.96 P PHIF-model
> 11 NONE 0.0 3119.6 1165 93.23 119.71 119.71 36.90 1.96 F 2FOFCWT_1
> 12 NONE -180.0 180.0 1165 93.23 4.33 89.89 36.90 1.96 P PH2FOFCWT_1
> 13 NONE 0.0 3119.6 0 100.00 119.85 119.85 52.38 1.96 F 2FOFCWT_filled
> 14 NONE -180.0 180.0 0 100.00 4.44 90.05 52.38 1.96 P PH2FOFCWT_filled
> 15 NONE 0.0 780.4 1165 93.23 45.28 45.28 36.90 1.96 F FOFCWT_2
> 16 NONE -180.0 180.0 1165 93.23 4.21 90.36 36.90 1.96 P PHFOFCWT_2
> 17 NONE 0.0 3119.6 0 100.00 119.74 119.74 52.38 1.96 F 2FOFCWT_4
> 18 NONE -180.0 180.0 0 100.00 4.60 90.01 52.38 1.96 P PH2FOFCWT_4
> 19 NONE 0.0 3119.6 321 98.14 119.56 119.56 36.90 1.96 F 2FOFCWT_no_fill
> 20 NONE -180.0 180.0 321 98.14 4.40 89.94 36.90 1.96 P PH2FOFCWT_no_fill
> 21 NONE 0.0 780.4 321 98.14 45.68 45.68 36.90 1.96 F FOFCWT_5
> 22 NONE -180.0 180.0 321 98.14 4.12 90.23 36.90 1.96 P PHFOFCWT_5
>
> Best wishes
> Kevin
>
> --
> Kevin Jude, PhD
> Structural Biology Research Specialist, Garcia Lab
> Howard Hughes Medical Institute
> Stanford University School of Medicine
> Beckman B177, 279 Campus Drive, Stanford CA 94305
> Phone: (650) 723-6431
>
> _______________________________________________
> phenixbb mailing list
> [email protected]<mailto:[email protected]>
> http://phenix-online.org/mailman/listinfo/phenixbb
> Unsubscribe: [email protected]<mailto:[email protected]>
>
> _______________________________________________
> phenixbb mailing list
> [email protected]<mailto:[email protected]>
> http://phenix-online.org/mailman/listinfo/phenixbb
> Unsubscribe: [email protected]<mailto:[email protected]>
> _______________________________________________
> phenixbb mailing list
> [email protected]
> http://phenix-online.org/mailman/listinfo/phenixbb
> Unsubscribe: [email protected]