Search results for "look through"

Re: [phenixbb] calculate Fc for the whole unit cell from a Fc of a single symmetric unit.

by Dale Tronrud

On 07/11/11 15:35, Edward A. Berry wrote: > It seems to me there are two things that could be meant by "expand to P1" > > One is when data has been reduced to the Reciprocal Space > asymmetric unit (or otherwise one asymmetric unit of a > symmetric dataset has been obtained) and you want to expand > it to P1 by using symmetry to generate all the > symmetry -equivalent reflections. > > The other is where a full P1 dataset has been calculated from just > one asymmetric unit of the crystal (and hence does not exhibit the > crystallographic symmetry) and you want to generate the transform > of the entire crystal. (I think this is how all the space-group - > specific fft programs used to work before computers got so fast it > was less bother to just do everything in P1 with the whole cell) > Presumably this involves applying the real space symmetry > operators to get n rotated (or phase-shifted for translational > symmetry) P1 datasets and adding them vectorially. If I recall, that is how XPLOR calculated structure factors but that is not how a Ten Eyck space group specific FFT works. First it expands the atoms into the largest subgroup composed only of symmetry elements that can be optimized (e.g. three-folds are out). Then the electron density of the asymmetric unit of that space group is calculated. Finally the structure factors are calculated by a combination of line group optimized 1-D FFTs and clever rearrangements that move data from useless positions in memory to places where the final 1-D FFTs can use them. Both the calculation of the map (the slowest step) and the FFT itself are sped up by a factor of the number of usable symmetry operators. In addition the amount of RAM required was greatly reduced. Back in the day the latter was often more important because memory was so limited. Dale Tronrud > > It would be important to decide which of these is required, and which > each of the suggested methods provides > > eab > > Ralf Grosse-Kunstleve wrote: >> We can expand reciprocal-space arrays, too, with the >> cctbx.miller.array.expand_to_p1() method. You can use it from the >> command line via >> >> phenix.reflection_file_converter --expand-to-p1 ... >> >> See also: >> http://www.phenix-online.org/documentation/reflection_file_tools.htm >> >> Ralf >> >> >> On Mon, Jul 11, 2011 at 10:56 AM, <zhangh1(a)umbc.edu >> <mailto:[email protected]>> wrote: >> >> Sorry I haven't got a chance to check my email recently. >> >> Yes, I meant expansion to P1. The thing is cctbx relies on the atomic >> model I think, but I only have model Fc available. >> >> Hailiang >> >> > I suspect what Hailang means is expansion into P1. >> > >> > I am sure this can be accomplished through some either existing or >> > easily coded cctbx tool. However, when I looked into a different >> task >> > recently that included P1 expansion as a step, I learned that >> SFTOOLs >> > can do this, albeit there was a bug there which caused trouble in >> > certain space groups (may be fixed by now so check if there is an >> > update). >> > >> > Hailang - if P1 expansion is what you need, I could share my own >> code as >> > well, let me know if that is something you want to try. >> > >> > Cheers, >> > >> > Ed. >> > >> > On Fri, 2011-07-08 at 14:44 -0700, Ralf Grosse-Kunstleve wrote: >> >> Did you get responses already? >> >> If not, could you explain your situation some more? >> >> We have algorithms that do the symmetry summation in reciprocal >> space. >> >> The input is a list of Fc in P1, based on the unit cell of the >> >> crystal. Is that what you have? >> >> Ralf >> >> >> >> On Wed, Jul 6, 2011 at 1:38 PM, <zhangh1(a)umbc.edu >> <mailto:[email protected]>> wrote: >> >> Hi, >> >> >> >> I am wondering if I only have structure factors >> calculated >> >> from a single >> >> symmetric unit, is there any phenix utility which can >> >> calculate the >> >> structure factor for the whole unit cell given the >> symmetric >> >> operation or >> >> space group and crystal parameters? Note I don't have an >> >> atomic model and >> >> only have Fc. >> >> >> >> Thanks! >> >> >> >> Hailiang >> >> >> >> _______________________________________________ >> >> phenixbb mailing list >> >> phenixbb(a)phenix-online.org <mailto:[email protected]> >> >> http://phenix-online.org/mailman/listinfo/phenixbb >> >> >> >> >> >> _______________________________________________ >> >> phenixbb mailing list >> >> phenixbb(a)phenix-online.org <mailto:[email protected]> >> >> http://phenix-online.org/mailman/listinfo/phenixbb >> > >> > >> > _______________________________________________ >> > phenixbb mailing list >> > phenixbb(a)phenix-online.org <mailto:[email protected]> >> > http://phenix-online.org/mailman/listinfo/phenixbb >> > >> > >> >> >> _______________________________________________ >> phenixbb mailing list >> phenixbb(a)phenix-online.org <mailto:[email protected]> >> http://phenix-online.org/mailman/listinfo/phenixbb >> >> >> >> >> _______________________________________________ >> phenixbb mailing list >> phenixbb(a)phenix-online.org >> http://phenix-online.org/mailman/listinfo/phenixbb > > _______________________________________________ > phenixbb mailing list > phenixbb(a)phenix-online.org > http://phenix-online.org/mailman/listinfo/phenixbb

13 years, 11 months

Re: [phenixbb] Validation of structure with modified residue

by Xavier Brazzolotto

After some careful inspection. The geometry on the C atom of the ligand is weird, I don’t get something close to tetrahedral (or similar). Probably some angles are missing or I did something wrong with the ligand cif file. Not fixed yet > Le 21 avr. 2022 à 13:39, Xavier Brazzolotto <xbrazzolotto(a)gmail.com> a écrit : > > I’ve re-processed the structure separating the SER residue from the ligand part. Now I have independent ligand. > In the « Custom Geometry Restraints » I’ve defined the bond between OG and the carbon atom of the ligand and I’ve defined the angles (I’ve used the values from the previously determined eLBOW run off the SER-bound ligand complex), saved the restraints and launched the refinement. At a first look it was processed correctly and the final cif file has now the whole protein in Chain A. > > I’ve used prepare PDB deposition using the FASTA sequence of the protein (wonder if I have to provide the ligand CIF file and add more options) and ran phenix.get_pdb_validation : the report looks ok for protein and some other basic ligands (sugars, buffer, glycerol, etc...) but the ligand of interest was not processed (EDS FAILED...). In the PDB file, all these extra ligands are also in Chain A, with water in chain B. > > If I try the validation through the website (PDBe@EBI) with both cif files from the Refine or the Prepare_PDB_Deposition process, both seem to crash the server as it takes forever without Finalizing... > > I wonder if I am missing something… Maybe declaration of removal of atoms : HG bound to OG in SER or/and removal of one H from the carbon of the ligand involved in the bond ? > > Xavier > >> Le 21 avr. 2022 à 08:06, Xavier Brazzolotto <xbrazzolotto(a)gmail.com <mailto:[email protected]>> a écrit : >> >> Thank you for your feedback. >> >> @Paul, I’ve run the « Prepare model for deposition » with the option modified residue (SLG). Not sure it will change if I change the name as it is already the PDB database, but I will give it another try. >> >> I think that I will have to describe only the ligand and add some parameters restricting distance and angles between the OG of SER and the ligand, I think this is right way. >> @ Nigel, is that what you mean with « details » ? If you have any other « tips/tricks » they are welcome. >> >> Best >> Xavier >> >>> Le 21 avr. 2022 à 02:47, Nigel Moriarty <nwmoriarty(a)lbl.gov <mailto:[email protected]>> a écrit : >>> >>> Xavier >>> >>> Paul's point is very valid because the "Prepare for Deposition" step is what generates the sequence (which is the crucial point here) for deposition. However, because you have "created" a new amino acid, there will still be issues as highlighted by Pavel. It is a corner case. >>> >>> One small addition point is that SLG is already taken in the PDB Ligand list. There are tools in Phenix to find an used code. >>> >>> Can you re-engineer it with SER+ligand? This will solve the problem using the current Phenix version. I can help with the details if needed. >>> >>> Cheers >>> >>> Nigel >>> >>> --- >>> Nigel W. Moriarty >>> Building 33R0349, Molecular Biophysics and Integrated Bioimaging >>> Lawrence Berkeley National Laboratory >>> Berkeley, CA 94720-8235 >>> Phone : 510-486-5709 Email : NWMoriarty(a)LBL.gov <mailto:[email protected]> >>> Fax : 510-486-5909 Web : CCI.LBL.gov <http://cci.lbl.gov/> >>> ORCID : orcid.org/0000-0001-8857-9464 <https://orcid.org/0000-0001-8857-9464> >>> >>> >>> On Wed, Apr 20, 2022 at 5:02 PM Paul Adams <pdadams(a)lbl.gov <mailto:[email protected]>> wrote: >>> >>> Please also remember that you need to run “Prepare model for PDB deposition” (in the GUI under "PDB Deposition") on the mmCIF file you get from phenix.refine. This provides important information that is required for the deposition at the PDB. >>> >>>> On Apr 20, 2022, at 1:58 PM, Xavier Brazzolotto <xbrazzolotto(a)gmail.com <mailto:[email protected]>> wrote: >>>> >>>> Dear Phenix users, >>>> >>>> I don’t know if my problem is related to Phenix but for information I’m running Phenix 1.20.1-4487 under MacOS 12.3.1. >>>> >>>> I’ve finalized a structure where a ligand covalently modified the protein. >>>> >>>> I’ve generated the modified residue (named SLG for serine modified by ligand). For this I’ve generated the molecules in SMILES and used eLBOW to generate the restraints. Then I’ve modified the cif file defining the molecule as a L-peptide and replacing the atom names of the Serine part (CA, CB, OG, C, O, N, and OXT) >>>> In coot (from CCP4 : 0.9.6 EL), I’ve used the modified cif file and it allowed merging of the modified residue into the polypeptide chain as expected and further refinements went without any issue in Phenix (providing the modified cif file of course). Everything seems well interpreted. So far so good. >>>> >>>> However, now I would like to validate the structure and both Phenix validation tool and the PDB web server do not accept the final cif file. >>>> >>>> Checking this file I’ve noticed that the protein seems split into 3 pieces (chain A, first residue up to the one before the modified residue; chain B the modified residue by itself described as HETATM and chain C the rest of the polypeptide up to the C-ter). >>>> The PDB file presents only one chain A for the whole protein with the modified residue... >>>> >>>> I don’t know if this is an issue with Phenix generating this final cif file in this specific case or if I need to modify this final file by hand ? >>>> >>>> Any help is welcome. >>>> Thanks >>>> >>>> Xavier >>>> >>>> >>>> >>>> _______________________________________________ >>>> phenixbb mailing list >>>> phenixbb(a)phenix-online.org <mailto:[email protected]> >>>> http://phenix-online.org/mailman/listinfo/phenixbb <http://phenix-online.org/mailman/listinfo/phenixbb> >>>> Unsubscribe: phenixbb-leave(a)phenix-online.org <mailto:[email protected]> >>> -- >>> Paul Adams (he/him/his) >>> Associate Laboratory Director for Biosciences, LBL (https://biosciences.lbl.gov/leadership/ <https://biosciences.lbl.gov/leadership/>) >>> Principal Investigator, Computational Crystallography Initiative, LBL (https://cci.lbl.gov <https://cci.lbl.gov/>) >>> Vice President for Technology, the Joint BioEnergy Institute (http://www.jbei.org <http://www.jbei.org/>) >>> Principal Investigator, ALS-ENABLE, Advanced Light Source (https://als-enable.lbl.gov <https://als-enable.lbl.gov/>) >>> Division Deputy for Biosciences, Advanced Light Source (https://als.lbl.gov <https://als.lbl.gov/>) >>> Laboratory Research Manager, ENIGMA Science Focus Area (http://enigma.lbl.gov <http://enigma.lbl.gov/>) >>> Adjunct Professor, Department of Bioengineering, UC Berkeley (http://bioeng.berkeley.edu <http://bioeng.berkeley.edu/>) >>> Member of the Graduate Group in Comparative Biochemistry, UC Berkeley (http://compbiochem.berkeley.edu <http://compbiochem.berkeley.edu/>) >>> >>> Building 33, Room 250 >>> Building 978, Room 4126 >>> Building 977, Room 268 >>> Tel: 1-510-486-4225 >>> http://cci.lbl.gov/paul <http://cci.lbl.gov/paul> >>> ORCID: 0000-0001-9333-8219 >>> >>> Lawrence Berkeley Laboratory >>> 1 Cyclotron Road >>> BLDG 33R0345 >>> Berkeley, CA 94720, USA. >>> >>> Executive Assistant: Michael Espinosa [ MEEspinosa(a)lbl.gov <mailto:[email protected]> ] [ 1-510-333-6788 ] >>> Phenix Consortium: Ashley Dawn [ AshleyDawn(a)lbl.gov <mailto:[email protected]> ][ 1-510-486-5455 ] >>> >>> -- >>> >>> _______________________________________________ >>> phenixbb mailing list >>> phenixbb(a)phenix-online.org <mailto:[email protected]> >>> http://phenix-online.org/mailman/listinfo/phenixbb <http://phenix-online.org/mailman/listinfo/phenixbb> >>> Unsubscribe: phenixbb-leave(a)phenix-online.org <mailto:[email protected]> >

3 years, 2 months

Re: [cctbxbb] use_internal_variance in iotbx.merging_statistics

by Keitaro Yamashita

Dear Richard and everyone, Thanks for your reply. What kind of input do you give to iotbx.merging_statistics in xia2? For example, when XDS file is given, use_internal_variance=False is not passed to merge_equivalents() function. Please look at the lines of filter_intensities_by_sigma.__init__() in iotbx/merging_statistics.py. When sigma_filtering == "xds" or sigma_filtering == "scalepack", array_merged is recalculated using merge_equivalents() with default arguments. If nobody disagrees, I would like to commit the fix so that use_internal_variance variable is passed to all merge_equivalents() function calls. I am afraid that the behavior in the phenix-1.11 would be confusing. In phenix.table_one (mmtbx/command_line/table_one.py), use_internal_variance=False is default. This will be OK with the fix I suggested above. Can it also be default in phenix.merging_statistics, not to change the program behavior through phenix versions? Best regards, Keitaro 2016-11-01 18:21 GMT+09:00 <richard.gildea(a)diamond.ac.uk>: > Dear Keitaro, > > iotbx.merging_statistics does have the option to change the parameter use_internal_variance. In xia2 we use the defaults use_internal_variance=False, eliminate_sys_absent=False, n_bins=20, when calculating merging statistics which give comparable results to those calculate by Aimless: > > $ iotbx.merging_statistics > Usage: > phenix.merging_statistics [data_file] [options...] > > Calculate merging statistics for non-unique data, including R-merge, R-meas, > R-pim, and redundancy. Any format supported by Phenix is allowed, including > MTZ, unmerged Scalepack, or XDS/XSCALE (and possibly others). Data should > already be on a common scale, but with individual observations unmerged. > Diederichs K & Karplus PA (1997) Nature Structural Biology 4:269-275 > (with erratum in: Nat Struct Biol 1997 Jul;4(7):592) > Weiss MS (2001) J Appl Cryst 34:130-135. > Karplus PA & Diederichs K (2012) Science 336:1030-3. > > > Full parameters: > > file_name = None > labels = None > space_group = None > unit_cell = None > symmetry_file = None > high_resolution = None > low_resolution = None > n_bins = 10 > extend_d_max_min = False > anomalous = False > sigma_filtering = *auto xds scala scalepack > .help = "Determines how data are filtered by SigmaI and I/SigmaI. XDS" > "discards reflections whose intensity after merging is less than" > "-3*sigma, Scalepack uses the same cutoff before merging, and" > "SCALA does not do any filtering. Reflections with negative SigmaI" > "will always be discarded." > use_internal_variance = True > eliminate_sys_absent = True > debug = False > loggraph = False > estimate_cutoffs = False > job_title = None > .help = "Job title in PHENIX GUI, not used on command line" > > > Below is my email to Pavel and Billy when we discussed this issue by email a while back: > > The difference between use_internal_variance=True/False is explained in Luc's document here: > > libtbx.pdflatex $(libtbx.find_in_repositories cctbx/miller)/equivalent_reflection_merging.tex > > Essentially use_internal_variance=False uses only the unmerged sigmas to compute the merged sigmas, whereas use_internal_variance=True uses instead the spread of the unmerged intensities to compute the merged sigmas. Furthermore, use_internal_variance=True uses the largest of the variance coming from the spread of the intensities and that computed from the unmerged sigmas. As a result, use_internal_variance=True can only ever give lower I/sigI than use_internal_variance=False. The relevant code in the cctbx is here: > > https://sourceforge.net/p/cctbx/code/HEAD/tree/trunk/cctbx/miller/merge_equ… > > Aimless has a similar option for the SDCORRECTION keyword, if you set the option SAMPLESD, which I think is equivalent to use_internal_variance=True. The default behaviour of Aimless is equivalent to use_internal_variance=False: > > http://www.mrc-lmb.cam.ac.uk/harry/pre/aimless.html#sdcorrection > > "SAMPLESD is intended for very high multiplicity data such as XFEL serial data. The final SDs are estimated from the weighted population variance, assuming that the input sigma(I)^2 values are proportional to the true errors. This probably gives a more realistic estimate of the error in <I>. In this case refinement of the corrections is switched off unless explicitly requested." > > I think that the "external" variance is probably better if the sigmas from the scaling program are reliable, or for low multiplicity data. For high multiplicity data or if the sigmas from the scaling program are not reliable, then "internal" variance is probably better. > > Cheers, > > Richard > > Dr Richard Gildea > Data Analysis Scientist > Tel: +441235 77 8078 > > Diamond Light Source Ltd. > Diamond House > Harwell Science & Innovation Campus > Didcot > Oxfordshire > OX11 0DE > > ________________________________________ > From: cctbxbb-bounces(a)phenix-online.org [cctbxbb-bounces(a)phenix-online.org] on behalf of Keitaro Yamashita [k.yamashita(a)spring8.or.jp] > Sent: 01 November 2016 07:23 > To: cctbx mailing list > Subject: [cctbxbb] use_internal_variance in iotbx.merging_statistics > > Dear Phenix/CCTBX developers, > > iotbx/merging_statistics.py is used by phenix.merging_statistics, > phenix.table_one, and so on. By upgrading phenix from 1.10.1 to 1.11, > merging statistics-related codes were significantly changed. > > Previously, miller.array.merge_equivalents() was always called with > argument use_internal_variance=False, which is consistent with XDS, > Aimless and so on. Currently, use_internal_variance=True is default, > and cannot be changed by users (see below). > > These changes were made by @afonine and @rjgildea in rev. 22973 (Sep > 26, 2015) and 23961 (Mar 8, 2016). Could anyone explain why these > changes were introduced? > > https://sourceforge.net/p/cctbx/code/22973 > https://sourceforge.net/p/cctbx/code/23961 > > > My points are: > > - We actually cannot control use_internal_variance= parameter because > it is not passed to merge_equivalents() in class > filter_intensities_by_sigma. > > - In previous versions, if I gave XDS output to > phenix.merging_statistics, <I/sigma> values calculated in the same way > (as XDS does) were shown; but not in the current version. > > - For (for example) phenix.table_one users who expect this behavior, > it can give inconsistency. The statistics would not be consistent with > the data used in refinement. > > > cf. the related discussion in cctbxbb: > http://phenix-online.org/pipermail/cctbxbb/2012-October/000611.html > > > Best regards, > Keitaro > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org > http://phenix-online.org/mailman/listinfo/cctbxbb > > -- > This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. > Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. > Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. > Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom > > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org > http://phenix-online.org/mailman/listinfo/cctbxbb

8 years, 8 months

Re: [cctbxbb] use_internal_variance in iotbx.merging_statistics

by Keitaro Yamashita

Dear Richard, Thanks a lot. I hope some Phenix developer will make a comment. Best regards, Keitaro 2016-11-01 20:19 GMT+09:00 <richard.gildea(a)diamond.ac.uk>: > Dear Keitaro, > > I've made the change you suggested in merging_statistics.py - it looks like an oversight, which didn't affect xia2 since we are always calculating merging statistics given an scaled but unmerged mtz file, never an XDS or scalepack-format file. > > As to what defaults Phenix uses, that is better left to one of the Phenix developers to comment on. > > Cheers, > > Richard > > Dr Richard Gildea > Data Analysis Scientist > Tel: +441235 77 8078 > > Diamond Light Source Ltd. > Diamond House > Harwell Science & Innovation Campus > Didcot > Oxfordshire > OX11 0DE > > ________________________________________ > From: cctbxbb-bounces(a)phenix-online.org [cctbxbb-bounces(a)phenix-online.org] on behalf of Keitaro Yamashita [k.yamashita(a)spring8.or.jp] > Sent: 01 November 2016 10:41 > To: cctbx mailing list > Subject: Re: [cctbxbb] use_internal_variance in iotbx.merging_statistics > > Dear Richard and everyone, > > Thanks for your reply. What kind of input do you give to > iotbx.merging_statistics in xia2? For example, when XDS file is given, > use_internal_variance=False is not passed to merge_equivalents() > function. Please look at the lines of > filter_intensities_by_sigma.__init__() in iotbx/merging_statistics.py. > When sigma_filtering == "xds" or sigma_filtering == "scalepack", > array_merged is recalculated using merge_equivalents() with default > arguments. > > If nobody disagrees, I would like to commit the fix so that > use_internal_variance variable is passed to all merge_equivalents() > function calls. > > > I am afraid that the behavior in the phenix-1.11 would be confusing. > In phenix.table_one (mmtbx/command_line/table_one.py), > use_internal_variance=False is default. This will be OK with the fix > I suggested above. > > Can it also be default in phenix.merging_statistics, not to change the > program behavior through phenix versions? > > > Best regards, > Keitaro > > 2016-11-01 18:21 GMT+09:00 <richard.gildea(a)diamond.ac.uk>: >> Dear Keitaro, >> >> iotbx.merging_statistics does have the option to change the parameter use_internal_variance. In xia2 we use the defaults use_internal_variance=False, eliminate_sys_absent=False, n_bins=20, when calculating merging statistics which give comparable results to those calculate by Aimless: >> >> $ iotbx.merging_statistics >> Usage: >> phenix.merging_statistics [data_file] [options...] >> >> Calculate merging statistics for non-unique data, including R-merge, R-meas, >> R-pim, and redundancy. Any format supported by Phenix is allowed, including >> MTZ, unmerged Scalepack, or XDS/XSCALE (and possibly others). Data should >> already be on a common scale, but with individual observations unmerged. >> Diederichs K & Karplus PA (1997) Nature Structural Biology 4:269-275 >> (with erratum in: Nat Struct Biol 1997 Jul;4(7):592) >> Weiss MS (2001) J Appl Cryst 34:130-135. >> Karplus PA & Diederichs K (2012) Science 336:1030-3. >> >> >> Full parameters: >> >> file_name = None >> labels = None >> space_group = None >> unit_cell = None >> symmetry_file = None >> high_resolution = None >> low_resolution = None >> n_bins = 10 >> extend_d_max_min = False >> anomalous = False >> sigma_filtering = *auto xds scala scalepack >> .help = "Determines how data are filtered by SigmaI and I/SigmaI. XDS" >> "discards reflections whose intensity after merging is less than" >> "-3*sigma, Scalepack uses the same cutoff before merging, and" >> "SCALA does not do any filtering. Reflections with negative SigmaI" >> "will always be discarded." >> use_internal_variance = True >> eliminate_sys_absent = True >> debug = False >> loggraph = False >> estimate_cutoffs = False >> job_title = None >> .help = "Job title in PHENIX GUI, not used on command line" >> >> >> Below is my email to Pavel and Billy when we discussed this issue by email a while back: >> >> The difference between use_internal_variance=True/False is explained in Luc's document here: >> >> libtbx.pdflatex $(libtbx.find_in_repositories cctbx/miller)/equivalent_reflection_merging.tex >> >> Essentially use_internal_variance=False uses only the unmerged sigmas to compute the merged sigmas, whereas use_internal_variance=True uses instead the spread of the unmerged intensities to compute the merged sigmas. Furthermore, use_internal_variance=True uses the largest of the variance coming from the spread of the intensities and that computed from the unmerged sigmas. As a result, use_internal_variance=True can only ever give lower I/sigI than use_internal_variance=False. The relevant code in the cctbx is here: >> >> https://sourceforge.net/p/cctbx/code/HEAD/tree/trunk/cctbx/miller/merge_equ… >> >> Aimless has a similar option for the SDCORRECTION keyword, if you set the option SAMPLESD, which I think is equivalent to use_internal_variance=True. The default behaviour of Aimless is equivalent to use_internal_variance=False: >> >> http://www.mrc-lmb.cam.ac.uk/harry/pre/aimless.html#sdcorrection >> >> "SAMPLESD is intended for very high multiplicity data such as XFEL serial data. The final SDs are estimated from the weighted population variance, assuming that the input sigma(I)^2 values are proportional to the true errors. This probably gives a more realistic estimate of the error in <I>. In this case refinement of the corrections is switched off unless explicitly requested." >> >> I think that the "external" variance is probably better if the sigmas from the scaling program are reliable, or for low multiplicity data. For high multiplicity data or if the sigmas from the scaling program are not reliable, then "internal" variance is probably better. >> >> Cheers, >> >> Richard >> >> Dr Richard Gildea >> Data Analysis Scientist >> Tel: +441235 77 8078 >> >> Diamond Light Source Ltd. >> Diamond House >> Harwell Science & Innovation Campus >> Didcot >> Oxfordshire >> OX11 0DE >> >> ________________________________________ >> From: cctbxbb-bounces(a)phenix-online.org [cctbxbb-bounces(a)phenix-online.org] on behalf of Keitaro Yamashita [k.yamashita(a)spring8.or.jp] >> Sent: 01 November 2016 07:23 >> To: cctbx mailing list >> Subject: [cctbxbb] use_internal_variance in iotbx.merging_statistics >> >> Dear Phenix/CCTBX developers, >> >> iotbx/merging_statistics.py is used by phenix.merging_statistics, >> phenix.table_one, and so on. By upgrading phenix from 1.10.1 to 1.11, >> merging statistics-related codes were significantly changed. >> >> Previously, miller.array.merge_equivalents() was always called with >> argument use_internal_variance=False, which is consistent with XDS, >> Aimless and so on. Currently, use_internal_variance=True is default, >> and cannot be changed by users (see below). >> >> These changes were made by @afonine and @rjgildea in rev. 22973 (Sep >> 26, 2015) and 23961 (Mar 8, 2016). Could anyone explain why these >> changes were introduced? >> >> https://sourceforge.net/p/cctbx/code/22973 >> https://sourceforge.net/p/cctbx/code/23961 >> >> >> My points are: >> >> - We actually cannot control use_internal_variance= parameter because >> it is not passed to merge_equivalents() in class >> filter_intensities_by_sigma. >> >> - In previous versions, if I gave XDS output to >> phenix.merging_statistics, <I/sigma> values calculated in the same way >> (as XDS does) were shown; but not in the current version. >> >> - For (for example) phenix.table_one users who expect this behavior, >> it can give inconsistency. The statistics would not be consistent with >> the data used in refinement. >> >> >> cf. the related discussion in cctbxbb: >> http://phenix-online.org/pipermail/cctbxbb/2012-October/000611.html >> >> >> Best regards, >> Keitaro >> _______________________________________________ >> cctbxbb mailing list >> cctbxbb(a)phenix-online.org >> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> -- >> This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. >> Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. >> Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. >> Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom >> >> >> _______________________________________________ >> cctbxbb mailing list >> cctbxbb(a)phenix-online.org >> http://phenix-online.org/mailman/listinfo/cctbxbb > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org > http://phenix-online.org/mailman/listinfo/cctbxbb > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org > http://phenix-online.org/mailman/listinfo/cctbxbb

8 years, 8 months

Re: [phenixbb] help with phenix.ligand_identification

by Edward A. Berry

Ligandfit itself has more helpful error message: Sorry the string 'S2063' cannot be interpreted as a residue number? (Duh!) and does actually place the ligand in the requested blob with "search_center=" So I can run ligandfit in a foreach loop with each of the 180 ligands left by ligand_identification ================= On 09/26/2017 03:32 PM, Edward A. Berry wrote: > I'm having some problems using ligand_identification. > I would like to restrict the search to a specific density peak, even if it is not the highest unmodeled peak or the highest Fo-Fc peak in the map. > I tried using options: > search_center="67.5 18.3 11.8" > or > ligand_near_res=S2063, > Will the search be restricted to that region, or if a particuar ligand doesn't fit that blob, > will it search through the rest of the map? If it is restricted, in what radius? > Is this radius affected by the "search_dist" or "local_search" parameters? > and local_search = True is default? > Is there a threshold level for density level, below which building a ligand in a blob will not be attempted? > > I've tried with both search_center= and ligand_near_res=, and something gets > built far away from that site. But there were errors, so I may have something wrong: > > > phenix.ligand_identification mtz_in=sqr2803or13_031.mtz input_labels="2FOFCWT PH2FOFCWT" \ > model=sqr2803or13_031.pdb ligand_near_res=S2063 nproc=2 > > After preparing the ligand library, then: > Running LigandFit process 1... > > Number of atoms in ligand suc.pdb is 23 > Running job sequence 1, ligand 2, in /tb/sb/usr20c/berry/ref/sqrdep/sqr2803dep2/phenix2/Temp_1... > Evaluating all ligands in ligand-lib now...and placing fittedligand ### in resolve_ligand_###.pdb > > Number of atoms in ligand 2pe.pdb is 28 > Running job sequence 0, ligand 1, in /tb/sb/usr20c/berry/ref/sqrdep/sqr2803dep2/phenix2/Temp_0... > Process Process-2: > Traceback (most recent call last): > File "/sw/lnx/phenix-1.12-2829/base/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap > self.run() > File "/sw/lnx/phenix-1.12-2829/base/lib/python2.7/multiprocessing/process.py", line 114, in run > self._target(*self._args, **self._kwargs) > File "/sw/lnx/phenix-1.12-2829/build/../modules/phenix/phenix/command_line/ligand_identification.py", line 1382, in RunLigandFit > shutil.rmtree(ligandfit_dir) > File "/sw/lnx/phenix-1.12-2829/base/lib/python2.7/shutil.py", line 247, in rmtree > rmtree(fullname, ignore_errors, onerror) > File "/sw/lnx/phenix-1.12-2829/base/lib/python2.7/shutil.py", line 256, in rmtree > onerror(os.rmdir, path, sys.exc_info()) > File "/sw/lnx/phenix-1.12-2829/base/lib/python2.7/shutil.py", line 254, in rmtree > os.rmdir(path) > OSError: [Errno 39] Directory not empty: '/tb/sb/usr20c/berry/ref/sqrdep/sqr2803dep2/phenix2/Temp_1/LigandFit_run_1_/TEMP0' > Process Process-1: > Traceback (most recent call last): > File "/sw/lnx/phenix-1.12-2829/base/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap > self.run() > File "/sw/lnx/phenix-1.12-2829/base/lib/python2.7/multiprocessing/process.py", line 114, in run > self._target(*self._args, **self._kwargs) > File "/sw/lnx/phenix-1.12-2829/build/../modules/phenix/phenix/command_line/ligand_identification.py", line 1382, in RunLigandFit > shutil.rmtree(ligandfit_dir) > File "/sw/lnx/phenix-1.12-2829/base/lib/python2.7/shutil.py", line 247, in rmtree > rmtree(fullname, ignore_errors, onerror) > File "/sw/lnx/phenix-1.12-2829/base/lib/python2.7/shutil.py", line 256, in rmtree > onerror(os.rmdir, path, sys.exc_info()) > File "/sw/lnx/phenix-1.12-2829/base/lib/python2.7/shutil.py", line 254, in rmtree > os.rmdir(path) > OSError: [Errno 39] Directory not empty: '/tb/sb/usr20c/berry/ref/sqrdep/sqr2803dep2/phenix2/Temp_0/LigandFit_run_1_/TEMP0' > > Evaluating LigandFit results ... > > The run continues, but it does not test any more ligands after those first two but goes on to evaluate the results. > With nproc = 1, only 1 ligand gets tested. In all cases the first ligand in the library (2PE.pdb) is evaluated as the best. > It is placed in density, but density that has already been built out with (and looks more like) a string of water molecules. > And this is far from the selected residue or coordinates specified. > > The directory that raised the error when attempting to be deleted does eventually get removed: after the run there is no TEMP_N in the parent directory. > > Any suggestions would be welcome. > Ed > > P.S. > - running with .eff file: > > > ['--show_defaults'] > ligand_identification { > mtz_in = sqr2803or13_031.mtz > mtz_type = *F diffmap > model = sqr2803or13_031.pdb > ncpu = 1 > n_indiv_tries_min = 30 > n_indiv_tries_max = 300 > n_group_search = 4 > search_dist = 10 > local_search = True > search_center = "67.5 18.3 11.8" > # ligand_near_res = S2063 > verbose = False > debug = False > use_ligandfit = True > search_mode = *default LigandFit > temp_dir = Auto > dry_run = False > # number_of_ligands = 1 > cc_min = 0.75 > open_in_coot = False > non_bonded = True > keep_all_files = False > # cif_def_file_list = > real_space_target_weight = 10 > # job_title = None > ligandfit { > } > } > > gives: > > [['2pe.pdb', 'suc.pdb', . . . 'upl.pdb']] > > /tb/sb/usr20c/berry/ref/sqrdep/sqr2803dep2/phenix2 > > ******************************************************************************* > > Sorry, the protein model file None does not seem to exist? > > ******************************************************************************* > > > Running LigandFit process 0... > > Process Process-1: > Traceback (most recent call last): > File "/sw/lnx/phenix-1.12-2829/base/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap > self.run() > File "/sw/lnx/phenix-1.12-2829/base/lib/python2.7/multiprocessing/process.py", line 114, in run > self._target(*self._args, **self._kwargs) > File "/sw/lnx/phenix-1.12-2829/build/../modules/phenix/phenix/command_line/ligand_identification.py", line 1122, in RunLigandFit > shutil.copyfile(mtz_in,data_local) > File "/sw/lnx/phenix-1.12-2829/base/lib/python2.7/shutil.py", line 82, in copyfile > with open(src, 'rb') as fsrc: > IOError: [Errno 2] No such file or directory: 'None' > > Evaluating LigandFit results ... > > Lig_seq Placed/total cc_all cc cc_adj score Code HBscore > > Cannot find overall_ligand_scores.log0. This could mean that none of that (sub)set of ligand fitted well. > > > > None of the ligand fit the difference desity well enough. Please try the following -- > 1) if you input a custom library, try to use the default library (no extra keywords needed), or > 2) if you used the default library already, ususlly this means that the density is too small (> 6 atome or more is needed.) > Exiting ...... > > > > No good ligand found. > > > _______________________________________________ > phenixbb mailing list > phenixbb(a)phenix-online.org > http://phenix-online.org/mailman/listinfo/phenixbb > Unsubscribe: phenixbb-leave(a)phenix-online.org >

7 years, 9 months

Re: [cctbxbb] some thoughts on cctbx and pip

by Luc Bourhis

Hi Graeme, Yes, I know. But “black" is a program doing a very particular task (code formatting from the top of my head). Requiring to use a wrapper for python itself is another level. But ok, I think I am mellowing to the idea after all! Talking with people around me, and extrapolating, I would bet that, right now, a great majority of people interested by cctbx in pip have already used the cctbx, so they know about the Python wrapper, and they would not be too sanguine about that. My concern is for the future, when pip will be the first time some people use cctbx. Big fat warning notices on PyPI page and a better error message when cctbx fails because LIBTBX_BUILD is not set would be needed but that could be all right. If we do a pip installer, we should aim at a minimal install: cctbx, iotbx and their dependencies, and that’s it. Best wishes, Luc > On 23 Aug 2019, at 07:17, Graeme.Winter(a)Diamond.ac.uk <Graeme.Winter(a)diamond.ac.uk> wrote: > > Without discussing the merits of this or whether we _choose_ to make the move to supporting PIP, I am certain it would be _possible_ - many other packages make dispatcher scripts when you pip install them e.g. > > Silver-Surfer rescale_f2 :) $ which black; cat $(which black) > /Library/Frameworks/Python.framework/Versions/3.6/bin/black > #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 > > # -*- coding: utf-8 -*- > import re > import sys > > from black import main > > if __name__ == '__main__': > sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) > sys.exit(main()) > > So we _could_ work around the absence of LIBTBX_BUILD etc. in the system. Whether or not we elect to do the work is a different question, and it seems clear that here are very mixed opinions on this. > > Best wishes Graeme > > > On 23 Aug 2019, at 01:21, Luc Bourhis <luc_j_bourhis(a)mac.com<mailto:[email protected]>> wrote: > > Hi, > > Even if we managed to ship our the boost dynamic libraries with pip, it would still not be pip-like, as we would still need our python wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal pip packages work with the standard python exe. LD_LIBRARY_PATH, we could get around that by changing the way we compile, using -Wl,-R, which is the runtime equivalent of build time -L. That’s a significant change that would need to be tested. But there is no way around setting LIBTBX_BUILD right now. Leaving that to the user is horrible. Perhaps there is a way to hack libtbx/env_config.py so that we can hardwire LIBTBX_BUILD in there when pip installs? > > Best wishes, > > Luc > > > On 16 Aug 2019, at 22:47, Luc Bourhis <luc_j_bourhis(a)mac.com<mailto:[email protected]>> wrote: > > Hi, > > I did look into that many years ago, and even toyed with building a pip installer. What stopped me is the exact conclusion you reached too: the user would not have the pip experience he expects. You are right that it is a lot of effort but is it worth it? Considering that remark, I don’t think so. Now, Conda was created specifically to go beyond pip pure-python-only support. Since cctbx has garnered support for Conda, the best avenue imho is to go the extra length to have a package on Anaconda.org<http://anaconda.org/>, and then to advertise it hard to every potential user out there. > > Best wishes, > > Luc > > > On 16 Aug 2019, at 21:45, Aaron Brewster <asbrewster(a)lbl.gov<mailto:[email protected]>> wrote: > > Hi, to avoid clouding Dorothee's documentation email thread, which I think is a highly useful enterprise, here's some thoughts about putting cctbx into pip. Pip doesn't install non-python dependencies well. I don't think boost is available as a package on pip (at least the package version we use). wxPython4 isn't portable through pip (https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython…). MPI libraries are system dependent. If cctbx were a pure python package, pip would be fine, but cctbx is not. > > All that said, we could build a manylinux1 version of cctbx and upload it to PyPi (I'm just learning about this). For a pip package to be portable (which is a requirement for cctbx), it needs to conform to PEP513, the manylinux1 standard (https://www.python.org/dev/peps/pep-0513/). For example, numpy is built according to this standard (see https://pypi.org/project/numpy/#files, where you'll see the manylinux1 wheel). Note, the manylinux1 standard is built with Centos 5.11 which we no longer support. > > There is also a manylinux2010 standard, which is based on Centos 6 (https://www.python.org/dev/peps/pep-0571/). This is likely a more attainable target (note though by default C++11 is not supported on Centos 6). > > If we built a manylinuxX version of cctbx and uploaded it to PyPi, the user would need all the non-python dependencies. There's no way to specify these in pip. For example, cctbx requires boost 1.63 or better. The user will need to have it in a place their python can find it, or we could package it ourselves and supply it, similar to how the pip h5py package now comes with an hd5f library, or how the pip numpy package includes an openblas library. We'd have to do the same for any packages we depend on that aren't on pip using the manylinux standards, such as wxPython4. > > Further, we need to think about how dials and other cctbx-based packages interact. If pip install cctbx is set up, how does pip install dials work, such that any dials shared libraries can find the cctbx libraries? Can shared libraries from one pip package link against libraries in another pip package? Would each package need to supply its own boost? Possibly this is well understood in the pip field, but not by me :) > > Finally, there's the option of providing a source pip package. This would require the full compiler toolchain for any given platform (macOS, linux, windows). These are likely available for developers, but not for general users. > > Anyway, these are some of the obstacles. Not saying it isn't possible, it's just a lot of effort. > > Thanks, > -Aaron > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org<mailto:[email protected]> > http://phenix-online.org/mailman/listinfo/cctbxbb > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org<mailto:[email protected]> > http://phenix-online.org/mailman/listinfo/cctbxbb > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org<mailto:[email protected]> > http://phenix-online.org/mailman/listinfo/cctbxbb > > > -- > This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. > Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. > Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. > Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org > http://phenix-online.org/mailman/listinfo/cctbxbb

5 years, 10 months

Re: [phenixbb] Geometry Restraints - Anisotropic truncation

by Frank von Delft

Is the explanation not simpler? The volumes of reciprocal space that were left out did not in fact contain signal, and it's by removing those noise non-reflections that the actual R is revealed. As James Holton posted a while ago, Rfactors calculated for noise give randomly large values. So it seems less less misleading to refer to it as "anisotropy-TRUNCATED". phx. On 03/05/2012 07:40, Pavel Afonine wrote: > Hi Kendall, > > I just did this quick test: calculated R-factors using original and > anisotropy-corrected Mike Sawaya's data (*) > > Original: > r_work : 0.3026 > r_free : 0.3591 > number of reflections: 26944 > > Truncated: > r_work : 0.2640 > r_free : 0.3178 > number of reflections: 18176 > > The difference in R-factors is not too surprising given how many > reflections was removed (about 33%). > > Pavel > > (*) Note, the data available in PDB is anisotropy corrected. The > original data set was kindly provided to me by the author. > > > On 5/2/12 5:25 AM, Kendall Nettles wrote: >> I didnt think the structure was publishable with Rfree of 33% >> because I was expecting the reviewers to complain. >> >> We have tested a number of data sets on the UCLA server and it >> usually doesn't make much difference. I wouldn't expect truncation >> alone to change Rfree by 5%, and it usually doesn't. The two times I >> have seen dramatic impacts on the maps ( and Rfree ), the highly >> anisotrophic sets showed strong waves of difference density as well, >> which was fixed by throwing out the noise. We have moved to using >> loose data cutoffs for most structures, but I do think anisotropic >> truncation can be helpful in rare cases. >> >> Kendall >> >> On May 1, 2012, at 3:07 PM, "Dale >> Tronrud"<det102(a)uoxray.uoregon.edu> wrote: >> >>> While philosophically I see no difference between a spherical >>> resolution >>> cutoff and an elliptical one, a drop in the free R can't be the >>> justification >>> for the switch. A model cannot be made more "publishable" simply by >>> discarding >>> data. >>> >>> We have a whole bunch of empirical guides for judging the >>> quality of this >>> and that in our field. We determine the resolution limit of a data >>> set (and >>> imposing a "limit" is another empirical choice made) based on Rmrg, >>> or Rmes, >>> or Rpim getting too big or I/sigI getting too small and there is no >>> agreement >>> on how "too big/small" is too "too big/small". >>> >>> We then have other empirical guides for judging the quality of >>> the models >>> we produce (e.g. Rwork, Rfree, rmsds of various sorts). Most people >>> seem to >>> recognize that the these criteria need to be applied differently for >>> different >>> resolutions. A lower resolution model is allowed a higher Rfree, for >>> example. >>> >>> Isn't is also true that a model refined to data with a cutoff of >>> I/sigI of >>> 1 would be expected to have a free R higher than a model refined to >>> data with >>> a cutoff of 2? Surely we cannot say that the decrease in free R >>> that results >>> from changing the cutoff criteria from 1 to 2 reflects an improved >>> model. It >>> is the same model after all. >>> >>> Sometimes this shifting application of empirical criteria >>> enhances the >>> adoption of new technology. Certainly the TLS parametrization of >>> atomic >>> motion has been widely accepted because it results in lower working >>> and free >>> Rs. I've seen it knock 3 to 5 percent off, and while that certainly >>> means >>> that the model fits the data better, I'm not sure that the quality >>> of the >>> hydrogen bond distances, van der Waals distances, or maps are any >>> better. >>> The latter details are what I really look for in a model. >>> >>> On the other hand, there has been good evidence through the >>> years that >>> there is useful information in the data beyond an I/sigI of 2 or an >>> Rmeas> 100% but getting people to use this data has been a hard >>> slog. The >>> reason for this reluctance is that the R values of the resulting models >>> are higher. Of course they are higher! That does not mean the models >>> are of poorer quality, only that data with lower signal/noise has been >>> used that was discarded in the models you used to develop your "gut >>> feeling" >>> for the meaning of R. >>> >>> When you change your criteria for selecting data you have to >>> discard >>> your old notions about the acceptable values of empirical quality >>> measures. >>> You either have to normalize your measure, as Phil Jeffrey >>> recommends, by >>> ensuring that you calculate your R's with the same reflections, or by >>> making objective measures of map quality. >>> >>> Dale Tronrud >>> >>> P.S. It is entirely possible that refining a model to a very optimistic >>> resolution cutoff and calculating the map to a lower resolution >>> might be >>> better than throwing out the data altogether. >>> >>> On 5/1/2012 10:34 AM, Kendall Nettles wrote: >>>> I have seen dramatic improvements in maps and behavior during >>>> refinement following use of the UCLA anisotropy server in two >>>> different cases. For one of them the Rfree went from 33% to 28%. I >>>> don't think it would have been publishable otherwise. >>>> Kendall >>>> >>>> On May 1, 2012, at 11:10 AM, Bryan Lepore wrote: >>>> >>>>> On Mon, Apr 30, 2012 at 4:22 AM, Phil >>>>> Evans<pre(a)mrc-lmb.cam.ac.uk> wrote: >>>>>> Are anisotropic cutoff desirable? >>>>> is there a peer-reviewed publication - perhaps from Acta >>>>> Crystallographica - which describes precisely why scaling or >>>>> refinement programs are inadequate to ameliorate the problem of >>>>> anisotropy, and argues why the method applied in Strong, et. al. 2006 >>>>> satisfies this need? >>>>> >>>>> -Bryan > > _______________________________________________ > phenixbb mailing list > phenixbb(a)phenix-online.org > http://phenix-online.org/mailman/listinfo/phenixbb

13 years, 2 months

Re: [phenixbb] Geometry Restraints - Anisotropic truncation

by Pavel Afonine

Hi Kendall, removing same amount of data randomly gives Rwork/Rfree ~ 30/35%. Pavel On 5/3/12 4:13 AM, Kendall Nettles wrote: > Hi Pavel, > What happens if you throw out that many reflections that have signal? Can you take out a random set of the same size? > Best, > Kendall > > On May 3, 2012, at 2:41 AM, "Pavel Afonine"<pafonine(a)lbl.gov> wrote: > >> Hi Kendall, >> >> I just did this quick test: calculated R-factors using original and >> anisotropy-corrected Mike Sawaya's data (*) >> >> Original: >> r_work : 0.3026 >> r_free : 0.3591 >> number of reflections: 26944 >> >> Truncated: >> r_work : 0.2640 >> r_free : 0.3178 >> number of reflections: 18176 >> >> The difference in R-factors is not too surprising given how many >> reflections was removed (about 33%). >> >> Pavel >> >> (*) Note, the data available in PDB is anisotropy corrected. The >> original data set was kindly provided to me by the author. >> >> >> On 5/2/12 5:25 AM, Kendall Nettles wrote: >>> I didnt think the structure was publishable with Rfree of 33% because I was expecting the reviewers to complain. >>> >>> We have tested a number of data sets on the UCLA server and it usually doesn't make much difference. I wouldn't expect truncation alone to change Rfree by 5%, and it usually doesn't. The two times I have seen dramatic impacts on the maps ( and Rfree ), the highly anisotrophic sets showed strong waves of difference density as well, which was fixed by throwing out the noise. We have moved to using loose data cutoffs for most structures, but I do think anisotropic truncation can be helpful in rare cases. >>> >>> Kendall >>> >>> On May 1, 2012, at 3:07 PM, "Dale Tronrud"<det102(a)uoxray.uoregon.edu> wrote: >>> >>>> While philosophically I see no difference between a spherical resolution >>>> cutoff and an elliptical one, a drop in the free R can't be the justification >>>> for the switch. A model cannot be made more "publishable" simply by discarding >>>> data. >>>> >>>> We have a whole bunch of empirical guides for judging the quality of this >>>> and that in our field. We determine the resolution limit of a data set (and >>>> imposing a "limit" is another empirical choice made) based on Rmrg, or Rmes, >>>> or Rpim getting too big or I/sigI getting too small and there is no agreement >>>> on how "too big/small" is too "too big/small". >>>> >>>> We then have other empirical guides for judging the quality of the models >>>> we produce (e.g. Rwork, Rfree, rmsds of various sorts). Most people seem to >>>> recognize that the these criteria need to be applied differently for different >>>> resolutions. A lower resolution model is allowed a higher Rfree, for example. >>>> >>>> Isn't is also true that a model refined to data with a cutoff of I/sigI of >>>> 1 would be expected to have a free R higher than a model refined to data with >>>> a cutoff of 2? Surely we cannot say that the decrease in free R that results >>>> from changing the cutoff criteria from 1 to 2 reflects an improved model. It >>>> is the same model after all. >>>> >>>> Sometimes this shifting application of empirical criteria enhances the >>>> adoption of new technology. Certainly the TLS parametrization of atomic >>>> motion has been widely accepted because it results in lower working and free >>>> Rs. I've seen it knock 3 to 5 percent off, and while that certainly means >>>> that the model fits the data better, I'm not sure that the quality of the >>>> hydrogen bond distances, van der Waals distances, or maps are any better. >>>> The latter details are what I really look for in a model. >>>> >>>> On the other hand, there has been good evidence through the years that >>>> there is useful information in the data beyond an I/sigI of 2 or an >>>> Rmeas> 100% but getting people to use this data has been a hard slog. The >>>> reason for this reluctance is that the R values of the resulting models >>>> are higher. Of course they are higher! That does not mean the models >>>> are of poorer quality, only that data with lower signal/noise has been >>>> used that was discarded in the models you used to develop your "gut feeling" >>>> for the meaning of R. >>>> >>>> When you change your criteria for selecting data you have to discard >>>> your old notions about the acceptable values of empirical quality measures. >>>> You either have to normalize your measure, as Phil Jeffrey recommends, by >>>> ensuring that you calculate your R's with the same reflections, or by >>>> making objective measures of map quality. >>>> >>>> Dale Tronrud >>>> >>>> P.S. It is entirely possible that refining a model to a very optimistic >>>> resolution cutoff and calculating the map to a lower resolution might be >>>> better than throwing out the data altogether. >>>> >>>> On 5/1/2012 10:34 AM, Kendall Nettles wrote: >>>>> I have seen dramatic improvements in maps and behavior during refinement following use of the UCLA anisotropy server in two different cases. For one of them the Rfree went from 33% to 28%. I don't think it would have been publishable otherwise. >>>>> Kendall >>>>> >>>>> On May 1, 2012, at 11:10 AM, Bryan Lepore wrote: >>>>> >>>>>> On Mon, Apr 30, 2012 at 4:22 AM, Phil Evans<pre(a)mrc-lmb.cam.ac.uk> wrote: >>>>>>> Are anisotropic cutoff desirable? >>>>>> is there a peer-reviewed publication - perhaps from Acta >>>>>> Crystallographica - which describes precisely why scaling or >>>>>> refinement programs are inadequate to ameliorate the problem of >>>>>> anisotropy, and argues why the method applied in Strong, et. al. 2006 >>>>>> satisfies this need? >>>>>> >>>>>> -Bryan >> _______________________________________________ >> phenixbb mailing list >> phenixbb(a)phenix-online.org >> http://phenix-online.org/mailman/listinfo/phenixbb > _______________________________________________ > phenixbb mailing list > phenixbb(a)phenix-online.org > http://phenix-online.org/mailman/listinfo/phenixbb

13 years, 2 months

[phenixbb] Fwd: Geometry Restraints - Anisotropic truncation

by Kendall Nettles

Hi Tom, Do you think something like this could be used during refinement to identify the "best" resolution limits? If you have an Rsleep set would Rfree be sufficient for this? I imagine collecting data with a ring of noise and then let the optimal resolution be determined during refinement. My understanding of this is that the modern refinement algorithms can handle some noise in the reflections, but maybe this could be a way to optimize how much signal is needed to contribute in a positive fashion? Kendall From: "Terwilliger, Thomas C" <terwilliger(a)lanl.gov<mailto:[email protected]>> Date: May 3, 2012 10:23:29 AM EDT To: PHENIX user mailing list <phenixbb(a)phenix-online.org<mailto:[email protected]>> Subject: Re: [phenixbb] Geometry Restraints - Anisotropic truncation Reply-To: PHENIX user mailing list <phenixbb(a)phenix-online.org<mailto:[email protected]>> Hi Kendall, This could work. You could define a fixed set of test reflections, and never touch these, and never include them in refinement, and always use this fixed set to calculate a free R. Then you could do whatever you want, throw away some work reflections, etc, refine, and evaluate how things are working with the fixed free R set. All the best, Tom T ________________________________________ From: phenixbb-bounces(a)phenix-online.org<mailto:[email protected]> [phenixbb-bounces(a)phenix-online.org<mailto:[email protected]>] on behalf of Kendall Nettles [knettles(a)scripps.edu<mailto:[email protected]>] Sent: Thursday, May 03, 2012 7:05 AM To: PHENIX user mailing list Subject: Re: [phenixbb] Geometry Restraints - Anisotropic truncation Hi Pavel, Could you use a similar approach to figuring out where to cut your data in general? Could you compare the effects of throwing out reflections in different bins, based on I/sigma, for example, and use this to determine what is truly noise? I might predict that as you throw out "noise" reflections you will see a larger drop in Rfree than from throwing out "signal" reflections, which should converge as you approach the "true" resolution. While we don't use I/sigma exclusively, we do to tend towards cutting most of our data sets at the same i/sigma, around 1.5. It would be great if there was a more scientific approach. Best, Kendall On May 3, 2012, at 7:45 AM, Pavel Afonine wrote: Hi Kendall, removing same amount of data randomly gives Rwork/Rfree ~ 30/35%. Pavel On 5/3/12 4:13 AM, Kendall Nettles wrote: Hi Pavel, What happens if you throw out that many reflections that have signal? Can you take out a random set of the same size? Best, Kendall On May 3, 2012, at 2:41 AM, "Pavel Afonine"<pafonine(a)lbl.gov<mailto:[email protected]>> wrote: Hi Kendall, I just did this quick test: calculated R-factors using original and anisotropy-corrected Mike Sawaya's data (*) Original: r_work : 0.3026 r_free : 0.3591 number of reflections: 26944 Truncated: r_work : 0.2640 r_free : 0.3178 number of reflections: 18176 The difference in R-factors is not too surprising given how many reflections was removed (about 33%). Pavel (*) Note, the data available in PDB is anisotropy corrected. The original data set was kindly provided to me by the author. On 5/2/12 5:25 AM, Kendall Nettles wrote: I didnt think the structure was publishable with Rfree of 33% because I was expecting the reviewers to complain. We have tested a number of data sets on the UCLA server and it usually doesn't make much difference. I wouldn't expect truncation alone to change Rfree by 5%, and it usually doesn't. The two times I have seen dramatic impacts on the maps ( and Rfree ), the highly anisotrophic sets showed strong waves of difference density as well, which was fixed by throwing out the noise. We have moved to using loose data cutoffs for most structures, but I do think anisotropic truncation can be helpful in rare cases. Kendall On May 1, 2012, at 3:07 PM, "Dale Tronrud"<det102(a)uoxray.uoregon.edu<mailto:[email protected]>> wrote: While philosophically I see no difference between a spherical resolution cutoff and an elliptical one, a drop in the free R can't be the justification for the switch. A model cannot be made more "publishable" simply by discarding data. We have a whole bunch of empirical guides for judging the quality of this and that in our field. We determine the resolution limit of a data set (and imposing a "limit" is another empirical choice made) based on Rmrg, or Rmes, or Rpim getting too big or I/sigI getting too small and there is no agreement on how "too big/small" is too "too big/small". We then have other empirical guides for judging the quality of the models we produce (e.g. Rwork, Rfree, rmsds of various sorts). Most people seem to recognize that the these criteria need to be applied differently for different resolutions. A lower resolution model is allowed a higher Rfree, for example. Isn't is also true that a model refined to data with a cutoff of I/sigI of 1 would be expected to have a free R higher than a model refined to data with a cutoff of 2? Surely we cannot say that the decrease in free R that results from changing the cutoff criteria from 1 to 2 reflects an improved model. It is the same model after all. Sometimes this shifting application of empirical criteria enhances the adoption of new technology. Certainly the TLS parametrization of atomic motion has been widely accepted because it results in lower working and free Rs. I've seen it knock 3 to 5 percent off, and while that certainly means that the model fits the data better, I'm not sure that the quality of the hydrogen bond distances, van der Waals distances, or maps are any better. The latter details are what I really look for in a model. On the other hand, there has been good evidence through the years that there is useful information in the data beyond an I/sigI of 2 or an Rmeas> 100% but getting people to use this data has been a hard slog. The reason for this reluctance is that the R values of the resulting models are higher. Of course they are higher! That does not mean the models are of poorer quality, only that data with lower signal/noise has been used that was discarded in the models you used to develop your "gut feeling" for the meaning of R. When you change your criteria for selecting data you have to discard your old notions about the acceptable values of empirical quality measures. You either have to normalize your measure, as Phil Jeffrey recommends, by ensuring that you calculate your R's with the same reflections, or by making objective measures of map quality. Dale Tronrud P.S. It is entirely possible that refining a model to a very optimistic resolution cutoff and calculating the map to a lower resolution might be better than throwing out the data altogether. On 5/1/2012 10:34 AM, Kendall Nettles wrote: I have seen dramatic improvements in maps and behavior during refinement following use of the UCLA anisotropy server in two different cases. For one of them the Rfree went from 33% to 28%. I don't think it would have been publishable otherwise. Kendall On May 1, 2012, at 11:10 AM, Bryan Lepore wrote: On Mon, Apr 30, 2012 at 4:22 AM, Phil Evans<pre(a)mrc-lmb.cam.ac.uk<mailto:[email protected]>> wrote: Are anisotropic cutoff desirable? is there a peer-reviewed publication - perhaps from Acta Crystallographica - which describes precisely why scaling or refinement programs are inadequate to ameliorate the problem of anisotropy, and argues why the method applied in Strong, et. al. 2006 satisfies this need? -Bryan _______________________________________________ phenixbb mailing list phenixbb(a)phenix-online.org<mailto:[email protected]> http://phenix-online.org/mailman/listinfo/phenixbb _______________________________________________ phenixbb mailing list phenixbb(a)phenix-online.org<mailto:[email protected]> http://phenix-online.org/mailman/listinfo/phenixbb _______________________________________________ phenixbb mailing list phenixbb(a)phenix-online.org<mailto:[email protected]> http://phenix-online.org/mailman/listinfo/phenixbb _______________________________________________ phenixbb mailing list phenixbb(a)phenix-online.org<mailto:[email protected]> http://phenix-online.org/mailman/listinfo/phenixbb _______________________________________________ phenixbb mailing list phenixbb(a)phenix-online.org<mailto:[email protected]> http://phenix-online.org/mailman/listinfo/phenixbb

13 years, 2 months

Re: [cctbxbb] Boost Python 1.56

by richard.gildea＠diamond.ac.uk

I've made a start on transcribing this document here: https://github.com/cctbx/cctbx_project/wiki/cctbx-Developer-Guidance It probably still needs cleaning up a bit (e.g. I couldn't figure out quickly how to do 3-level list nesting, 1.a.i etc.) and updating to reflect current practice (e.g. git not svn). Cheers, Richard Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE ________________________________________ From: cctbxbb-bounces(a)phenix-online.org [cctbxbb-bounces(a)phenix-online.org] on behalf of Pavel Afonine [pafonine(a)lbl.gov] Sent: 06 April 2017 09:07 To: cctbx mailing list Subject: Re: [cctbxbb] Boost Python 1.56 Hi Graeme, hm.. this is a good question. We've been through back-and-forth iterations of editing this file and I think the latest I have is from Paul. But I can't find a non-PDF version of it. Paul: do you have an editable version of this file? Thanks, Pavel On 4/6/17 13:45, Graeme.Winter(a)diamond.ac.uk wrote: > Hi Pavel > > These all seem sensible > > If you have the original non pdf document it may be easier to transcribe this over.. > > I also note that it lacks the actual detail on how to run tests! However would be happy to add this once on wiki > > Best wishes Graeme > > On 6 Apr 2017, at 04:00, Pavel Afonine <pafonine(a)lbl.gov<mailto:[email protected]>> wrote: > > Not sure if that answers your questions but once upon a time we here at Berkeley tried to write a some sort of document that was supposed to answer questions like this. Attached. By no means it is complete, up-to-date, etc, but it might be worth reading for anyone who contributes to cctbx. (Even not sure if I'm sending the latest version). > Unfortunately, nobody bothered to put it in some central place. > > Pavel > > On 4/6/17 10:51, James Holton wrote: > Hey Billy, > > On a related note. How do I run these regression tests before committing something into Git? Is there a document on dials regression testing I can't find? > > -James > > On Apr 5, 2017, at 3:38 PM, Billy Poon <bkpoon(a)lbl.gov<mailto:[email protected]>> wrote: > > Hi all, > > I tested Boost 1.56 on our buildbot servers and got some new test failures with > > cctbx_project/scitbx/array_family/boost_python/tst_flex.py > cctbx_project/scitbx/random/tests/tst_random.py > > The full log for CentOS 6 can be found at > > http://cci-vm-6.lbl.gov:8010/builders/phenix-nightly-intel-linux-2.6-x86_64… > > It looks like the errors are related to random number generation. For a given seed, would the sequence of numbers change when Boost is changed? I did a diff between Boost 1.56 and the current Boost and could not see any changes that immediately stood out as being related to random numbers. > > Are these tests failing for others as well? > > -- > Billy K. Poon > Research Scientist, Molecular Biophysics and Integrated Bioimaging > Lawrence Berkeley National Laboratory > 1 Cyclotron Road, M/S 33R0345 > Berkeley, CA 94720 > Tel: (510) 486-5709 > Fax: (510) 486-5909 > Web: https://phenix-online.org<https://phenix-online.org/> > > On Wed, Apr 5, 2017 at 8:12 AM, Charles Ballard <charles.ballard(a)stfc.ac.uk<mailto:[email protected]>> wrote: > FYI, we (CCP4) have been using 1.56 for building cctbx/phaser/dials for the last while with no issues. Don't know about 1.60, but 1.59 causes issues with the boost python make_getter and make_setter (initialisation of none const reference if the passed type is a temporary). > > Charles > > On 3 Apr 2017, at 14:31, Luc Bourhis wrote: > > Hi all, > > everybody seemed to agree but then it was proposed to move straight to Boost 1.60, and this caused troubles. Could we consider again to move to at least 1.56? As far as I can tell, this does not cause any issue and as stated one year ago, it would help me and Olex 2. > > Thanks, > > Luc > > On 10 Feb 2016, at 15:17, Nicholas Sauter <nksauter(a)lbl.gov<mailto:[email protected]>> wrote: > > Nigel, Billy & Aaron, > > I completely endorse this move to Boost 1.56. Can we update our build? > > Nick > > Nicholas K. Sauter, Ph. D. > Computer Staff Scientist, Molecular Biophysics and Integrated Bioimaging Division > Lawrence Berkeley National Laboratory > 1 Cyclotron Rd., Bldg. 33R0345 > Berkeley, CA 94720 > (510) 486-5713<tel:%28510%29%20486-5713> > > On Wed, Feb 10, 2016 at 2:41 PM, Luc Bourhis <luc_j_bourhis(a)mac.com<mailto:[email protected]>> wrote: > Hi, > > I have improvements to the smtbx on their way to be committed which require Boost version 1.56. This is related to Boost.Threads, whose support I re-activated a few months ago on Nick’s request. I need the function boost::thread::physical_concurrency which returns the number of physical cores on the machine, as opposed to virtual cores when hyperthreading is enabled (which it is by default on any Intel machine). That function is not available in Boost 1.55 which is the version currently used in the nightly tests: it appeared in 1.56. > > So, would it be possible to move to Boost 1.56? Otherwise, I will need to backport that function. Not too difficult but not thrilling. > > Best wishes, > > Luc > > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org<mailto:[email protected]> > http://phenix-online.org/mailman/listinfo/cctbxbb > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org<mailto:[email protected]> > http://phenix-online.org/mailman/listinfo/cctbxbb > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org<mailto:[email protected]> > http://phenix-online.org/mailman/listinfo/cctbxbb > > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org<mailto:[email protected]> > http://phenix-online.org/mailman/listinfo/cctbxbb > > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org<mailto:[email protected]> > http://phenix-online.org/mailman/listinfo/cctbxbb > > > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org<mailto:[email protected]> > http://phenix-online.org/mailman/listinfo/cctbxbb > > > <cctbx-developer-guidance-08-2015.pdf>_______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org<mailto:[email protected]> > http://phenix-online.org/mailman/listinfo/cctbxbb > > _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org http://phenix-online.org/mailman/listinfo/cctbxbb -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

8 years, 2 months

Search results for query "look through"