- cctbxbb - phenix-online.org

Failing tests on python3 builds
by richard.gildea＠diamond.ac.uk 30 Jul '19

30 Jul '19

Hi, Whilst looking into test failures reported against a pull request I currently have open, I noticed that there are several (9) failing tests on the Python 3 build. It appears that Markus reported these 3 weeks ago - is anyone looking into these failures? Having builds permanently reporting failure isn't particularly helpful as it discourages people from looking at the results, and will likely lead to further Python 3 regressions being introduced into the codebase. https://dev.azure.com/cctbx/cctbx_project/_build/results?buildId=240 https://github.com/cctbx/cctbx_project/issues Cheers, Richard Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 4

"duplicate" cif categories
by wtempel 06 Jul '19

06 Jul '19

Hi, wwpdb onedep recently rejected a file that I had manipulated (but not validated, I admit) in IOTBX. The error was "duplicate categories", or a similar complaint. I had added _reflns.* items related to data processing to a phenix.refine cif file that already included a _reflns.B_iso_Wilson_estimate. Merely running sort() on the relevant block made the error go away, but who knows what side effects there may have been. Does the CCTBX ecosystem provide some higher-level management of cif categories? Thanks. Wolfram

1 0

align density maps
by Peter Schmidtke 02 Jul '19

02 Jul '19

2 3

pickling nan values
by Robert Oeffner 13 Jun '19

13 Jun '19

Hi all, It seems that a flex.double array containing "nan" values doesn't survive pickling as in: oeffner@grove:~$ cctbx.python Python 2.7.15 (default, Sep 11 2018, 23:48:45) [GCC 4.4.7 20120313 (Red Hat 4.4.7-18)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> >>> from cctbx.array_family import flex >>> import pickle >>> flex_nans = flex.double(3, float('nan')) >>> list(flex_nans) [nan, nan, nan] >>> list(pickle.loads( pickle.dumps(flex_nans))) [0.0, 0.0, 0.0] >>> >>> If the flex double is cast into a python list then it does survive pickling as in: >>> >>> list(pickle.loads( pickle.dumps(list( flex_nans )))) [nan, nan, nan] >>> I would like to use nan values to indicate missing or invalid data values. Would anyone have a suggestion how to go about this? Many thanks, Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus The Keith Peters Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234

2 1

Fwd: Conda/DIALS 1.14.5 source build with --use-conda does not work
by Nicholas Devenish 28 May '19

28 May '19

(Forwarding this manually as apparently my Diamond address isn't on cctbxbb) Hi All, On 24 May 2019, at 14:12, David Waterman - UKRI STFC < david.waterman(a)stfc.ac.uk<mailto:[email protected]>> wrote: > In the process, I discovered that the source builds we point to from > https://dials.github.io/installation.html don't work with --use-conda > because the paths to various packages are out of date. This seems to be to be a fundamental problem with the conda implementation? Let’s try to reproduce an old build from, say, 1st march: the point in cctbx_project at which dials 1.14 split off. This is https://github.com/cctbx/cctbx_project/commit/35652e06564da7ff989500a6d929b… . Try reproducing this point: git clone git(a)github.com<mailto:[email protected]>:cctbx/cctbx_project.git modules/cctbx_project ( cd modules/cctbx_project/ && git checkout 35652e06564da7ff989500a6d929bc595f69040f ) ln -s modules/cctbx_project/libtbx/auto_build/bootstrap.py . ./bootstrap.py base --use-conda —> many "An HTTP error occurred when trying to retrieve this URL.” This means that with the current conda implementation it’s impossible to guarantee reproducibility for any build beyond the past few days, without lots and lots of manual work reconstructing the versioning out of the file list dump. Because the version constraints aren’t kept with the repository, this means that a source distribution is not nearly enough to recreate any given release. This seems like a problem? Nick

3 2

Python3 / scons / non-conda builds
by Graeme.Winter＠Diamond.ac.uk 13 May '19

13 May '19

Dear All, Back in the mists of time, one could build cctbx modules without bootstrap or conda by having the right modules available yourself - a so-called baseless install which I think is still important in X-fel land? Anyhow, trying this with Python3 suggests that we have some special constraints which are perhaps too tight regarding module locations - for example scons - Grey-Area p3_compile :( $ pip3 install scons Collecting scons Downloading https://files.pythonhosted.org/packages/65/41/ac58638421414acc64e26fc120c81… (831kB) 100% |████████████████████████████████| 839kB 14.3MB/s Installing collected packages: scons Successfully installed scons-3.0.5 You are using pip version 19.0.1, however version 19.1.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command. Grey-Area p3_compile :) $ make ./bin/libtbx.scons -j "`./bin/libtbx.show_number_of_processors`" ModuleNotFoundError: No module named 'SCons' During handling of the above exception, another exception occurred: Sorry: SCons is not available. A possible solution is to unpack a SCons distribution in one of these directories: "/Users/graeme/svn/scr" "/Users/graeme/svn/scr/cctbx_project" SCons distributions are available at this location: http://www.scons.org/ It may be necessary to rename the unpacked distribution, e.g.: mv scons-0.96.1 scons make: *** [default] Error 1 Grey-Area p3_compile :( $ which scons /Library/Frameworks/Python.framework/Versions/3.7/bin/scons Should we require that scons is installed as part of cctbx when it is already installed on the system? If we *do* have to have this constraint, maybe that should be checked at configure time (where it complains about six and future) Thoughts? In other news - the syntax changing branch referenced in https://github.com/cctbx/cctbx_project/pull/325 now compiles correctly on Python3 (note: not run, just compile but this means that it’s purpose is done) for the modules which it intends to touch and has no conflicts right now… Best wishes Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 2

Overloading Python builtin names with local variables
by Graeme.Winter＠Diamond.ac.uk 03 May '19

03 May '19

Morning all, As a side-effect of looking at some of the Python3 refactoring I stumbled across a few places where e.g. range is used as a variable name. While this is fine and legal, when moving the code to Python3 xrange => range and from builtins import range is a back-port of Python3 range to Python2 which allows the existing behaviour of xrange generator to be maintained This is fine except for where range is a variable name and we then get a “variable referenced before assignment” type error As a general statement I would say overloading Python reserved names with local variables is untidy at best, and can easily cause confusion (as well as real problems as identified above) - I have therefore taken the liberty of adding a tool to libtbx which can be used to find such variables to give people a chance to avoid them: Grey-Area dxtbx :) [master] $ libtbx.find_reserved_names format/ Checking format/ format/FormatSMVRigakuSaturn.py:239 "format" format = self._scan_factory.format("SMV") format/FormatTIFFRayonixESRF.py:36 "format" format = {LITTLE_ENDIAN: "<", BIG_ENDIAN: ">"}[order] format/FormatTIFFRayonixESRF.py:29 "bytes" width, height, depth, order, bytes = FormatTIFFRayonix.get_tiff_header( format/FormatCBFMini.py:231 "format" format = self._scan_factory.format("CBF") format/FormatTIFFRayonix.py:176 "format" format = self._scan_factory.format("TIFF") format/FormatTIFFRayonix.py:31 "bytes" width, height, depth, order, bytes = FormatTIFF.get_tiff_header(image_file) format/FormatTIFFRayonix.py:74 "bytes" width, height, depth, order, bytes = FormatTIFF.get_tiff_header(image_file) format/FormatSMVRigakuEiger.py:243 "format" format = self._scan_factory.format("SMV") format/FormatSMVADSC.py:212 "format" format = self._scan_factory.format("SMV") format/Registry.py:88 "format" for format in sorted(self._formats, key=lambda x: x.__name__): format/FormatSMVJHSim.py:141 "format" format = self._scan_factory.format("SMV") format/FormatSMVRigakuPilatus.py:188 "format" format = self._scan_factory.format("SMV") format/FormatSMVADSCSN928.py:47 "format" format = self._scan_factory.format("SMV") format/FormatRAXISIVSpring8.py:134 "format" format = self._scan_factory.format("RAXIS") format/FormatTIFFRayonixSPring8.py:31 "bytes" width, height, depth, order, bytes = FormatTIFFRayonix.get_tiff_header( format/FormatSMVADSCmlfsom.py:37 "format" format = self._scan_factory.format("SMV") format/FormatTIFFBruker.py:174 "format" format = self._scan_factory.format("TIFF") format/FormatTIFFBruker.py:31 "bytes" width, height, depth, order, bytes = FormatTIFF.get_tiff_header(image_file) format/FormatTIFFBruker.py:71 "bytes" width, height, depth, order, bytes = FormatTIFF.get_tiff_header(image_file) format/FormatCBFMiniADSCHF4M.py:26 "format" for format in ["%a_%b_%d_%H:%M:%S_%Y"]: format/FormatCBFMiniADSCHF4M.py:169 "format" format = self._scan_factory.format("CBF") format/FormatSMVADSCNoDateStamp.py:54 "format" format = self._scan_factory.format("SMV") format/FormatSMVRigakuSaturnNoTS.py:228 "format" format = self._scan_factory.format("SMV") format/FormatPYunspecified.py:152 "format" format = "" format/FormatSMVRigakuA200.py:234 "format" format = self._scan_factory.format("SMV") format/FormatCBFMiniEigerPhotonFactory.py:131 "format" format = self._scan_factory.format("CBF") format/FormatSMVTimePix_SU.py:97 "format" format = self._scan_factory.format("SMV") format/FormatCBFMiniPilatus.py:83 "format" format = self._scan_factory.format("CBF") format/FormatCBFMiniPilatusHelpers.py:22 "format" for format in ["%Y-%b-%dT%H:%M:%S", "%Y-%m-%dT%H:%M:%S", "%Y/%b/%d %H:%M:%S"]: format/FormatSMVADSCSNAPSID19.py:91 "format" format = self._scan_factory.format("SMV") format/FormatCBFMiniEiger.py:216 "format" format = self._scan_factory.format("CBF") format/FormatSMVNOIR.py:218 "format" format = self._scan_factory.format("SMV") format/FormatSMVCMOS1.py:197 "format" format = self._scan_factory.format("SMV") format/FormatRAXIS.py:238 "format" format = self._scan_factory.format("RAXIS") is an example where “format” is scattered all over the place but is also a useful function - replacing with fmt will mean the same and avoid this overloading, as an example. This uses AST parsing to work through the code and find variable assignments - doing this for dials found lots of examples of “file” and “type” being used. How much people care about this is a local issue, but I thought the tool would be useful All the best Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

modulo operation on flex.double
by Robert Oeffner 11 Apr '19

11 Apr '19

Hi, I have a flex.double array of phase values between -360 and 360 degrees I would like to cast them all in one go to lie within 0 to 360. Does anybody know if there is a nice way of doing this with flex arrays? The modulo operator doesn't seem to be supported for flex arrays unlike the +, -, * and the / operators. This would otherwise be handy as I could then convert the array in one line of code like phases = phases % 360.0 If not I guess I'll have to numpy the problem :-) Many thanks, Rob

5 5

Re: [cctbxbb] cannot compile Phenix
by R. D. Oeffner 06 Apr '19

06 Apr '19

>From the messages on the cctbx github mailing list it seems that Dials people have been extricating dxtbx from cctbx_project. Rob -----Original Message----- From: Pavel Afonine Sent: Saturday, April 6, 2019 7:07 PM To: phenixdev Subject: cannot compile Phenix Hi, anyone knows why current Phenix code is broken: python bootstrap.py --builder=phenix --sfuser=afonine --cciuser=afonine update python bootstrap.py --builder=phenix --sfuser=afonine --cciuser=afonine build --nproc=12 leads to (...) AttributeError: 'empty' object has no attribute 'dxtbx_include': File "/Users/pafonine/phenix/build/SConstruct", line 31: SConscript("spotfinder/SConscript") File "/Users/pafonine/phenix/modules/scons/engine/SCons/Script/SConscript.py", line 607: return method(*args, **kw) File "/Users/pafonine/phenix/modules/scons/engine/SCons/Script/SConscript.py", line 544: return _SConscript(self.fs, *files, **subst_kw) File "/Users/pafonine/phenix/modules/scons/engine/SCons/Script/SConscript.py", line 253: call_stack[-1].globals) File "/Users/pafonine/phenix/modules/cctbx_project/spotfinder/SConscript", line 22: env_etc.dxtbx_include, usr+sys time: 1.94 seconds wall clock time: 4.51 seconds Process failed with return code 2 Any idea for a workaround? Pavel --- This email has been checked for viruses by AVG. https://www.avg.com

2 3

jupyter notebook
by wtempel 02 Apr '19

02 Apr '19

Hi all, how does one configure, and run inside a jupyter notebook, a cctbx-enabled kernel, beginning with a downloaded [cctbx build]( http://cci.lbl.gov/cctbx_build/)? [This page]( https://medium.com/@sljack1992/making-a-custom-ipython-notebook-kernel-c59e…) seems to describe the first part, but may no longer apply to more recent cctbx builds. Thanks. Wolfram Tempel

2 1

mtz2map gridding
by Elliot Nelson 21 Feb '19

21 Feb '19

Hi All, I've been trying to use Phenix.mtz2map to sample maps to a particular array size with the "gridding" option, but found that this seems to have no effect on the output of the program. I was wondering whether there were some restrictions to the integers that are permitted or otherwise what might be the cause of this behaviour. Having hunted through the code it seems to just set n_real in the crystal gridding, which is then fed into an fft - possibly the latter ignores n_reals that aren't sensible? Any idea? Thanks, Conor on behalf of Conor Wild: conor.wild(a)oriel.ox.ac.uk Elliot DPhil Student, Systems Approaches to Biomedical Science<http://www.sabsidc.ox.ac.uk/> Protein Crystallography<http://www.thesgc.org/groupprofile/9489>, Structural Genomics Consortium, NDM, University of Oxford Oxford Protein Informatics Group<http://opig.stats.ox.ac.uk/>, Department of Statistics, University of Oxford

2 1

OpenMP in fftpack
by Keitaro Yamashita 31 Jan '19

31 Jan '19

Dear CCTBX/PHENIX developers, According to PHENIX FAQ: https://www.phenix-online.org/documentation/faqs/index.html#general > In addition to these options, it is also possible to compile phenix.refine and Phaser with the OpenMP library, which automatically parallelizes specific instructions such as the FFT. This requires using the source installer for Phenix, and adding the argument "--openmp" to the install command. Because of threading conflicts, OpenMP is not compatible with the Phenix GUI. FFT in phenix.refine can be parallelized if OpenMP is enabled. However, it seems incorrect since 2017 due to this change: https://github.com/cctbx/cctbx_project/commit/3e932248e0491847a04ddeee429f8… The FAQ may need updating, but I am curious to know why it is disabled now. Best regards, Keitaro

1 0

This is a test
by Billy Poon 28 Jan '19

28 Jan '19

The Phenix website was down for a few hours, so this is a test to see if the bulletin board is working. -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

1 0

Re: [cctbxbb] [git/cctbx] master: Fix typo geobox (1ad3fe055)
by Pavel Afonine 22 Dec '18

22 Dec '18

Bad idea to hard-wire package name into lowest level code. Now imagine we need the same for afit, amber, quantumbio, rosetta, etc.. Are we going to have a page-long block of "if" statements. Please re-think and remove. For example, whether you want to shift it or not can be a parameter that you cast way level up in the context specific code. Pavel On 12/20/18 09:53, CCTBX commit wrote: > Repository : ssh://g18-sc-serv-04.diamond.ac.uk/cctbx > On branch : master > > ------------------------------------------------------------------------ > > > commit 1ad3fe05571162839e770c35ed032a7fc38a33b7 > Author: Gydo van Zundert <gydo.vanzundert(a)schrodinger.com> > Date: Wed Dec 19 14:58:36 2018 -0500 > > Fix typo geobox > > > ------------------------------------------------------------------------ > > > 1ad3fe05571162839e770c35ed032a7fc38a33b7 > mmtbx/refinement/real_space/individual_sites.py | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/mmtbx/refinement/real_space/individual_sites.py > b/mmtbx/refinement/real_space/individual_sites.py > index c61e88b01..20f38a6b0 100644 > --- a/mmtbx/refinement/real_space/individual_sites.py > +++ b/mmtbx/refinement/real_space/individual_sites.py > @@ -350,7 +350,7 @@ class box_refinement_manager(object): > > # When using the Schrodinger force field, move the whole > structure as the > # selected atoms are environment aware. > - if geobox.get_source() == 'SCHRODINGER': > + if geo_box.get_source() == 'SCHRODINGER': > geo_box.shift_cart(box.shift_cart) > > rsr_simple_refiner = simple( > @@ -377,7 +377,7 @@ class box_refinement_manager(object): > iselection, sites_cart_refined) > self.xray_structure.set_sites_cart(sites_cart_moving) > self.sites_cart = self.xray_structure.sites_cart() > - if geobox.get_source() == 'SCHRODINGER': > + if geo_box.get_source() == 'SCHRODINGER': > geo_box.shift_cart(shift_back) > else: # NCS constraints are present > # select on xrs, grm, ncs_groups > > ------------------------------------------------------------------------ > > To unsubscribe from the CCTBX-COMMIT list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCTBX-COMMIT&A=1 >

4 6

Broken iotbx/pdb/cif test
by richard.gildea＠diamond.ac.uk 19 Dec '18

19 Dec '18

According to Jenkins these two tests have been broken for several days - does anyone know what the issue is and could a fix be applied? Both look like the same error. iotbx/regression/tst_pdb_as_cif.py iotbx/regression/tst_data_manager.py These have been failing since this commit: https://github.com/cctbx/cctbx_project/commit/2c119fcf012ca4cc60dfce231902c… The tests also fail locally for me (not just on Jenkins). Cheers, Richard Traceback (most recent call last): File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/iotbx/regression/tst_pdb_as_cif.py", line 106, in <module> exercise_01() File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/iotbx/regression/tst_pdb_as_cif.py", line 37, in exercise_01 logger=null_out(), File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/iotbx/cli_parser.py", line 71, in run_program task.run() File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/programs/pdb_as_cif.py", line 37, in run txt = model.model_as_mmcif() File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/model/model.py", line 879, in model_as_mmcif if self.get_restraints_manager() is not None: File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/model/model.py", line 622, in get_restraints_manager self.setup_restraints_manager() File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/model/model.py", line 1042, in setup_restraints_manager self._process_input_model() File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/model/model.py", line 959, in _process_input_model use_neutron_distances = self._pdb_interpretation_params.pdb_interpretation.use_neutron_distances) File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/utils/__init__.py", line 998, in __init__ use_neutron_distances=use_neutron_distances, File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/monomer_library/server.py", line 675, in __init__ use_neutron_distances=use_neutron_distances, File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/monomer_library/server.py", line 689, in convert_all use_neutron_distances=use_neutron_distances, File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/monomer_library/server.py", line 700, in convert_lib_atom entry = cif_types.energy_lib_atom(**dict(row)) File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/monomer_library/cif_types.py", line 49, in __init__ raise_value_error("Floating-point") File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/monomer_library/cif_types.py", line 39, in raise_value_error expected_type, looped_id.split(":",1)[0], val)) ValueError: Floating-point value expected in CIF file: _lib_atom.ion_radius 1,7 Traceback (most recent call last): File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/iotbx/regression/tst_data_manager.py", line 532, in <module> test_model_datatype() File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/iotbx/regression/tst_data_manager.py", line 233, in test_model_datatype dm.get_model().model_as_mmcif(), overwrite=True) File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/model/model.py", line 879, in model_as_mmcif if self.get_restraints_manager() is not None: File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/model/model.py", line 622, in get_restraints_manager self.setup_restraints_manager() File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/model/model.py", line 1042, in setup_restraints_manager self._process_input_model() File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/model/model.py", line 959, in _process_input_model use_neutron_distances = self._pdb_interpretation_params.pdb_interpretation.use_neutron_distances) File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/utils/__init__.py", line 998, in __init__ use_neutron_distances=use_neutron_distances, File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/monomer_library/server.py", line 675, in __init__ use_neutron_distances=use_neutron_distances, File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/monomer_library/server.py", line 689, in convert_all use_neutron_distances=use_neutron_distances, File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/monomer_library/server.py", line 700, in convert_lib_atom entry = cif_types.energy_lib_atom(**dict(row)) File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/monomer_library/cif_types.py", line 49, in __init__ raise_value_error("Floating-point") File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/monomer_library/cif_types.py", line 39, in raise_value_error expected_type, looped_id.split(":",1)[0], val)) ValueError: Floating-point value expected in CIF file: _lib_atom.ion_radius 1,7 Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 7

forcing building boost_python early in SConscript
by Robert Oeffner 17 Nov '18

17 Nov '18

We are working on a newer version of Phaser. The new SConscript file for Phaser imports features from boost_python. Therefore it is currently not straightforward to build it since an empty build directory does not yet contain the boost_python module to be imported by SConscript (or python). So SCons crashes when it reads the new Phaser SCsonscript stating that boost_python cannot be imported. Currently our workaround is to first build CCTBX alone and then subsequently to build Phaser. My question is if it is possible to specify dependencies in cctbx_project/boost_adaptbx/SConscript to force building boost_python early on. On Windows boost_thread and a few other libraries are built before other SConscript files are read. So I have put some Depends(boost_thread, boost_python) andDepends(boost_thread, boost_python_meta_ext) statements into cctbx_project/boost_adaptbx/SConscript. These do seem to force building the boost_python.lib and .dll files. But building the boost_python_meta_ext python module fails with the error: boost_adaptbx\meta_ext.cpp(14): fatal error C1083: Cannot open include file: 'boost_adaptbx/type_id_eq.h' It's possible that I'm not using SCons as it's supposed to be used. Let me know if anyone has a suggestion on how to force building boost_python before other SConscript are read. Many thanks, Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234

5 5

Fwd: Re: indexing diffraction pattern
by Pavel Afonine 14 Nov '18

14 Nov '18

Perhaps someone can reply Wen.. Pavel -------- Forwarded Message -------- Subject: Re: indexing diffraction pattern Date: Sat, 10 Nov 2018 09:30:53 -0500 From: Wen Jiang <jiang12(a)purdue.edu> To: Pavel Afonine <pafonine(a)lbl.gov> Pavel, Sorry for insufficient details. The scenario is that I have a numpy 2D array of float values that represents an image of "diffraction pattern" (generated by an image processing operation, not actual X-ray diffraction). I need to determine the peak positions and then find a 2D lattice (two 2D vectors) going through these peaks. I have my own simple functions that can detect the peaks and find a 2-D lattice to cross the peaks but it often fails when the "diffraction pattern" is not very clean. Thus, I want to find a more robust method that works well when the peaks are relatively weak, not very sharp, and when there are some spurious peaks off the lattice. I think these tasks are exactly what cctbx has solved for indexing X-ray diffraction images. Hope that these explanations help clarify the problem. Thanks, Wen Prof. Wen Jiang Department of Biological Sciences Department of Chemistry (Courtesy) Scientific Director, Purdue Cryo-EM Facility Purdue University Hockmeyer Hall of Structural Biology 240 S. Martin Jischke Drive West Lafayette, IN 47907 http://jiang.bio.purdue.edu On Sat, Nov 10, 2018 at 8:03 AM Pavel Afonine <pafonine(a)lbl.gov <mailto:[email protected]>> wrote: Hi Wen, I'm not sure I fully understand your question... You say a function that takes a numpy array.. Array of what? Then, "a list of diffraction spots" -- these are measured in diffraction experiment. If you could provide more details as to what you are trying to do as well as what exactly the inputs are then I will do my best to see how to do this in CCTBX or forward your questions to CCTBX mailing list so that others can comment. All the best, Pavel On 11/10/18 03:56, Wen Jiang wrote: > Hi Pavel, > I am looking for a function that takes a numpy array and returns a > list of diffraction spots and lattice vectors. I think > cctbx should have such a function but I am not familiar with it. > Can you point me in the right direction? > Thanks, > > Wen > > Prof. Wen Jiang > Department of Biological Sciences > Department of Chemistry (Courtesy) > Scientific Director, Purdue Cryo-EM Facility > Purdue University > Hockmeyer Hall of Structural Biology > 240 S. Martin Jischke Drive > West Lafayette, IN 47907 > http://jiang.bio.purdue.edu >

2 1

Windows woes
by Luc Bourhis 09 Nov '18

09 Nov '18

Hi, Trying to install cctbx on Windows 10, 1. install latest python 2 from Python.org 2. python2 bootstrap.py —builder=cctbxlite —with-python=\path-to-python-from-step-1 It fails because of missing module “builtins”. So I installed pip, then used it to install “future”. Then I had to install “six” with pip. Am I being stupid or cctbx does not install out of the box anymore? Best wishes, Luc J Bourhis

3 3

Python 3 and CCP4
by David Waterman 02 Nov '18

02 Nov '18

Hi folks, It's great to see the momentum building behind the project to migrate to Python 3. From the roadmap <https://github.com/cctbx/cctbx_project/wiki/Python-3-roadmap>, there is clearly still some way to go, but current activity makes me wonder if the proposed deadline of complete migration by 2020 might be beaten easily. What is not clear from the roadmap is whether there are any plans for a post-transition grace period, in which we have switched from writing forwards compatible Python 2 code to writing backwards compatible Python 3 code. Has this been thought about? Or will it be the case that the first commit of Py3-only code seals the deal? I ask because CCP4 has to support a large number of Python projects and these have not all considered the Python 3 switch nearly as seriously as cctbx yet. What I would like to request is a reasonable period of backwards compatibility is ensured to allow other CCP4 projects to catch up, and that this protected period is enshrined within the roadmap or developer guidelines. What do you think? Best wishes -- David

3 2

construct hkl arrays from number of reflections and resolution
by Robert Oeffner 29 Oct '18

29 Oct '18

I would like a function that spits out an HKL array as a function of number of reflections, dmin and dmax. It's quite likely this has already been done somewhere in the CCTBX. Would anyone know if this is the case rather than me having to reinvent the wheel? Many thanks, Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234

2 1

Bootstrap on macOS Mojave
by Graeme.Winter＠Diamond.ac.uk 17 Oct '18

17 Oct '18

Folks, For info: https://github.com/cctbx/cctbx_project/issues/240 wxPython appears to not build on macOS Mojave - possibly updated C++ standards or something who knows? But I guess because we are fixed on an old (4 years, just about [1]) version of wxPython this should not be a surprise. We should probably have some kind of answer for this though - thoughts? Many thanks Graeme ——— [1] https://wxpython.org/news/wxpython-3.0.2.0-release/index.html -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 2

Fwd: cctbx conventions
by Graeme.Winter＠Diamond.ac.uk 08 Oct '18

08 Oct '18

HI Folks, Question from a colleague - any ideas? Many thanks Graeme Begin forwarded message: From: "Bombardi, Alessandro (DLSLtd,RAL,SCI)" <alessandro.bombardi(a)diamond.ac.uk<mailto:[email protected]>> Subject: cctbx conventions Date: 8 October 2018 at 14:54:10 BST To: "Winter, Graeme (DLSLtd,RAL,LSCI)" <Graeme.Winter(a)Diamond.ac.uk<mailto:[email protected]>> Hi Graeme, I am trying to understand the convention used by cctbx, for the anomalous part of the atomic scattering factor. Do you know if it is written as f+f'+if'' with f'' >0 or with f+f'-if'' with f'' >0 can you help me? All the best, Alessandro -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 3

non-standard test naming in mmtbx
by Pearce, N.M. (Nick) 03 Oct '18

03 Oct '18

Hi, I’m trying to use the automatic pytest discovery in mmtbx and I notice that there are a few “libtbx-style” tests, namely "$D/conformation_dependent_library/test_cdl.py", "$D/conformation_dependent_library/test_cdl_esd.py", “$D/conformation_dependent_library/test_rdl.py", "$D/conformation_dependent_library/test_hpdl.py”, that follow the “pytest” naming convention, and are thus picked up by the pytest.discover() function. They then error because pytest cannot run them. There are — as far as I can tell — normal libtbx tests (they are registered in run_tests.py). Is there any reason they do not follow the typical naming convention? Changing their filenames and modifying run_tests.py accordingly seems to lead to the same behaviour when running tests. Is there any reason they should not be renamed from “test_*.py” to “tst_*.py”? Thanks, Nick

3 3

Allocators
by Graeme.Winter＠Diamond.ac.uk 25 Sep '18

25 Sep '18

Hello All, In keeping with the spirit of https://github.com/cctbx/cctbx_project/wiki/cctbx-contributors-guide we would like to revisit the allocator work now Phenix 1.14 is out in the wild - however since we now know that this needs to be done we are interested in other points of view people have on ways to do this, and options people would like to have. It is important to note that we have no intention of changing the default behaviour, but we do want to be able to use customised allocators with flex arrays e.g. for handing out memory of well known and repeatedly used sizes (say, 16 M pixel double arrays) or allocate in specific areas for NUMA systems such as xeon phi. We prototyped this a while back and stepped the changes back out again to enable the phenix release to proceed smoothly, so we have a pretty good idea of the sorts of things we would expect to do however will welcome input if you would like additional features. As a baseline, we are talking as a parameter that PHENIX are no longer supporting CENTOS 5 32 bit (since ISTR this was a constraint before) Any thoughts? Appreciate a number of folks will be travelling back after the PHENIX workshop. Many thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 2

Conda
by Graeme.Winter＠Diamond.ac.uk 24 Sep '18

24 Sep '18

Hello again, Trying to keep threads from getting warped :-) We’re interested in what the plans and timescales are for the move to Conda for a base to cctbx-related projects, and in particular - - what is planned? - when i.e. timescales (rough) - will bootstrap go away? I know from talking to Billy on Wednesday that this is back on the cards again now the PHENIX release is out, but I didn’t ask at the time fo the details… oops! Many thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 2