- cctbxbb - phenix-online.org

Re: [cctbxbb] Molprobity / CCTBX incompatibility
by Nigel Moriarty 12 Aug '19

12 Aug '19

Not sure about the selection problem but the missing commands are likely molprobity.reduce and molprobity.probe. Chris, you should be able to change that. Cheers Nigel --- Nigel W. Moriarty Building 33R0349, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709 Email : NWMoriarty(a)LBL.gov Fax : 510-486-5909 Web : CCI.LBL.gov On Mon, Aug 12, 2019 at 11:43 AM Charitha Belluru Shetty < charitha-belluru.b.shetty(a)gsk.com> wrote: > Hi Christopher/Nigel, > > > > We downloaded the molprobity after the patch and ran the configure.sh and > setup scripts. Now things got worst. > > > > Can you please help us on this. > > > > Here is the output we are getting. > > > > > > > > > > > > > > When we press the *“Click here”* button, below log file is generated. > > > > [PLACE YOUR COMMENTS HERE] > > > > ================================================== > > USER / SERVER INFORMATION > > ================================================== > > User IP : 11.27.95.128 > > User browser : Mozilla/5.0 (Windows NT 10.0; Win64; x64) > AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36 > > Session ID : c2h8n399dagb0ggfn8kpla6fu4 > > Session URL : > http://uptus002vm.corpnet2.com/molprobity/index.php?MolProbSID=c2h8n399dagb… > > > > MolProb. ver. : 4.4 > > Server name : uptus002vm.corpnet2.com > > Server time : 2019 Aug 12 18:24:41 > > > > ================================================== > > ERRORS FROM THIS SESSION > > ================================================== > > > > > > sh: phenix.reduce: command not found > > sh: phenix.reduce: command not found > > sh: phenix.reduce: command not found > > sh: phenix.probe: command not found > > sh: phenix.probe: command not found > > sh: phenix.reduce: command not found > > sh: phenix.reduce: command not found > > sh: phenix.probe: command not found > > sh: phenix.probe: command not found > > > > Starting phenix.clashscore > > on Mon Aug 12 13:50:31 2019 by appmgmt > > > =============================================================================== > > > > Processing files: > > > ------------------------------------------------------------------------------- > > > > Files not used by program: > > -------------------------- > > > /hpc/apps/2018/molprobity/v4.4.app/MolProbity/public_html/data/c2h8n399dagb0ggfn8kpla6fu4/coordinates/1irkFH.pdb > > > > Processing PHIL parameters: > > > ------------------------------------------------------------------------------- > > > > Adding command-line PHIL: > > ------------------------- > > b_factor_cutoff=40 > > clash_cutoff=-0.4 > > > > Final processed PHIL parameters: > > > ------------------------------------------------------------------------------- > > data_manager { > > default_output_filename = "cctbx_program" > > } > > b_factor_cutoff = 40 > > > > > > Starting job > > > =============================================================================== > > Sorry: 0 model(s) found. Expected at least 1. > > chartClashlist ran for 5 seconds > > do Charts ran for 0 seconds > > sh: phenix.probe: command not found > > do Kinemage ran for 0 seconds > > > > Traceback (most recent call last): > > File > "/hpc/apps/2018/molprobity/v4.4.app/MolProbity/build/../modules/cctbx_project/mmtbx/command_line/mp_geo.py", > line 7, in <module> > > mp_geo.run(sys.argv[1:]) > > File > "/hpc/apps/2018/molprobity/v4.4.app/MolProbity/modules/cctbx_project/mmtbx/validation/molprobity/mp_geo.py", > line 132, in run > > pdb_hierarchy.models()[0].chains(): > > IndexError: list index out of range > > > > > > > > > > sh: phenix.reduce: command not found > > > > sh: phenix.reduce: command not found > > > > sh: phenix.reduce: command not found > > > > sh: phenix.probe: command not found > > > > sh: phenix.probe: command not found > > > > sh: phenix.reduce: command not found > > > > sh: phenix.reduce: command not found > > > > sh: phenix.probe: command not found > > > > sh: phenix.probe: command not found > > > > > > > > Starting phenix.clashscore > > > > on Mon Aug 12 13:50:31 2019 by appmgmt > > > > > =============================================================================== > > > > > > > > Processing files: > > > > > ------------------------------------------------------------------------------- > > > > > > > > Files not used by program: > > > > -------------------------- > > > > > /hpc/apps/2018/molprobity/v4.4.app/MolProbity/public_html/data/c2h8n399dagb0ggfn8kpla6fu4/coordinates/1irkFH.pdb > > > > > > > > Processing PHIL parameters: > > > > > ------------------------------------------------------------------------------- > > > > > > > > Adding command-line PHIL: > > > > ------------------------- > > > > b_factor_cutoff=40 > > > > clash_cutoff=-0.4 > > > > > > > > Final processed PHIL parameters: > > > > > ------------------------------------------------------------------------------- > > > > data_manager { > > > > default_output_filename = "cctbx_program" > > > > } > > > > b_factor_cutoff = 40 > > > > > > > > > > > > Starting job > > > > > =============================================================================== > > > > Sorry: 0 model(s) found. Expected at least 1. > > > > chartClashlist ran for 5 seconds > > > > do Charts ran for 0 seconds > > > > sh: phenix.probe: command not found > > > > do Kinemage ran for 0 seconds > > > > > > > > Traceback (most recent call last): > > > > File > "/hpc/apps/2018/molprobity/v4.4.app/MolProbity/build/../modules/cctbx_project/mmtbx/command_line/mp_geo.py", > line 7, in <module> > > > > mp_geo.run(sys.argv[1:]) > > > > File > "/hpc/apps/2018/molprobity/v4.4.app/MolProbity/modules/cctbx_project/mmtbx/validation/molprobity/mp_geo.py", > line 132, in run > > > > pdb_hierarchy.models()[0].chains(): > > > > IndexError: list index out of range > > > > > > > > > > > > sh: phenix.reduce: command not found > > > > sh: phenix.reduce: command not found > > > > sh: phenix.reduce: command not found > > > > sh: phenix.probe: command not found > > > > sh: phenix.probe: command not found > > > > sh: phenix.reduce: command not found > > > > sh: phenix.reduce: command not found > > > > sh: phenix.probe: command not found > > > > sh: phenix.probe: command not found > > > > > > > > Starting phenix.clashscore > > > > on Mon Aug 12 13:50:31 2019 by appmgmt > > > > > =============================================================================== > > > > > > > > Processing files: > > > > > ------------------------------------------------------------------------------- > > > > > > > > Files not used by program: > > > > -------------------------- > > > > > /hpc/apps/2018/molprobity/v4.4.app/MolProbity/public_html/data/c2h8n399dagb0ggfn8kpla6fu4/coordinates/1irkFH.pdb > > > > > > > > Processing PHIL parameters: > > > > > ------------------------------------------------------------------------------- > > > > > > > > Adding command-line PHIL: > > > > ------------------------- > > > > b_factor_cutoff=40 > > > > clash_cutoff=-0.4 > > > > > > > > Final processed PHIL parameters: > > > > > ------------------------------------------------------------------------------- > > > > data_manager { > > > > default_output_filename = "cctbx_program" > > > > } > > > > b_factor_cutoff = 40 > > > > > > > > > > > > Starting job > > > > > =============================================================================== > > > > Sorry: 0 model(s) found. Expected at least 1. > > > > chartClashlist ran for 5 seconds > > > > do Charts ran for 0 seconds > > > > sh: phenix.probe: command not found > > > > do Kinemage ran for 0 seconds > > > > > > > > Traceback (most recent call last): > > > > File > "/hpc/apps/2018/molprobity/v4.4.app/MolProbity/build/../modules/cctbx_project/mmtbx/command_line/mp_geo.py", > line 7, in <module> > > > > mp_geo.run(sys.argv[1:]) > > > > File > "/hpc/apps/2018/molprobity/v4.4.app/MolProbity/modules/cctbx_project/mmtbx/validation/molprobity/mp_geo.py", > line 132, in run > > > > pdb_hierarchy.models()[0].chains(): > > > > IndexError: list index out of range > > > > > > > > > > > > > > > > > > > > sh: phenix.reduce: command not found > > > > > > > > sh: phenix.reduce: command not found > > > > > > > > sh: phenix.reduce: command not found > > > > > > > > sh: phenix.probe: command not found > > > > > > > > sh: phenix.probe: command not found > > > > > > > > sh: phenix.reduce: command not found > > > > > > > > sh: phenix.reduce: command not found > > > > > > > > sh: phenix.probe: command not found > > > > > > > > sh: phenix.probe: command not found > > > > > > > > > > > > > > > > Starting phenix.clashscore > > > > > > > > on Mon Aug 12 13:50:31 2019 by appmgmt > > > > > > > > > =============================================================================== > > > > > > > > > > > > > > > > Processing files: > > > > > > > > > ------------------------------------------------------------------------------- > > > > > > > > > > > > > > > > Files not used by program: > > > > > > > > -------------------------- > > > > > > > > > /hpc/apps/2018/molprobity/v4.4.app/MolProbity/public_html/data/c2h8n399dagb0ggfn8kpla6fu4/coordinates/1irkFH.pdb > > > > > > > > > > > > > > > > Processing PHIL parameters: > > > > > > > > > ------------------------------------------------------------------------------- > > > > > > > > > > > > > > > > Adding command-line PHIL: > > > > > > > > ------------------------- > > > > > > > > b_factor_cutoff=40 > > > > > > > > clash_cutoff=-0.4 > > > > > > > > > > > > > > > > Final processed PHIL parameters: > > > > > > > > > ------------------------------------------------------------------------------- > > > > > > > > data_manager { > > > > > > > > default_output_filename = "cctbx_program" > > > > > > > > } > > > > > > > > b_factor_cutoff = 40 > > > > > > > > > > > > > > > > > > > > > > > > Starting job > > > > > > > > > =============================================================================== > > > > > > > > Sorry: 0 model(s) found. Expected at least 1. > > > > > > > > chartClashlist ran for 5 seconds > > > > > > > > do Charts ran for 0 seconds > > > > > > > > sh: phenix.probe: command not found > > > > > > > > do Kinemage ran for 0 seconds > > > > > > > > > > > > > > > > Traceback (most recent call last): > > > > > > > > File > "/hpc/apps/2018/molprobity/v4.4.app/MolProbity/build/../modules/cctbx_project/mmtbx/command_line/mp_geo.py", > line 7, in <module> > > > > > > > > mp_geo.run(sys.argv[1:]) > > > > > > > > File > "/hpc/apps/2018/molprobity/v4.4.app/MolProbity/modules/cctbx_project/mmtbx/validation/molprobity/mp_geo.py", > line 132, in run > > > > > > > > pdb_hierarchy.models()[0].chains(): > > > > > > > > IndexError: list index out of range > > > > > > Best Regards, > > Charitha Shetty > > > > *From:* Robert Nolte > *Sent:* Sunday, August 11, 2019 12:39 PM > *To:* Nigel Moriarty (nwmoriarty(a)lbl.gov) <nwmoriarty(a)lbl.gov>; > cctbxbb(a)phenix-online.org > *Cc:* Charitha Belluru Shetty <charitha-belluru.b.shetty(a)gsk.com> > *Subject:* Molprobity / CCTBX incompatibility > > > > We are trying to reinstall our internal molprobity server, and have hit an > issue with the github version where it appears to be a version conflict > with the version of cctbx that is currently being downloaded by > configuration script in the master branch of molprobity. We are getting > errors when we run the validation routine such as > > > > Unrecogized PHIL parameter: data_version=8000 > > > > or > > > > Unrecognized PHIL parameter: clash_cutoff=-0.4 > > > > The analysis for clashscore, rotalyze, are failing to run and we get a > number of divide by zero errors for several of the other ‘visualize’ and > ‘eff_resolution’ routines. > > > > Can you help get this working. > > Thanks, > > Bob Nolte > > > > > > > > *GSK monitors email communications sent to and from GSK in order to > protect GSK, our employees, customers, suppliers and business partners, > from cyber threats and loss of GSK Information. GSK monitoring is conducted > with appropriate confidentiality controls and in accordance with local laws > and after appropriate consultation.* >

1 0

Molprobity / CCTBX incompatibility
by Robert Nolte 11 Aug '19

11 Aug '19

We are trying to reinstall our internal molprobity server, and have hit an issue with the github version where it appears to be a version conflict with the version of cctbx that is currently being downloaded by configuration script in the master branch of molprobity. We are getting errors when we run the validation routine such as Unrecogized PHIL parameter: data_version=8000 or Unrecognized PHIL parameter: clash_cutoff=-0.4 The analysis for clashscore, rotalyze, are failing to run and we get a number of divide by zero errors for several of the other 'visualize' and 'eff_resolution' routines. Can you help get this working. Thanks, Bob Nolte GSK monitors email communications sent to and from GSK in order to protect GSK, our employees, customers, suppliers and business partners, from cyber threats and loss of GSK Information. GSK monitoring is conducted with appropriate confidentiality controls and in accordance with local laws and after appropriate consultation.

1 0

Azure pipelines Linux builds failing with segmentation fault
by richard.gildea＠diamond.ac.uk 31 Jul '19

31 Jul '19

Recent Azure builds are failing for the Linux builds with the following error: ##[section]Starting: Test ============================================================================== Task : Command line Description : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows Version : 2.151.2 Author : Microsoft Corporation Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line ============================================================================== Generating script. ========================== Starting Command Output =========================== [command]/bin/bash --noprofile --norc /__w/_temp/823d09b8-e953-4f93-a9a8-0d863a25f1ce.sh /__w/_temp/823d09b8-e953-4f93-a9a8-0d863a25f1ce.sh: line 11: 6307 Segmentation fault (core dumped) libtbx.run_tests_parallel module=boost_adaptbx module=cctbx module=cma_es module=fable module=gltbx module=iotbx module=libtbx module=rstbx module=scitbx module=simtbx module=smtbx module=spotfinder module=annlib_adaptbx module=cbflib_adaptbx nproc=4 ##[error]Bash exited with code '139'. ##[section]Finishing: Test https://dev.azure.com/cctbx/cctbx_project/_build/results?buildId=256 The last successful Linux Python 2.7 master build was https://dev.azure.com/cctbx/cctbx_project/_build/results?buildId=240 The first failing Linux Python 2.7 master build was https://dev.azure.com/cctbx/cctbx_project/_build/results?buildId=245 Mac builds appear to be fine. Does anyone have any idea what is the issue? Cheers, Richard Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 2

Failing tests on python3 builds
by richard.gildea＠diamond.ac.uk 30 Jul '19

30 Jul '19

Hi, Whilst looking into test failures reported against a pull request I currently have open, I noticed that there are several (9) failing tests on the Python 3 build. It appears that Markus reported these 3 weeks ago - is anyone looking into these failures? Having builds permanently reporting failure isn't particularly helpful as it discourages people from looking at the results, and will likely lead to further Python 3 regressions being introduced into the codebase. https://dev.azure.com/cctbx/cctbx_project/_build/results?buildId=240 https://github.com/cctbx/cctbx_project/issues Cheers, Richard Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 4

"duplicate" cif categories
by wtempel 06 Jul '19

06 Jul '19

Hi, wwpdb onedep recently rejected a file that I had manipulated (but not validated, I admit) in IOTBX. The error was "duplicate categories", or a similar complaint. I had added _reflns.* items related to data processing to a phenix.refine cif file that already included a _reflns.B_iso_Wilson_estimate. Merely running sort() on the relevant block made the error go away, but who knows what side effects there may have been. Does the CCTBX ecosystem provide some higher-level management of cif categories? Thanks. Wolfram

1 0

align density maps
by Peter Schmidtke 02 Jul '19

02 Jul '19

2 3

pickling nan values
by Robert Oeffner 13 Jun '19

13 Jun '19

Hi all, It seems that a flex.double array containing "nan" values doesn't survive pickling as in: oeffner@grove:~$ cctbx.python Python 2.7.15 (default, Sep 11 2018, 23:48:45) [GCC 4.4.7 20120313 (Red Hat 4.4.7-18)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> >>> from cctbx.array_family import flex >>> import pickle >>> flex_nans = flex.double(3, float('nan')) >>> list(flex_nans) [nan, nan, nan] >>> list(pickle.loads( pickle.dumps(flex_nans))) [0.0, 0.0, 0.0] >>> >>> If the flex double is cast into a python list then it does survive pickling as in: >>> >>> list(pickle.loads( pickle.dumps(list( flex_nans )))) [nan, nan, nan] >>> I would like to use nan values to indicate missing or invalid data values. Would anyone have a suggestion how to go about this? Many thanks, Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus The Keith Peters Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234

2 1

Fwd: Conda/DIALS 1.14.5 source build with --use-conda does not work
by Nicholas Devenish 27 May '19

27 May '19

(Forwarding this manually as apparently my Diamond address isn't on cctbxbb) Hi All, On 24 May 2019, at 14:12, David Waterman - UKRI STFC < david.waterman(a)stfc.ac.uk<mailto:[email protected]>> wrote: > In the process, I discovered that the source builds we point to from > https://dials.github.io/installation.html don't work with --use-conda > because the paths to various packages are out of date. This seems to be to be a fundamental problem with the conda implementation? Let’s try to reproduce an old build from, say, 1st march: the point in cctbx_project at which dials 1.14 split off. This is https://github.com/cctbx/cctbx_project/commit/35652e06564da7ff989500a6d929b… . Try reproducing this point: git clone git(a)github.com<mailto:[email protected]>:cctbx/cctbx_project.git modules/cctbx_project ( cd modules/cctbx_project/ && git checkout 35652e06564da7ff989500a6d929bc595f69040f ) ln -s modules/cctbx_project/libtbx/auto_build/bootstrap.py . ./bootstrap.py base --use-conda —> many "An HTTP error occurred when trying to retrieve this URL.” This means that with the current conda implementation it’s impossible to guarantee reproducibility for any build beyond the past few days, without lots and lots of manual work reconstructing the versioning out of the file list dump. Because the version constraints aren’t kept with the repository, this means that a source distribution is not nearly enough to recreate any given release. This seems like a problem? Nick

3 2

Python3 / scons / non-conda builds
by Graeme.Winter＠Diamond.ac.uk 12 May '19

12 May '19

Dear All, Back in the mists of time, one could build cctbx modules without bootstrap or conda by having the right modules available yourself - a so-called baseless install which I think is still important in X-fel land? Anyhow, trying this with Python3 suggests that we have some special constraints which are perhaps too tight regarding module locations - for example scons - Grey-Area p3_compile :( $ pip3 install scons Collecting scons Downloading https://files.pythonhosted.org/packages/65/41/ac58638421414acc64e26fc120c81… (831kB) 100% |████████████████████████████████| 839kB 14.3MB/s Installing collected packages: scons Successfully installed scons-3.0.5 You are using pip version 19.0.1, however version 19.1.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command. Grey-Area p3_compile :) $ make ./bin/libtbx.scons -j "`./bin/libtbx.show_number_of_processors`" ModuleNotFoundError: No module named 'SCons' During handling of the above exception, another exception occurred: Sorry: SCons is not available. A possible solution is to unpack a SCons distribution in one of these directories: "/Users/graeme/svn/scr" "/Users/graeme/svn/scr/cctbx_project" SCons distributions are available at this location: http://www.scons.org/ It may be necessary to rename the unpacked distribution, e.g.: mv scons-0.96.1 scons make: *** [default] Error 1 Grey-Area p3_compile :( $ which scons /Library/Frameworks/Python.framework/Versions/3.7/bin/scons Should we require that scons is installed as part of cctbx when it is already installed on the system? If we *do* have to have this constraint, maybe that should be checked at configure time (where it complains about six and future) Thoughts? In other news - the syntax changing branch referenced in https://github.com/cctbx/cctbx_project/pull/325 now compiles correctly on Python3 (note: not run, just compile but this means that it’s purpose is done) for the modules which it intends to touch and has no conflicts right now… Best wishes Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 2

Overloading Python builtin names with local variables
by Graeme.Winter＠Diamond.ac.uk 03 May '19

03 May '19

Morning all, As a side-effect of looking at some of the Python3 refactoring I stumbled across a few places where e.g. range is used as a variable name. While this is fine and legal, when moving the code to Python3 xrange => range and from builtins import range is a back-port of Python3 range to Python2 which allows the existing behaviour of xrange generator to be maintained This is fine except for where range is a variable name and we then get a “variable referenced before assignment” type error As a general statement I would say overloading Python reserved names with local variables is untidy at best, and can easily cause confusion (as well as real problems as identified above) - I have therefore taken the liberty of adding a tool to libtbx which can be used to find such variables to give people a chance to avoid them: Grey-Area dxtbx :) [master] $ libtbx.find_reserved_names format/ Checking format/ format/FormatSMVRigakuSaturn.py:239 "format" format = self._scan_factory.format("SMV") format/FormatTIFFRayonixESRF.py:36 "format" format = {LITTLE_ENDIAN: "<", BIG_ENDIAN: ">"}[order] format/FormatTIFFRayonixESRF.py:29 "bytes" width, height, depth, order, bytes = FormatTIFFRayonix.get_tiff_header( format/FormatCBFMini.py:231 "format" format = self._scan_factory.format("CBF") format/FormatTIFFRayonix.py:176 "format" format = self._scan_factory.format("TIFF") format/FormatTIFFRayonix.py:31 "bytes" width, height, depth, order, bytes = FormatTIFF.get_tiff_header(image_file) format/FormatTIFFRayonix.py:74 "bytes" width, height, depth, order, bytes = FormatTIFF.get_tiff_header(image_file) format/FormatSMVRigakuEiger.py:243 "format" format = self._scan_factory.format("SMV") format/FormatSMVADSC.py:212 "format" format = self._scan_factory.format("SMV") format/Registry.py:88 "format" for format in sorted(self._formats, key=lambda x: x.__name__): format/FormatSMVJHSim.py:141 "format" format = self._scan_factory.format("SMV") format/FormatSMVRigakuPilatus.py:188 "format" format = self._scan_factory.format("SMV") format/FormatSMVADSCSN928.py:47 "format" format = self._scan_factory.format("SMV") format/FormatRAXISIVSpring8.py:134 "format" format = self._scan_factory.format("RAXIS") format/FormatTIFFRayonixSPring8.py:31 "bytes" width, height, depth, order, bytes = FormatTIFFRayonix.get_tiff_header( format/FormatSMVADSCmlfsom.py:37 "format" format = self._scan_factory.format("SMV") format/FormatTIFFBruker.py:174 "format" format = self._scan_factory.format("TIFF") format/FormatTIFFBruker.py:31 "bytes" width, height, depth, order, bytes = FormatTIFF.get_tiff_header(image_file) format/FormatTIFFBruker.py:71 "bytes" width, height, depth, order, bytes = FormatTIFF.get_tiff_header(image_file) format/FormatCBFMiniADSCHF4M.py:26 "format" for format in ["%a_%b_%d_%H:%M:%S_%Y"]: format/FormatCBFMiniADSCHF4M.py:169 "format" format = self._scan_factory.format("CBF") format/FormatSMVADSCNoDateStamp.py:54 "format" format = self._scan_factory.format("SMV") format/FormatSMVRigakuSaturnNoTS.py:228 "format" format = self._scan_factory.format("SMV") format/FormatPYunspecified.py:152 "format" format = "" format/FormatSMVRigakuA200.py:234 "format" format = self._scan_factory.format("SMV") format/FormatCBFMiniEigerPhotonFactory.py:131 "format" format = self._scan_factory.format("CBF") format/FormatSMVTimePix_SU.py:97 "format" format = self._scan_factory.format("SMV") format/FormatCBFMiniPilatus.py:83 "format" format = self._scan_factory.format("CBF") format/FormatCBFMiniPilatusHelpers.py:22 "format" for format in ["%Y-%b-%dT%H:%M:%S", "%Y-%m-%dT%H:%M:%S", "%Y/%b/%d %H:%M:%S"]: format/FormatSMVADSCSNAPSID19.py:91 "format" format = self._scan_factory.format("SMV") format/FormatCBFMiniEiger.py:216 "format" format = self._scan_factory.format("CBF") format/FormatSMVNOIR.py:218 "format" format = self._scan_factory.format("SMV") format/FormatSMVCMOS1.py:197 "format" format = self._scan_factory.format("SMV") format/FormatRAXIS.py:238 "format" format = self._scan_factory.format("RAXIS") is an example where “format” is scattered all over the place but is also a useful function - replacing with fmt will mean the same and avoid this overloading, as an example. This uses AST parsing to work through the code and find variable assignments - doing this for dials found lots of examples of “file” and “type” being used. How much people care about this is a local issue, but I thought the tool would be useful All the best Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

modulo operation on flex.double
by Robert Oeffner 11 Apr '19

11 Apr '19

Hi, I have a flex.double array of phase values between -360 and 360 degrees I would like to cast them all in one go to lie within 0 to 360. Does anybody know if there is a nice way of doing this with flex arrays? The modulo operator doesn't seem to be supported for flex arrays unlike the +, -, * and the / operators. This would otherwise be handy as I could then convert the array in one line of code like phases = phases % 360.0 If not I guess I'll have to numpy the problem :-) Many thanks, Rob

5 5

Re: [cctbxbb] cannot compile Phenix
by R. D. Oeffner 06 Apr '19

06 Apr '19

>From the messages on the cctbx github mailing list it seems that Dials people have been extricating dxtbx from cctbx_project. Rob -----Original Message----- From: Pavel Afonine Sent: Saturday, April 6, 2019 7:07 PM To: phenixdev Subject: cannot compile Phenix Hi, anyone knows why current Phenix code is broken: python bootstrap.py --builder=phenix --sfuser=afonine --cciuser=afonine update python bootstrap.py --builder=phenix --sfuser=afonine --cciuser=afonine build --nproc=12 leads to (...) AttributeError: 'empty' object has no attribute 'dxtbx_include': File "/Users/pafonine/phenix/build/SConstruct", line 31: SConscript("spotfinder/SConscript") File "/Users/pafonine/phenix/modules/scons/engine/SCons/Script/SConscript.py", line 607: return method(*args, **kw) File "/Users/pafonine/phenix/modules/scons/engine/SCons/Script/SConscript.py", line 544: return _SConscript(self.fs, *files, **subst_kw) File "/Users/pafonine/phenix/modules/scons/engine/SCons/Script/SConscript.py", line 253: call_stack[-1].globals) File "/Users/pafonine/phenix/modules/cctbx_project/spotfinder/SConscript", line 22: env_etc.dxtbx_include, usr+sys time: 1.94 seconds wall clock time: 4.51 seconds Process failed with return code 2 Any idea for a workaround? Pavel --- This email has been checked for viruses by AVG. https://www.avg.com

2 3

jupyter notebook
by wtempel 02 Apr '19

02 Apr '19

Hi all, how does one configure, and run inside a jupyter notebook, a cctbx-enabled kernel, beginning with a downloaded [cctbx build]( http://cci.lbl.gov/cctbx_build/)? [This page]( https://medium.com/@sljack1992/making-a-custom-ipython-notebook-kernel-c59e…) seems to describe the first part, but may no longer apply to more recent cctbx builds. Thanks. Wolfram Tempel

2 1

mtz2map gridding
by Elliot Nelson 21 Feb '19

21 Feb '19

Hi All, I've been trying to use Phenix.mtz2map to sample maps to a particular array size with the "gridding" option, but found that this seems to have no effect on the output of the program. I was wondering whether there were some restrictions to the integers that are permitted or otherwise what might be the cause of this behaviour. Having hunted through the code it seems to just set n_real in the crystal gridding, which is then fed into an fft - possibly the latter ignores n_reals that aren't sensible? Any idea? Thanks, Conor on behalf of Conor Wild: conor.wild(a)oriel.ox.ac.uk Elliot DPhil Student, Systems Approaches to Biomedical Science<http://www.sabsidc.ox.ac.uk/> Protein Crystallography<http://www.thesgc.org/groupprofile/9489>, Structural Genomics Consortium, NDM, University of Oxford Oxford Protein Informatics Group<http://opig.stats.ox.ac.uk/>, Department of Statistics, University of Oxford

2 1

OpenMP in fftpack
by Keitaro Yamashita 30 Jan '19

30 Jan '19

Dear CCTBX/PHENIX developers, According to PHENIX FAQ: https://www.phenix-online.org/documentation/faqs/index.html#general > In addition to these options, it is also possible to compile phenix.refine and Phaser with the OpenMP library, which automatically parallelizes specific instructions such as the FFT. This requires using the source installer for Phenix, and adding the argument "--openmp" to the install command. Because of threading conflicts, OpenMP is not compatible with the Phenix GUI. FFT in phenix.refine can be parallelized if OpenMP is enabled. However, it seems incorrect since 2017 due to this change: https://github.com/cctbx/cctbx_project/commit/3e932248e0491847a04ddeee429f8… The FAQ may need updating, but I am curious to know why it is disabled now. Best regards, Keitaro

1 0

This is a test
by Billy Poon 28 Jan '19

28 Jan '19

The Phenix website was down for a few hours, so this is a test to see if the bulletin board is working. -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

1 0

Re: [cctbxbb] [git/cctbx] master: Fix typo geobox (1ad3fe055)
by Pavel Afonine 22 Dec '18

22 Dec '18

Bad idea to hard-wire package name into lowest level code. Now imagine we need the same for afit, amber, quantumbio, rosetta, etc.. Are we going to have a page-long block of "if" statements. Please re-think and remove. For example, whether you want to shift it or not can be a parameter that you cast way level up in the context specific code. Pavel On 12/20/18 09:53, CCTBX commit wrote: > Repository : ssh://g18-sc-serv-04.diamond.ac.uk/cctbx > On branch : master > > ------------------------------------------------------------------------ > > > commit 1ad3fe05571162839e770c35ed032a7fc38a33b7 > Author: Gydo van Zundert <gydo.vanzundert(a)schrodinger.com> > Date: Wed Dec 19 14:58:36 2018 -0500 > > Fix typo geobox > > > ------------------------------------------------------------------------ > > > 1ad3fe05571162839e770c35ed032a7fc38a33b7 > mmtbx/refinement/real_space/individual_sites.py | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/mmtbx/refinement/real_space/individual_sites.py > b/mmtbx/refinement/real_space/individual_sites.py > index c61e88b01..20f38a6b0 100644 > --- a/mmtbx/refinement/real_space/individual_sites.py > +++ b/mmtbx/refinement/real_space/individual_sites.py > @@ -350,7 +350,7 @@ class box_refinement_manager(object): > > # When using the Schrodinger force field, move the whole > structure as the > # selected atoms are environment aware. > - if geobox.get_source() == 'SCHRODINGER': > + if geo_box.get_source() == 'SCHRODINGER': > geo_box.shift_cart(box.shift_cart) > > rsr_simple_refiner = simple( > @@ -377,7 +377,7 @@ class box_refinement_manager(object): > iselection, sites_cart_refined) > self.xray_structure.set_sites_cart(sites_cart_moving) > self.sites_cart = self.xray_structure.sites_cart() > - if geobox.get_source() == 'SCHRODINGER': > + if geo_box.get_source() == 'SCHRODINGER': > geo_box.shift_cart(shift_back) > else: # NCS constraints are present > # select on xrs, grm, ncs_groups > > ------------------------------------------------------------------------ > > To unsubscribe from the CCTBX-COMMIT list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCTBX-COMMIT&A=1 >

4 6

Broken iotbx/pdb/cif test
by richard.gildea＠diamond.ac.uk 19 Dec '18

19 Dec '18

According to Jenkins these two tests have been broken for several days - does anyone know what the issue is and could a fix be applied? Both look like the same error. iotbx/regression/tst_pdb_as_cif.py iotbx/regression/tst_data_manager.py These have been failing since this commit: https://github.com/cctbx/cctbx_project/commit/2c119fcf012ca4cc60dfce231902c… The tests also fail locally for me (not just on Jenkins). Cheers, Richard Traceback (most recent call last): File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/iotbx/regression/tst_pdb_as_cif.py", line 106, in <module> exercise_01() File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/iotbx/regression/tst_pdb_as_cif.py", line 37, in exercise_01 logger=null_out(), File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/iotbx/cli_parser.py", line 71, in run_program task.run() File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/programs/pdb_as_cif.py", line 37, in run txt = model.model_as_mmcif() File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/model/model.py", line 879, in model_as_mmcif if self.get_restraints_manager() is not None: File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/model/model.py", line 622, in get_restraints_manager self.setup_restraints_manager() File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/model/model.py", line 1042, in setup_restraints_manager self._process_input_model() File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/model/model.py", line 959, in _process_input_model use_neutron_distances = self._pdb_interpretation_params.pdb_interpretation.use_neutron_distances) File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/utils/__init__.py", line 998, in __init__ use_neutron_distances=use_neutron_distances, File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/monomer_library/server.py", line 675, in __init__ use_neutron_distances=use_neutron_distances, File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/monomer_library/server.py", line 689, in convert_all use_neutron_distances=use_neutron_distances, File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/monomer_library/server.py", line 700, in convert_lib_atom entry = cif_types.energy_lib_atom(**dict(row)) File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/monomer_library/cif_types.py", line 49, in __init__ raise_value_error("Floating-point") File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/monomer_library/cif_types.py", line 39, in raise_value_error expected_type, looped_id.split(":",1)[0], val)) ValueError: Floating-point value expected in CIF file: _lib_atom.ion_radius 1,7 Traceback (most recent call last): File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/iotbx/regression/tst_data_manager.py", line 532, in <module> test_model_datatype() File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/iotbx/regression/tst_data_manager.py", line 233, in test_model_datatype dm.get_model().model_as_mmcif(), overwrite=True) File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/model/model.py", line 879, in model_as_mmcif if self.get_restraints_manager() is not None: File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/model/model.py", line 622, in get_restraints_manager self.setup_restraints_manager() File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/model/model.py", line 1042, in setup_restraints_manager self._process_input_model() File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/model/model.py", line 959, in _process_input_model use_neutron_distances = self._pdb_interpretation_params.pdb_interpretation.use_neutron_distances) File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/utils/__init__.py", line 998, in __init__ use_neutron_distances=use_neutron_distances, File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/monomer_library/server.py", line 675, in __init__ use_neutron_distances=use_neutron_distances, File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/monomer_library/server.py", line 689, in convert_all use_neutron_distances=use_neutron_distances, File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/monomer_library/server.py", line 700, in convert_lib_atom entry = cif_types.energy_lib_atom(**dict(row)) File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/monomer_library/cif_types.py", line 49, in __init__ raise_value_error("Floating-point") File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/mmtbx/monomer_library/cif_types.py", line 39, in raise_value_error expected_type, looped_id.split(":",1)[0], val)) ValueError: Floating-point value expected in CIF file: _lib_atom.ion_radius 1,7 Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 7

forcing building boost_python early in SConscript
by Robert Oeffner 17 Nov '18

17 Nov '18

We are working on a newer version of Phaser. The new SConscript file for Phaser imports features from boost_python. Therefore it is currently not straightforward to build it since an empty build directory does not yet contain the boost_python module to be imported by SConscript (or python). So SCons crashes when it reads the new Phaser SCsonscript stating that boost_python cannot be imported. Currently our workaround is to first build CCTBX alone and then subsequently to build Phaser. My question is if it is possible to specify dependencies in cctbx_project/boost_adaptbx/SConscript to force building boost_python early on. On Windows boost_thread and a few other libraries are built before other SConscript files are read. So I have put some Depends(boost_thread, boost_python) andDepends(boost_thread, boost_python_meta_ext) statements into cctbx_project/boost_adaptbx/SConscript. These do seem to force building the boost_python.lib and .dll files. But building the boost_python_meta_ext python module fails with the error: boost_adaptbx\meta_ext.cpp(14): fatal error C1083: Cannot open include file: 'boost_adaptbx/type_id_eq.h' It's possible that I'm not using SCons as it's supposed to be used. Let me know if anyone has a suggestion on how to force building boost_python before other SConscript are read. Many thanks, Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234

5 5

Fwd: Re: indexing diffraction pattern
by Pavel Afonine 14 Nov '18

14 Nov '18

Perhaps someone can reply Wen.. Pavel -------- Forwarded Message -------- Subject: Re: indexing diffraction pattern Date: Sat, 10 Nov 2018 09:30:53 -0500 From: Wen Jiang <jiang12(a)purdue.edu> To: Pavel Afonine <pafonine(a)lbl.gov> Pavel, Sorry for insufficient details. The scenario is that I have a numpy 2D array of float values that represents an image of "diffraction pattern" (generated by an image processing operation, not actual X-ray diffraction). I need to determine the peak positions and then find a 2D lattice (two 2D vectors) going through these peaks. I have my own simple functions that can detect the peaks and find a 2-D lattice to cross the peaks but it often fails when the "diffraction pattern" is not very clean. Thus, I want to find a more robust method that works well when the peaks are relatively weak, not very sharp, and when there are some spurious peaks off the lattice. I think these tasks are exactly what cctbx has solved for indexing X-ray diffraction images. Hope that these explanations help clarify the problem. Thanks, Wen Prof. Wen Jiang Department of Biological Sciences Department of Chemistry (Courtesy) Scientific Director, Purdue Cryo-EM Facility Purdue University Hockmeyer Hall of Structural Biology 240 S. Martin Jischke Drive West Lafayette, IN 47907 http://jiang.bio.purdue.edu On Sat, Nov 10, 2018 at 8:03 AM Pavel Afonine <pafonine(a)lbl.gov <mailto:[email protected]>> wrote: Hi Wen, I'm not sure I fully understand your question... You say a function that takes a numpy array.. Array of what? Then, "a list of diffraction spots" -- these are measured in diffraction experiment. If you could provide more details as to what you are trying to do as well as what exactly the inputs are then I will do my best to see how to do this in CCTBX or forward your questions to CCTBX mailing list so that others can comment. All the best, Pavel On 11/10/18 03:56, Wen Jiang wrote: > Hi Pavel, > I am looking for a function that takes a numpy array and returns a > list of diffraction spots and lattice vectors. I think > cctbx should have such a function but I am not familiar with it. > Can you point me in the right direction? > Thanks, > > Wen > > Prof. Wen Jiang > Department of Biological Sciences > Department of Chemistry (Courtesy) > Scientific Director, Purdue Cryo-EM Facility > Purdue University > Hockmeyer Hall of Structural Biology > 240 S. Martin Jischke Drive > West Lafayette, IN 47907 > http://jiang.bio.purdue.edu >

2 1

Windows woes
by Luc Bourhis 09 Nov '18

09 Nov '18

Hi, Trying to install cctbx on Windows 10, 1. install latest python 2 from Python.org 2. python2 bootstrap.py —builder=cctbxlite —with-python=\path-to-python-from-step-1 It fails because of missing module “builtins”. So I installed pip, then used it to install “future”. Then I had to install “six” with pip. Am I being stupid or cctbx does not install out of the box anymore? Best wishes, Luc J Bourhis

3 3

Python 3 and CCP4
by David Waterman 02 Nov '18

02 Nov '18

Hi folks, It's great to see the momentum building behind the project to migrate to Python 3. From the roadmap <https://github.com/cctbx/cctbx_project/wiki/Python-3-roadmap>, there is clearly still some way to go, but current activity makes me wonder if the proposed deadline of complete migration by 2020 might be beaten easily. What is not clear from the roadmap is whether there are any plans for a post-transition grace period, in which we have switched from writing forwards compatible Python 2 code to writing backwards compatible Python 3 code. Has this been thought about? Or will it be the case that the first commit of Py3-only code seals the deal? I ask because CCP4 has to support a large number of Python projects and these have not all considered the Python 3 switch nearly as seriously as cctbx yet. What I would like to request is a reasonable period of backwards compatibility is ensured to allow other CCP4 projects to catch up, and that this protected period is enshrined within the roadmap or developer guidelines. What do you think? Best wishes -- David

3 2

construct hkl arrays from number of reflections and resolution
by Robert Oeffner 29 Oct '18

29 Oct '18

I would like a function that spits out an HKL array as a function of number of reflections, dmin and dmax. It's quite likely this has already been done somewhere in the CCTBX. Would anyone know if this is the case rather than me having to reinvent the wheel? Many thanks, Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234

2 1

Bootstrap on macOS Mojave
by Graeme.Winter＠Diamond.ac.uk 16 Oct '18

16 Oct '18

Folks, For info: https://github.com/cctbx/cctbx_project/issues/240 wxPython appears to not build on macOS Mojave - possibly updated C++ standards or something who knows? But I guess because we are fixed on an old (4 years, just about [1]) version of wxPython this should not be a surprise. We should probably have some kind of answer for this though - thoughts? Many thanks Graeme ——— [1] https://wxpython.org/news/wxpython-3.0.2.0-release/index.html -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 2

Fwd: cctbx conventions
by Graeme.Winter＠Diamond.ac.uk 08 Oct '18

08 Oct '18

HI Folks, Question from a colleague - any ideas? Many thanks Graeme Begin forwarded message: From: "Bombardi, Alessandro (DLSLtd,RAL,SCI)" <alessandro.bombardi(a)diamond.ac.uk<mailto:[email protected]>> Subject: cctbx conventions Date: 8 October 2018 at 14:54:10 BST To: "Winter, Graeme (DLSLtd,RAL,LSCI)" <Graeme.Winter(a)Diamond.ac.uk<mailto:[email protected]>> Hi Graeme, I am trying to understand the convention used by cctbx, for the anomalous part of the atomic scattering factor. Do you know if it is written as f+f'+if'' with f'' >0 or with f+f'-if'' with f'' >0 can you help me? All the best, Alessandro -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 3

non-standard test naming in mmtbx
by Pearce, N.M. (Nick) 02 Oct '18

02 Oct '18

Hi, I’m trying to use the automatic pytest discovery in mmtbx and I notice that there are a few “libtbx-style” tests, namely "$D/conformation_dependent_library/test_cdl.py", "$D/conformation_dependent_library/test_cdl_esd.py", “$D/conformation_dependent_library/test_rdl.py", "$D/conformation_dependent_library/test_hpdl.py”, that follow the “pytest” naming convention, and are thus picked up by the pytest.discover() function. They then error because pytest cannot run them. There are — as far as I can tell — normal libtbx tests (they are registered in run_tests.py). Is there any reason they do not follow the typical naming convention? Changing their filenames and modifying run_tests.py accordingly seems to lead to the same behaviour when running tests. Is there any reason they should not be renamed from “test_*.py” to “tst_*.py”? Thanks, Nick

3 3

Allocators
by Graeme.Winter＠Diamond.ac.uk 25 Sep '18

25 Sep '18

Hello All, In keeping with the spirit of https://github.com/cctbx/cctbx_project/wiki/cctbx-contributors-guide we would like to revisit the allocator work now Phenix 1.14 is out in the wild - however since we now know that this needs to be done we are interested in other points of view people have on ways to do this, and options people would like to have. It is important to note that we have no intention of changing the default behaviour, but we do want to be able to use customised allocators with flex arrays e.g. for handing out memory of well known and repeatedly used sizes (say, 16 M pixel double arrays) or allocate in specific areas for NUMA systems such as xeon phi. We prototyped this a while back and stepped the changes back out again to enable the phenix release to proceed smoothly, so we have a pretty good idea of the sorts of things we would expect to do however will welcome input if you would like additional features. As a baseline, we are talking as a parameter that PHENIX are no longer supporting CENTOS 5 32 bit (since ISTR this was a constraint before) Any thoughts? Appreciate a number of folks will be travelling back after the PHENIX workshop. Many thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 2

Conda
by Graeme.Winter＠Diamond.ac.uk 24 Sep '18

24 Sep '18

Hello again, Trying to keep threads from getting warped :-) We’re interested in what the plans and timescales are for the move to Conda for a base to cctbx-related projects, and in particular - - what is planned? - when i.e. timescales (rough) - will bootstrap go away? I know from talking to Billy on Wednesday that this is back on the cards again now the PHENIX release is out, but I didn’t ask at the time fo the details… oops! Many thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 2

Fwd: [git/cctbx] master: Python 3 syntax changes (790d9e315)
by Graeme.Winter＠Diamond.ac.uk 21 Sep '18

21 Sep '18

Folks, I expressed some surprise at this & friends however - - this is necessary to make the bootstrap process Python 3 compatible - the timetable for which said this should have been finished 9 months ago (https://github.com/cctbx/cctbx_project/wiki/Python-3-roadmap) - these changes were discussed and agreed (https://github.com/cctbx/cctbx_project/pull/199) The only part of this which was not approved was reverted before the request was merged (I felt that this annotation was helpful as the changes themselves go back ~6 months so people have possibly forgotten the context) Best wishes Graeme Begin forwarded message: From: CCTBX commit <diamondlightsource.jenkins(a)GMAIL.COM<mailto:[email protected]>> Subject: [git/cctbx] master: Python 3 syntax changes (790d9e315) Date: 21 September 2018 at 14:25:29 BST To: <CCTBX-COMMIT(a)JISCMAIL.AC.UK<mailto:[email protected]>> Reply-To: <cctbxbb(a)PHENIX-ONLINE.ORG<mailto:[email protected]>> Repository : ssh://ws133.diamond.ac.uk/cctbx On branch : master ________________________________ commit 790d9e315a2a6ae5a875ba86b461ed570c77280d Author: Markus Gerstel <markus.gerstel(a)diamond.ac.uk<mailto:[email protected]>> Date: Thu May 24 12:16:38 2018 +0100 Python 3 syntax changes Replace the reduce(add, ...) function by sum(). reduce() is no longer in the global namespace in python 3. ________________________________ 790d9e315a2a6ae5a875ba86b461ed570c77280d scitbx/math/__init__.py | 34 ++++++++++++++++------------------ 1 file changed, 16 insertions(+), 18 deletions(-) diff --git a/scitbx/math/__init__.py b/scitbx/math/__init__.py index a13405f92..49e295eb0 100644 --- a/scitbx/math/__init__.py +++ b/scitbx/math/__init__.py @@ -1,11 +1,13 @@ -from __future__ import division, absolute_import +from __future__ import absolute_import, division, print_function + +import math + from scitbx.math.ext import * import scitbx.linalg.eigensystem import scitbx.math.gaussian # implicit import from scitbx import matrix from scitbx.array_family import flex from boost import rational # implicit import -import math gaussian_fit_1d_analytical.__doc__ = """ Fits a gaussian function to a list of points. @@ -88,16 +90,14 @@ def r3_rotation_average_rotation_matrix_from_matrices(*matrices): """ if not matrices: - raise TypeError, "Average of empty sequence" + raise TypeError("Average of empty sequence") - import operator import scitbx.matrix - average = reduce( - operator.add, - [ scitbx.matrix.col( r3_rotation_matrix_as_unit_quaternion( e ) ) - for e in matrices ] - ) + average = sum( + scitbx.matrix.col( r3_rotation_matrix_as_unit_quaternion( e ) ) + for e in matrices + ) return r3_rotation_unit_quaternion_as_matrix( average.normalize() ) @@ -111,13 +111,12 @@ def r3_rotation_average_rotation_via_lie_algebra(matrices, maxiter = 100, conver """ if not matrices: - raise TypeError, "Average of empty sequence" + raise TypeError("Average of empty sequence") if maxiter < 0 or convergence <= 0: - raise ValueError, "Invalid iteration parameters" + raise ValueError("Invalid iteration parameters") from scitbx.math import so3_lie_algebra - import operator conv_sq = convergence * convergence norm = 1.0 / len( matrices ) @@ -125,10 +124,9 @@ def r3_rotation_average_rotation_via_lie_algebra(matrices, maxiter = 100, conver for i in range( maxiter ): inverse = current.inverse() - log_diff = norm * reduce( - operator.add, - [ so3_lie_algebra.element.from_rotation_matrix( matrix = inverse * m ) - for m in matrices ] + log_diff = norm * sum( + so3_lie_algebra.element.from_rotation_matrix( matrix = inverse * m ) + for m in matrices ) if log_diff.norm_sq() < conv_sq: @@ -136,7 +134,7 @@ def r3_rotation_average_rotation_via_lie_algebra(matrices, maxiter = 100, conver current *= log_diff.exponential() - raise RuntimeError, "Iteration limit exceeded" + raise RuntimeError("Iteration limit exceeded") def euler_angles_as_matrix(angles, deg=False): if (deg): @@ -174,7 +172,7 @@ class erf_verification(object): if (self.max_delta < delta): self.max_delta = delta if (delta > self.tolerance): - print x, expected_result, result, delta + print(x, expected_result, result, delta) class minimum_covering_sphere_nd(object): ________________________________ To unsubscribe from the CCTBX-COMMIT list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCTBX-COMMIT&A=1 -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

Re: [cctbxbb] [git/cctbx] master: Python 3 syntax changes (790d9e315)
by Graeme Winter 21 Sep '18

21 Sep '18

Folks, I expressed some surprise at this & friends however - - this is necessary to make the bootstrap process Python 3 compatible - the timetable for which said this should have been finished 9 months ago (https://github.com/cctbx/cctbx_project/wiki/Python-3-roadmap <https://github.com/cctbx/cctbx_project/wiki/Python-3-roadmap>) - these changes were discussed and agreed (https://github.com/cctbx/cctbx_project/pull/199 <https://github.com/cctbx/cctbx_project/pull/199>) The only part of this which was not approved was reverted before the request was merged (I felt that this annotation was helpful as the changes themselves go back ~6 months so people have possibly forgotten the context) Best wishes Graeme > On 21 Sep 2018, at 14:25, CCTBX commit <diamondlightsource.jenkins(a)GMAIL.COM> wrote: > > Repository : ssh://ws133.diamond.ac.uk/cctbx > On branch : master > > > > commit 790d9e315a2a6ae5a875ba86b461ed570c77280d > Author: Markus Gerstel <markus.gerstel(a)diamond.ac.uk> > Date: Thu May 24 12:16:38 2018 +0100 > > Python 3 syntax changes > > Replace the reduce(add, ...) function by sum(). > reduce() is no longer in the global namespace in python 3. > > > > > 790d9e315a2a6ae5a875ba86b461ed570c77280d > scitbx/math/__init__.py | 34 ++++++++++++++++------------------ > 1 file changed, 16 insertions(+), 18 deletions(-) > > diff --git a/scitbx/math/__init__.py b/scitbx/math/__init__.py > index a13405f92..49e295eb0 100644 > --- a/scitbx/math/__init__.py > +++ b/scitbx/math/__init__.py > @@ -1,11 +1,13 @@ > -from __future__ import division, absolute_import > +from __future__ import absolute_import, division, print_function > + > +import math > + > from scitbx.math.ext import * > import scitbx.linalg.eigensystem > import scitbx.math.gaussian # implicit import > from scitbx import matrix > from scitbx.array_family import flex > from boost import rational # implicit import > -import math > > gaussian_fit_1d_analytical.__doc__ = """ > Fits a gaussian function to a list of points. > @@ -88,16 +90,14 @@ def r3_rotation_average_rotation_matrix_from_matrices(*matrices): > """ > > if not matrices: > - raise TypeError, "Average of empty sequence" > + raise TypeError("Average of empty sequence") > > - import operator > import scitbx.matrix > > - average = reduce( > - operator.add, > - [ scitbx.matrix.col( r3_rotation_matrix_as_unit_quaternion( e ) ) > - for e in matrices ] > - ) > + average = sum( > + scitbx.matrix.col( r3_rotation_matrix_as_unit_quaternion( e ) ) > + for e in matrices > + ) > > return r3_rotation_unit_quaternion_as_matrix( average.normalize() ) > > @@ -111,13 +111,12 @@ def r3_rotation_average_rotation_via_lie_algebra(matrices, maxiter = 100, conver > """ > > if not matrices: > - raise TypeError, "Average of empty sequence" > + raise TypeError("Average of empty sequence") > > if maxiter < 0 or convergence <= 0: > - raise ValueError, "Invalid iteration parameters" > + raise ValueError("Invalid iteration parameters") > > from scitbx.math import so3_lie_algebra > - import operator > > conv_sq = convergence * convergence > norm = 1.0 / len( matrices ) > @@ -125,10 +124,9 @@ def r3_rotation_average_rotation_via_lie_algebra(matrices, maxiter = 100, conver > > for i in range( maxiter ): > inverse = current.inverse() > - log_diff = norm * reduce( > - operator.add, > - [ so3_lie_algebra.element.from_rotation_matrix( matrix = inverse * m ) > - for m in matrices ] > + log_diff = norm * sum( > + so3_lie_algebra.element.from_rotation_matrix( matrix = inverse * m ) > + for m in matrices > ) > > if log_diff.norm_sq() < conv_sq: > @@ -136,7 +134,7 @@ def r3_rotation_average_rotation_via_lie_algebra(matrices, maxiter = 100, conver > > current *= log_diff.exponential() > > - raise RuntimeError, "Iteration limit exceeded" > + raise RuntimeError("Iteration limit exceeded") > > def euler_angles_as_matrix(angles, deg=False): > if (deg): > @@ -174,7 +172,7 @@ class erf_verification(object): > if (self.max_delta < delta): > self.max_delta = delta > if (delta > self.tolerance): > - print x, expected_result, result, delta > + print(x, expected_result, result, delta) > > class minimum_covering_sphere_nd(object): > > > To unsubscribe from the CCTBX-COMMIT list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCTBX-COMMIT&A=1 <https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCTBX-COMMIT&A=1>

1 0

Post-PHENIX changes
by Graeme.Winter＠Diamond.ac.uk 21 Sep '18

21 Sep '18

Morning All, I see from http://phenix-online.org/ That PHENIX 1.14 has been released Now, I see at https://github.com/cctbx/cctbx_project/pulls we have a few fairly hefty pull requests parked as “hold for phenix release” - I would certainly not suggest merging all these in one go however probably we should give some thought to stepping through some of them, deciding which ones we should look at first and starting to merge things? I for one would like Nick’s allocator code to come back into play (so we can make use of it for DIALS) Thoughts / opinions? For info: PHENIX folks did you tag the version which was released in PHENIX 1.14 to allow a release branch for .1 etc. releases? Many thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 1

repr for flex arrays
by Graeme.Winter＠Diamond.ac.uk 14 Sep '18

14 Sep '18

Hi Folks, Is there a reason for not having a helpful __repr__ for flex arrays i.e. Grey-Area:002-decide-when graeme$ dials.python Python 2.7.14 (default, Apr 17 2018, 15:02:10) [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from dials.array_family import flex >>> a = flex.double([1, 2, 3]) >>> print a <scitbx_array_family_flex_ext.double object at 0x1095673c0> (I have a feeling I asked this before but cannot find reference) For example >>> print a.as_numpy_array() [ 1. 2. 3.] is more useful… but annoying to type I appreciate a could be “big” but numpy seems to handle this OK even - could __repr__ be punned to as_numpy_array(),__repr__()? Currently it prints the type of the array and its location in ram, which are adequately served by >>> type(a) <class 'scitbx_array_family_flex_ext.double'> >>> print id(a) 4451627968 If I print an array I more often want to see the content and less often the storage class and location (I humbly assert) Thoughts? I know this tends to confuse and annoy people who are new to flex arrays (and veterans who are busy debugging stuff…) Many thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

4 9

Python 3 status
by Graeme.Winter＠Diamond.ac.uk 06 Sep '18

06 Sep '18

Hi Folks, Are there any updates on the progress to Python 3 compatibility? I know there is a project here https://github.com/cctbx/cctbx_project/projects/2 but I have lost the thread of activities since I have been out of the office much of the summer. At the very least I know no progress has been made from this side of the pond as we’ve put this work down for now. i would guess that a lot is on hold while the PHENIX release is done - however I thought I should check in case I have missed anything? Part of why I ask is that there is a PHENIX meeting in Cambridge real soon now (mid September) and I wonder if it is worth folks from DIALS and PHENIX getting together at that time to discuss this? Any thoughts? Thanks & best wishes Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

7 12

Re: [cctbxbb] [git/cctbx] master: re-defining the shift/est convergence limit in terms of crystallographic rather than independent parameters to make it easier to relate to the convergence reported in the CIF; also caching the shifts for further analysis and reporting; (9339fd5f7)
by Nicholas Sauter 05 Sep '18

05 Sep '18

I'm worried about making changes like this to a fundamental toolbox component like normal equations solving. We use this for processing xfel data, and would like the opportunity to test any changes through a pull request, prior to having them committed to the main branch. Nick Sauter Nicholas K. Sauter, Ph. D. Senior Scientist, Molecular Biophysics & Integrated Bioimaging Division Lawrence Berkeley National Laboratory 1 Cyclotron Rd., Bldg. 33R0345 Berkeley, CA 94720 (510) 486-5713 On Wed, Sep 5, 2018 at 5:29 AM, CCTBX commit < diamondlightsource.jenkins(a)gmail.com> wrote: > Repository : ssh://g18-sc-serv-04.diamond.ac.uk/cctbx > On branch : master > > ------------------------------ > > > commit 9339fd5f7d240f0d6f85dd57ce42e796d19608f1 > Author: pcxod <oleg_dolomanov(a)hotmail.com> > Date: Wed Sep 5 13:29:29 2018 +0100 > > re-defining the shift/est convergence limit in terms of > crystallographic rather than independent parameters to make it easier to > relate to the convergence reported in the CIF; also caching the shifts for > further analysis and reporting; > > > ------------------------------ > > > 9339fd5f7d240f0d6f85dd57ce42e796d19608f1 > scitbx/lstbx/normal_eqns_solving.py | 17 +++++++++++------ > 1 file changed, 11 insertions(+), 6 deletions(-) > > diff --git a/scitbx/lstbx/normal_eqns_solving.py > b/scitbx/lstbx/normal_eqns_solving.py > index 660304eab..45f070622 100644 > --- a/scitbx/lstbx/normal_eqns_solving.py > +++ b/scitbx/lstbx/normal_eqns_solving.py > @@ -140,15 +140,20 @@ class iterations(object): > a = self.non_linear_ls.normal_matrix_packed_u() > a.matrix_packed_u_diagonal_add_in_place(value*a. > matrix_packed_u_diagonal()) > > - def do_scale_shifts(self, max_shift_over_esd): > + def do_scale_shifts(self, limit_shift_over_su): > x = self.non_linear_ls.step() > esd = self.non_linear_ls.covariance_matrix().matrix_packed_u_ > diagonal() > - x_over_esd = flex.abs(x/flex.sqrt(esd)) > - max_val = flex.max(x_over_esd) > - if max_val < self.convergence_as_shift_over_esd: > + ls_shifts_over_su = flex.abs(x/flex.sqrt(esd)) > + #max shift for the LS > + self.max_ls_shift_over_su = flex.max(ls_shifts_over_su) > + jac_tr = self.non_linear_ls.actual.\ > + reparametrisation.jacobian_transpose_matching_grad_fc() > + self.shifts_over_su = jac_tr.transpose() * ls_shifts_over_su > + self.max_shift_over_su = flex.max(self.shifts_over_su) > + if self.max_shift_over_su < self.convergence_as_shift_over_esd: > return True > - if max_val > max_shift_over_esd: > - shift_scale = max_shift_over_esd/max_val > + if self.max_ls_shift_over_su > limit_shift_over_su: > + shift_scale = limit_shift_over_su/self.max_ls_shift_over_su > x *= shift_scale > return False > > > ------------------------------ > > To unsubscribe from the CCTBX-COMMIT list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCTBX-COMMIT&A=1 >

2 3

Adding more parameters to xray::scatterer for anharmonic thermal displacements
by Luc J. Bourhis 21 Aug '18

21 Aug '18

Hi, we are working with Michal Chodkiewicz (Warsaw University) to add anharmonic thermal tensors to smtbx refinement. I was thinking to add one pointer to xray::scatterer, which can be NULL. When it is not so, it would point to an object which can compute the value and the derivatives for that thermal factor. That means we would have a base class with virtual methods, but I would say that is our problem! The issue for the rest of the cctbx community is the extra storage of that pointer in that class. Would that be all right? I remember quite a few years ago Ralf saying he wanted to replace the ADP's by a pointer to an external storage to save space. Our idea is along the same line. Best wishes, Luc

3 2

model selection
by Pavel Afonine 16 Aug '18

16 Aug '18

Currently model class has three selection methods: def selection(self, selstr, optional=True): def select(self, selection): def iselection(self, selstr): which is hideously confusing with zero chance to remember, and also inconsistent with the rest. Most objects in cctbx have ".select(self, selection)" methods, while being inconsistent about what they expect: iselection or boolean selection as "selection" argument. Some smart ones though can take both int or bool selections (figure out type automatically, which is trivial). Can we consolidate all three into one ".select()" that would allow taking bool or int or str? Or you think there are philosophical arguments against it? Pavel On 8/16/18 15:00, CCTBX commit wrote: > Repository : ssh://g18-sc-serv-04.diamond.ac.uk/cctbx > On branch : master > > ------------------------------------------------------------------------ > > > commit c98b44997e8eb7bf53ee828f510b933cd5167758 > Author: Oleg Sobolev <osobolev(a)lbl.gov> > Date: Thu Aug 16 15:00:20 2018 -0700 > > Proper use of model class > > > ------------------------------------------------------------------------ > > > c98b44997e8eb7bf53ee828f510b933cd5167758 > mmtbx/model/model.py | 3 +-- > 1 file changed, 1 insertion(+), 2 deletions(-) > > diff --git a/mmtbx/model/model.py b/mmtbx/model/model.py > index 6f2247452..87412f95c 100644 > --- a/mmtbx/model/model.py > +++ b/mmtbx/model/model.py > @@ -2312,8 +2312,7 @@ class manager(object): > sizes = flex.int() > h = self.get_hierarchy() > if(macro_molecule_only): > - asc = h.atom_selection_cache() > - s = asc.selection("protein or nucleotide") > + s = self.selection("protein or nucleotide") > h = h.select(s) > for r in h.residue_groups(): > sizes.append(r.atoms().size())

2 3

Extracting data from MTZ
by David Waterman 14 Aug '18

14 Aug '18

Hi all, I have a script that extracts Fobs and Fcalc from an MTZ file produced by Refmac for some analysis. I was just doing this in a trivial way: fobs = m.extract_reals("F").data fc = m.extract_reals("FC_ALL").data and this worked fine, until I came across a file with missing observations, so that the lengths of fobs and fc do not match. Can anybody advise a better way to do this please? I'm inexperienced with iotbx.mtz. I don't need anything other than the values of the structure factors, but I need each element of fobs and fc to contain the equivalent values. Thanks -- David

3 3

How does Jenkins work?
by Takanori Nakane 28 Jul '18

28 Jul '18

Hi, Why does Jenkins still run tests on my old commits on my deleted (dxtbx-sacla) branch? The test failure is probably caused by my commit dc17165 but I committed a fix 5c7416b and confirmed that it passes the tests. So I merged my topic branch to master and deleted the branch from GitHub. I know how git works but I am new to the practice in the cctbx repository (like using rebase instead of merge). If I am doing something wrong, could you please point it to me? Best regards, Takanori Nakane > <html><head></head><body><table style="width:100%; background-color: > #F59400; color: #FFF; border-radius: 0.5em; margin-bottom: 0.5em"><tr><td > style="margin: 0.5em 1.25em 0.5em 0.7em; font-weight: bold; font-family: > sans-serif; font-size: 110%"><a > href="http://jenkins.diamond.ac.uk:8080/job/cctbx_branch_tests/145/" > style="color:#FFF; text-decoration: none"><big>Build CCTBX Branch Tests is > unstable</big></a></td></tr></table><br/><table style="width:100%; > background-color: #F59400; color: #FFF; border-radius: 0.5em; > margin-bottom: 0.5em"><tr><td style="margin: 0.5em 1.25em 0.5em 0.7em; > font-weight: bold; font-family: sans-serif; font-size: 110%"><a > href="http://jenkins.diamond.ac.uk:8080/job/cctbx_branch_tests/145/testReport" > style="color:#FFF; text-decoration: none">3 out of 940 tests failed (132 > skipped)</a></td></tr></table><table style="margin: 0.5em 0 1em > 2em"><tr><td><pre style="font-size:80%">3 tests failed. > FAILED: > dxtbx.build_dials.modules.cctbx_project.dxtbx.tests.test_regression_images.test_no_multiple_format_understanding[SACLA_MPCCD_Cheetah/run266702-0-subset.h5] > Error Message: > exit code 1 > Stack Trace: > FAILED: > dxtbx.build_dials.modules.cctbx_project.dxtbx.tests.test_regression_images.test_no_multiple_format_understanding[putative_imgCIF_HDF5_mapping/minicbf.h5] > Error Message: > exit code 1 > Stack Trace: > FAILED: > dxtbx.build_dials.modules.cctbx_project.dxtbx.tests.test_regression_images.test_no_multiple_format_understanding[putative_imgCIF_HDF5_mapping/X4_wide_M1S4_1.nxs] > Error Message: > exit code 1 > Stack Trace: > </pre></td></tr></table><table style="width:100%; background-color: > #F59400; color: #FFF; border-radius: 0.5em; margin-bottom: 0.5em"><tr><td > style="margin: 0.5em 1.25em 0.5em 0.7em; font-weight: bold; font-family: > sans-serif; font-size: 110%">Changes since the last successful > build</td></tr></table><table style="margin: 0.5em 0 1em > 2em"><tr><td><table style="padding: 0.1em 0.4em; background-color: > #F59400; color:#fff"><tr><td>145</td></tr></table></td><td>No changes > </td></tr></table><table style="width:100%; background-color: #CCC; color: > #000; border-radius: 0.5em; margin-bottom: 0.5em"><tr><td style="margin: > 0.5em 1.25em 0.5em 0.7em; font-weight: bold; font-family: sans-serif; > font-size: 110%">Build environment</td></tr></table><table style="margin: > 0.5em 0 1em 2em"><tr><td><span > style="font-size:130%"> [ </span>cctbx <strong>71a4ab94a > 2018-07-25 06:47:08 +0900</strong><span > style="font-size:130%"> | </span>DIALS regression > <strong>1924</strong>, PHENIX regression <strong>4419</strong>, DLS > Jenkins scripts <strong>8de8b58 2018-07-04 23:29:22 +0100</strong><span > style="font-size:130%"> ] </span></td></tr></table><span > style="color:#CCC;font-size:80%">mail template revision > 2487</span></body></html> > <br> > <hr> > <p align="center">To unsubscribe from the CCTBX-COMMIT list, click the > following link:<br> > <a > href="https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCTBX-COMMIT&A=1" > target="_blank">https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCTBX-COMMIT&A=1</a> > </p>

4 5

mmtbx wilson_scaling in a small molecule context
by ben.williams＠diamond.ac.uk 23 Jul '18

23 Jul '18

Hello any and all, I was looking to make use of the functionality of wilson_scaling (Wilson plot, etc.) from mmtbx.scaling.data_statistics, in the context of small molecule data. I'm new to mmtbx, so here are two (groups of) questions for the enlightened: 1. Designed, as it is, with MX in mind, wilson_scaling's __init__ demands as an argument the number of protein residues, in order to form an estimate of the contents of the asymmetric unit. If this class were to be used in a small molecule context, what would be a sensible value to use? Would spoofing n_residues with an arbitrary small number (1?) be a robust bodge? Is it ever likely to howl? 2. Being crystals of small molecules, the samples in question may have inversion centres, mirror planes. Having been designed as a tool for MX, will wilson_scaling throw a wobbly when it sees my data? Many thanks, Ben Williams Diamond Light Source -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

subset of reflections from mtz object
by Robert Oeffner 18 Jul '18

18 Jul '18

Hi, I'm not as well versed using iotbx.mtz as I ought to be. I'd like to make a subset of reflections from an mtz file bounded by a chosen dmin and dmax, i.e. a resolution shell of reflections. What is the best way of doing this in python? Many thanks, Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234

2 1

Citation for CCTBX
by Thomas, Jens 17 Jul '18

17 Jul '18

Hi, I'm updating the references printed by AMPLE and would like to cite CCTBX. Is there a recommended citation or way to acknowledge the use of CCTBX in a program? Best wishes, Jens

2 2

cctbx debug build windows
by Pascal 10 Jul '18

10 Jul '18

Hi, How do you build a debug version of cctbx on windows? After getting an environement working with bootstrap, I reconfigured with configure.py --build=debug and it is now failing when linking against python27_d.lib Any advice? Pascal

3 2

Deprecating legacy support (CentOS 5, Windows 32-bit) in CCTBX and Phenix
by Billy Poon 06 Jul '18

06 Jul '18

Hi everyone, The upcoming Phenix 1.14 official release will still have installers for CentOS 5 and Windows 32-bit, but since CentOS is no longer supporting CentOS 5, and RedHat is only supporting RHEL5 through their Extended Life Cycle Support Add-On (until November 30, 2020), we are planning on deprecating CentOS 5 support (32-bit and 64-bit). In addition, we are planning on deprecating 32-bit support for Windows because only a few percent (< 5% for both Windows and CentOS 5) of our Phenix downloads are for 32-bit versions. Please let us know if support for CentOS 5 (32-bit and/or 64-bit) or 32-bit Windows is a requirement for your systems. If so, also please describe your setup. Thanks! -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

2 4

Install OpenBLAS or MKL in base
by Luc Bourhis 22 Jun '18

22 Jun '18

Hi, years ago, I had wrote on this discussion list about a project of mine to dramatically accelerate scitbx.lstbx by using optimised BLAS libraries. That work had been dormant on a branch but now we are planning to move forward with the help of Pascal Parlois who will do the actual coding in the coming months. In order for this new code to be exercised by nightly tests, we, the smtbx people, need to write code so that that optimised BLAS library is installed during bootstrap. Right now, as a stopgap, I used conda to install either OpenBLAS and MKL but that won’t do with the cctbx philosophy. There are basically two choices: either OpenBLAS or MKL. When I started this project, I chose OpenBLAS because MKL was not freely redistributable, and I had got the green light to install it as needed. However now MKL has become completely free, thus becoming a possible choice. MKL is more performant on Intel but OpenBLAS is better on AMD (OpenBLAS actually runs on pretty much any processor out there but we don’t care in the context of cctbx): c.f. Julia people conclusions <https://discourse.julialang.org/t/openblas-is-faster-than-intel-mkl-on-amd-…> Now I realise that most of you may not care one way or another!?! But we won’t move forward before we got the green light… Note that I am taking about installing a BLAS usable from C++ here, not getting a BLAS-accelerated numpy. The latter could be an alternative though. But that would require to modify the bootstrap code to compile a MKL- or OpenBLAS-enabled numpy anyway. Best wishes, Luc J Bourhis

6 22

Fwd: buildbot failure in Phenix on cctbx-ci-ci
by Graeme.Winter＠Diamond.ac.uk 05 Jun '18

05 Jun '18

Hi Folks, Am I alone in finding this kind of message ** very ** annoying I did not touch this file I have no business touching this file If I do fix this then it may cause conflicts for others, which will no doubt annoy them far more than the odd extra space in the code However I am also now to blame for breaking the Phenix build, in the run up to a release This is no a happy place Suggestions? Thanks & cheerio Graeme Begin forwarded message: From: phenix(a)phenix-online.org<mailto:[email protected]> Subject: buildbot failure in Phenix on cctbx-ci-ci Date: 4 June 2018 at 12:11:19 BST To: graeme.winter(a)gmail.com<mailto:[email protected]> Cc: ASBrewster(a)LBL.Gov<mailto:[email protected]>, NWMoriarty(a)LBL.Gov<mailto:[email protected]> Build status: FAILURE Buildslave for this Build: ci Complete logs for all build steps: http://cci-vm-6.lbl.gov:8010/builders/cctbx-ci-ci/builds/5580 Build Reason: The SingleBranchScheduler scheduler named 'cctbx-ci.poll' triggered this build Build Source Stamp: clipper: [branch master] HEAD cbflib: [branch master] HEAD cctbx_project: [branch master] 4de5e7f82b0302632282f78c6dda8b999cf6abf6 annlib_adaptbx: [branch master] HEAD tntbx: [branch master] HEAD ccp4io_adaptbx: [branch master] HEAD boost: [branch master] HEAD gui_resources: [branch master] HEAD Blamelist: graeme-winter If you got an email and you didn't touch this file, it may because someone commited this file before you with an error and hasn't fixed it yet. You might have noticed the error if you ran libtbx.find_clutter on the entire tree. Detailed log of last build step: http://cci-vm-6.lbl.gov:8010/builders/cctbx-ci-ci/builds/5580/steps/test ALL clutter/logs/stdio<http://cci-vm-6.lbl.gov:8010/builders/cctbx-ci-ci/builds/5580/steps/test%20…> Last 80 lines of "test ALL clutter.stdio" iota/command_line/iota_single_image.py: tabs or trailing whitespace=2, #10, #11 *** To clean up please use: libtbx.clean_clutter *** -The Buildbot -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

4 5

Re: [cctbxbb] [git/cctbx] master: Generate libtbx.* wrappers from correct directory (ad736e2c8)
by Billy Poon 01 Jun '18

01 Jun '18

I don't think this is working on macOS. The configuration step is generating error messages like, Ignored: [Errno 2] No such file or directory: <command path> In lines 1640-1643, self.write_dispatcher( source_file=libtbx.env.under_base(os.path.join('bin', ep.name)), target_file=os.path.join('bin', 'libtbx.' + ep.name), ) the source_file argument is always being constructed as <base>/bin/<command> instead of using the directory from self.get_setuptools_script_dir(). Should it be the bin_directory variable instead (e.g. source_file=os.path.join(bin_directory, ep.name))? As a side note, conda will put everything in bin because there is no framework. For access to the window manager, you would call pythonw, which points to a python inside a .app bundle. -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org On Fri, Jun 1, 2018 at 3:15 AM CCTBX commit < diamondlightsource.jenkins(a)gmail.com> wrote: > Repository : ssh://g18-sc-serv-04.diamond.ac.uk/cctbx > On branch : master > > ------------------------------ > > > commit ad736e2c8e65407615a7e6604b0544687ed8058c > Author: Markus Gerstel <markus.gerstel(a)diamond.ac.uk> > Date: Thu May 31 08:27:09 2018 +0100 > > Generate libtbx.* wrappers from correct directory > > Python console scripts don't necessarily land in base/bin, specifically > on MacOS this is not the case. Use setuptools etc. to find the actual > Python console script location and use that directory instead. Fixes > #177 > > Add a function to regenerate all Python console scripts which can be > called from an installer to relocate paths. > > > ------------------------------ > > > ad736e2c8e65407615a7e6604b0544687ed8058c > libtbx/env_config.py | 74 +++++++++++++++++++++++++----- > libtbx/fastentrypoints.py | 112 > ++++++++++++++++++++++++++++++++++++++++++++++ > 2 files changed, 176 insertions(+), 10 deletions(-) > > diff --git a/libtbx/env_config.py b/libtbx/env_config.py > index de5fc72dd..25daa2cb3 100644 > --- a/libtbx/env_config.py > +++ b/libtbx/env_config.py > @@ -1562,23 +1562,77 @@ selfx: > source_file=source_file, > target_file=module_name+"."+command) > > + def get_setuptools_script_dir(): > + ''' > + Find the location of python entry point console_scripts, ie. things > like > + 'pip', 'pytest', ... > + This is different from simple /base/bin, eg. on MacOS. > + > + > https://stackoverflow.com/questions/25066084/get-entry-point-script-file-lo… > + ''' > + from setuptools import Distribution > + from setuptools.command.install import install > + class OnlyGetScriptPath(install): > + def run(self): > + # does not call install.run() by design > + self.distribution.install_scripts = self.install_scripts > + dist = Distribution({'cmdclass': {'install': OnlyGetScriptPath}}) > + dist.dry_run = True # not sure if necessary, but to be safe > + dist.parse_config_files() > + command = dist.get_command_obj('install') > + command.ensure_finalized() > + command.run() > + return dist.install_scripts > + > + def regenerate_entry_point_console_scripts(self, verbose=True): > + ''' > + Creates all console_scripts entry point scripts from scratch and > overwrites existing ones. > + This is intended to be used by installers to relocate the entry point > script paths. > + ''' > + try: > + import distutils.dist > + import libtbx.fastentrypoints # monkeypatches setuptools > + import pkg_resources > + import setuptools.command.easy_install > + except ImportError: > + return > + > + # Prepare generic script generator > + distribution = distutils.dist.Distribution({'name': 'setuptools'}) > + command = setuptools.command.easy_install.easy_install(distribution) > + command.args = ['wheel'] # dummy argument > + command.finalize_options() > + > + # Force regeneration of all known console_scripts > + for pkg_resources_dist in pkg_resources.working_set: > + console_scripts = > pkg_resources_dist.get_entry_map().get('console_scripts') > + if console_scripts: > + if verbose: > + print("Regenerating commands for %s: %s" % ( > + pkg_resources_dist, > + list(console_scripts), > + )) > + command.install_wrapper_scripts(pkg_resources_dist) > + > def generate_entry_point_dispatchers(self): > - # Write indirect dispatcher scripts for all console_scripts entry > points > - # that have existing dispatcher scripts in the base/bin directory, but > - # add a 'libtbx.' prefix. > - base_bin_directory = libtbx.env.under_base('bin') > - if not os.path.isdir(base_bin_directory): > + ''' > + Write indirect dispatcher scripts for all console_scripts entry points > + that have existing dispatcher scripts in the base/bin directory, but > + add a 'libtbx.' prefix. > + ''' > + try: > + import pkg_resources > + bin_directory = get_setuptools_script_dir() > + except ImportError: > + return > + if not os.path.isdir(bin_directory): > return # do not create console_scripts dispatchers, only point to > them > > - base_bin_dispatchers = set(os.listdir(base_bin_directory)) > + base_bin_dispatchers = set(os.listdir(bin_directory)) > existing_dispatchers = filter(lambda f: f.startswith('libtbx.'), > self.bin_path.listdir()) > existing_dispatchers = set(map(lambda f: f[7:], existing_dispatchers)) > entry_point_candidates = base_bin_dispatchers - existing_dispatchers > > - try: > - import pkg_resources > - except ImportError: > - return > entry_points = pkg_resources.iter_entry_points('console_scripts') > entry_points = filter(lambda ep: ep.name in entry_point_candidates, > entry_points) > for ep in entry_points: > diff --git a/libtbx/fastentrypoints.py b/libtbx/fastentrypoints.py > new file mode 100644 > index 000000000..9707f74a3 > --- /dev/null > +++ b/libtbx/fastentrypoints.py > @@ -0,0 +1,112 @@ > +# noqa: D300,D400 > +# Copyright (c) 2016, Aaron Christianson > +# All rights reserved. > +# > +# Redistribution and use in source and binary forms, with or without > +# modification, are permitted provided that the following conditions are > +# met: > +# > +# 1. Redistributions of source code must retain the above copyright > +# notice, this list of conditions and the following disclaimer. > +# > +# 2. Redistributions in binary form must reproduce the above copyright > +# notice, this list of conditions and the following disclaimer in the > +# documentation and/or other materials provided with the distribution. > +# > +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS > +# IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED > +# TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A > +# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT > +# HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, > +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED > +# TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR > +# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF > +# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING > +# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS > +# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. > +''' > +Monkey patch setuptools to write faster console_scripts with this format: > + > + import sys > + from mymodule import entry_function > + sys.exit(entry_function()) > + > +This is better. > + > +(c) 2016, Aaron Christianson > +http://github.com/ninjaaron/fast-entry_points > +''' > +from setuptools.command import easy_install > +import re > +TEMPLATE = '''\ > +# -*- coding: utf-8 -*- > +# EASY-INSTALL-ENTRY-SCRIPT: '{3}','{4}','{5}' > +__requires__ = '{3}' > +import re > +import sys > + > +from {0} import {1} > + > +if __name__ == '__main__': > + sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) > + sys.exit({2}())''' > + > + > +@classmethod > +def get_args(cls, dist, header=None): # noqa: D205,D400 > + """ > + Yield write_script() argument tuples for a distribution's > + console_scripts and gui_scripts entry points. > + """ > + if header is None: > + # pylint: disable=E1101 > + header = cls.get_header() > + spec = str(dist.as_requirement()) > + for type_ in 'console', 'gui': > + group = type_ + '_scripts' > + for name, ep in dist.get_entry_map(group).items(): > + # ensure_safe_name > + if re.search(r'[\\/]', name): > + raise ValueError("Path separators not allowed in script > names") > + script_text = TEMPLATE.format( > + ep.module_name, ep.attrs[0], '.'.join(ep.attrs), > + spec, group, name) > + # pylint: disable=E1101 > + args = cls._get_script_args(type_, name, header, script_text) > + for res in args: > + yield res > + > + > +# pylint: disable=E1101 > +easy_install.ScriptWriter.get_args = get_args > + > + > +def main(): > + import os > + import re > + import shutil > + import sys > + dests = sys.argv[1:] or ['.'] > + filename = re.sub('\.pyc$', '.py', __file__) > + > + for dst in dests: > + shutil.copy(filename, dst) > + manifest_path = os.path.join(dst, 'MANIFEST.in') > + setup_path = os.path.join(dst, 'setup.py') > + > + # Insert the include statement to MANIFEST.in if not present > + with open(manifest_path, 'a+') as manifest: > + manifest.seek(0) > + manifest_content = manifest.read() > + if 'include fastentrypoints.py' not in manifest_content: > + manifest.write(('\n' if manifest_content else '') + > + 'include fastentrypoints.py') > + > + # Insert the import statement to setup.py if not present > + with open(setup_path, 'a+') as setup: > + setup.seek(0) > + setup_content = setup.read() > + if 'import fastentrypoints' not in setup_content: > + setup.seek(0) > + setup.truncate() > + setup.write('import fastentrypoints\n' + setup_content) > > ------------------------------ > > To unsubscribe from the CCTBX-COMMIT list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCTBX-COMMIT&A=1 >

1 0

show() not __str__ - why?
by Graeme.Winter＠Diamond.ac.uk 01 Jun '18

01 Jun '18

Folks, Came up in conversation - how come the motif for printing / providing object info is to call show() and pass a file handle rather than the more pythonic __str__? See e.g. https://github.com/dials/dials/blob/b499c6191caa18e04ddaf190321a3da6c14af55… I know in dials we could do it differently however we inherited the motif from cctbx & I am not sure why things work that way - figured there must be a reason... This is also under discussion under https://github.com/dials/dials/pull/567 but I thought it was worth making sure we knew the reasons why cctbx uses it's own way of doing something... (also it's quite annoying that printing things will give you say space_group object i.e. much like printing type(object), when you want to see it's contents - like printing j and being told it is an int!) Thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

Berkeley Server Downtown (5/29 - 5/31)
by Billy Poon 30 May '18

30 May '18

Dear all, On Tuesday, May 29, our servers will be shut down starting at 5:00 pm Pacific for scheduled maintenance in our building. We expect that the servers will be available by 10:00 am Pacific on Thursday (5/31) at the latest. This will affect, - The Phenix web site (http://phenix-online.org/) - The DIALS West web site (http://dials.lbl.gov/) - The CCI web site (http://cci.lbl.gov/) - The Structural Biology Technology Portal (http://technology.sbkb.org/) Bug reports and emails requesting help should be queued and we will receive them once the servers are back online. We apologize for any inconvenience. -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

1 1

Automated Travis CI for phenix
by Elliot Nelson 25 May '18

25 May '18

Hi I'm looking to run automated integration tests for software, which currently runs from CCTBX, but also calls phenix scripts. These are phenix.fmodel and phenix.maps. Currently I have a docker image which sources CCP4 to get access to CCTBX, but this will not allow me to test stuff reliant on the phenix scripts. As Phenix is not downloadable on command line (CCP4 build can be got through wget command), it is not obvious as the best way to setup up such a docker build on travis CI. I wouldn't want to source a local download of the software as i am unsure of the licesining implications of hosting a copy of the software. Any suggestions would be greatly appreciated? Software (with dockerfile): https://github.com/nelse003/exhaustive_search <https://github.com/nelse003/exhaustive_search>Current docker image: https://hub.docker.com/r/nelse003/ccp4_phenix_docker/ Thanks Elliot DPhil Student, Systems Approaches to Biomedical Science<http://www.sabsidc.ox.ac.uk/> Protein Crystallography<http://www.thesgc.org/groupprofile/9489>, Structural Genomics Consortium, NDM, University of Oxford Oxford Protein Informatics Group<http://opig.stats.ox.ac.uk/>, Department of Statistics, University of Oxford

3 3

mmtbx.f_model.manager.deep_copy() does not work correctly when twinned?
by Keitaro Yamashita 15 May '18

15 May '18

Dear CCTBX developers, I found phenix.maps did not recognize twinning. Clearly, "fmodel" in the following code should be "fmodel_". (underscore missing) https://github.com/cctbx/cctbx_project/blob/41df8498857c186a3f83351090683b3… However, this did not solve the problem. This is due to fmodel_.deep_copy() in line 1530. For example, fmodel_.info().show_rfactors_targets_scales_overall() fmodel_.deep_copy().info().show_rfactors_targets_scales_overall() These two lines show different stats in case of twinning (fmodel_ is mmtbx.twinning.twin_f_model.twin_model_manager object). So deep_copy() is not working correctly. I found that something was wrong with fmodel_ts1.deep_copy() here. https://github.com/cctbx/cctbx_project/blob/master/mmtbx/twinning/twin_f_mo… Does anybody know how to solve this copying issue? Best regards, Keitaro

2 3

mmtbx/examples/simulate_experimental_data.py
by Elliot Nelson 10 May '18

10 May '18

Hi All, I'm trying to generate some simulated mtz files. My main aim is to test a script which looks to determine ligand occupancy (https://github.com/nelse003/exhaustive_search/blob/master/exhaustive_search…) by looking at |Fo-Fc| over the ligand/ ground-state the ligand replaces. To test this I would like to generate mtz files from pdb models with a ligand occupancy that has varied between 0-1, which the above script would hopefully recover the occupancy. I have tried to use mmtbx/examples/simulate_experimental_data.py (https://github.com/cctbx/cctbx-playground/blob/8f23c190a9f8d2ed2b5845385f56… ) to generate an mtz file. So far I have been able to generate mtz files using the simulate_experimental_data.py, these have the columns: H K L FOBS(+) SIGFOBS(+) FOBS(-) SIGFOBS(-) I can use this outputted mtz for running tests of my scripts ,However as this has no PHI columns, I cannot display this in coot. I have tried to add the phi column from the refined mtz of the orignal data using CAD, but have not yet managed to do this succesfully. I feel that this may be an overcomplicated, and potentially superseded way of generating simulated mtz data. Any advice on alternatives would be really useful? Apologies if this is not very clear. Thanks Elliot Nelson

3 4

Jenkins testing on cctbx branches other than master
by markus.gerstel＠diamond.ac.uk 09 May '18

09 May '18

Hello everyone, As requested we now also run Jenkins tests on all branches that are not master. The current settings are as follows: (this is currently exactly the same we do for testing on the master branch) We test the following modules: libtbx, scitbx, cctbx, iotbx, rstbx, fable and dxtbx The following other modules are present and configured: dials_regression, phenix_regression The following modules are not configured: dials, xia2, xfel, iota, prime, simtbx Let me know if you want any changes here. Results will look like this: https://github.com/cctbx/cctbx_project/commits/testbranch If you hover over the green tick mark (or the red cross) of the top 3 commits you see that these are results from cctbx-branch testing. We do generate emails for failed tests and failed builds as usual, but there will be no 'this build is now fixed'/'back to normal' emails. -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 2

Re: [cctbxbb] [Dials-support] Issue with hierarchy.as_pdb_string
by Graeme.Winter＠Diamond.ac.uk 28 Apr '18

28 Apr '18

Morning all, Mike came to the DIALS list with some pdb-reading related issues - however in DIALS land we usually don’t mess with this kind of stuff - is there someone who can help him out here please? Many thanks Graeme On 27 Apr 2018, at 18:14, Wall, Michael E <mewall(a)lanl.gov<mailto:[email protected]>> wrote: Hi, I'm seeing the following issue with a fresh build of dials development source, downloaded within the last hour. There's no error using dials v1-9-1. Is this a bug? Something that's being revised right now? Thanks, Mike Traceback (most recent call last): File "../../src/xenm_new.py", line 609, in <module> ftmp.write(pdb_in.hierarchy.as_pdb_string(crystal_symmetry=pdb_in.input.crystal_symmetry())) File "/home/mewall/packages/dials_devel/modules/cctbx_project/iotbx/pdb/__init__.py", line 1191, in crystal_symmetry self_symmetry = self.crystal_symmetry_from_cryst1() File "/home/mewall/packages/dials_devel/modules/cctbx_project/iotbx/pdb/__init__.py", line 1168, in crystal_symmetry_from_cryst1 return cryst1_interpretation.crystal_symmetry(cryst1_record=line) File "/home/mewall/packages/dials_devel/modules/cctbx_project/iotbx/pdb/cryst1_interpretation.py", line 138, in crystal_symmetry unit_cell=u) File "/home/mewall/packages/dials_devel/modules/cctbx_project/iotbx/pdb/cryst1_interpretation.py", line 87, in space_group_info try: return sgtbx.space_group_info(self.symbol) File "/home/mewall/packages/dials_devel/modules/cctbx_project/cctbx/sgtbx/__init__.py", line 97, in __init__ symbols = space_group_symbols(symbol) Boost.Python.ArgumentError: Python argument types in space_group_symbols.__init__(space_group_symbols, str) did not match C++ signature: __init__(_object*, int space_group_number) __init__(_object*, int space_group_number, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > extension='') __init__(_object*, int space_group_number, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > extension='', std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > table_id='') __init__(_object*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > symbol) __init__(_object*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > symbol, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > table_id='') ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org<http://slashdot.org/>! http://sdm.link/slashdot_______________________________________________ DIALS-support mailing list DIALS-support(a)lists.sourceforge.net<mailto:[email protected]> https://lists.sourceforge.net/lists/listinfo/dials-support -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

scitbx without numpy - still supported?
by Nicholas Devenish 24 Apr '18

24 Apr '18

Hi All, I've been tweaking the code to allow versions of boost higher than 1.63, which has involved poking at the scitbx-numpy bridge code. There seems to be some level of branching in the code for supporting building without numpy - but on the other hand numpy seems to be used exclusively for all flex.type.__init__ functions that I can find? Is building without numpy still supported? Thanks, Nick

5 6

Misuse of libtbx/configure.py
by Robert Oeffner 09 Apr '18

09 Apr '18

Hi, I'm not sure when this has been put into configure.py as I have had taken my eyes of the ball lately regarding Windows builds. But it seems that during bootstrap build the configure.py step is now doing more than just configuring modules. On my PC the excerpt below from stdout shows an error during the configure.py step Processing: "c:\busers\oeffner\nightlybuild\x32\current\modules\dials\libtbx_refresh.py" requires mock>=2.0, has 2.0.0 requirement orderedset is not currently met, package not installed attempting install of orderedset... Collecting orderedset Using cached orderedset-2.0.1.tar.gz Installing collected packages: orderedset Running setup.py install for orderedset: started Running setup.py install for orderedset: finished with status 'error' Complete output from command C:\Busers\oeffner\NightlyBuild\x32\Current\base\bin\python\python.exe -u -c "import setuptools, tokenize;__file__='c:\\users\\oeffner\\appdata\\local\\temp\\pip-build-smhifw\\orderedset\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record c:\users\oeffner\appdata\local\temp\pip-0seb8q-record\install-record.txt --single-version-externally-managed --compile: running install running build running build_py creating build creating build\lib.win32-2.7 creating build\lib.win32-2.7\orderedset copying lib\orderedset\__init__.py -> build\lib.win32-2.7\orderedset running build_ext building 'orderedset._orderedset' extension error: Microsoft Visual C++ 9.0 is required (Unable to find vcvarsall.bat). Get it from http://aka.ms/vcpython27 ---------------------------------------- Command "C:\Busers\oeffner\NightlyBuild\x32\Current\base\bin\python\python.exe -u -c "import setuptools, tokenize;__file__='c:\\users\\oeffner\\appdata\\local\\temp\\pip-build-smhifw\\orderedset\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record c:\users\oeffner\appdata\local\temp\pip-0seb8q-record\install-record.txt --single-version-externally-managed --compile" failed with error code 1 in c:\users\oeffner\appdata\local\temp\pip-build-smhifw\orderedset\ You are using pip version 9.0.1, however version 9.0.3 is available. You should consider upgrading via the 'python -m pip install --upgrade pip' command. install failed. please check manually My first concern with this approach is that the bootstrap process doesn't stop. It carries on and eventually builds the Phenix software suite. This may or may not be a broken build as it is not clear whether the above failure to build orderedset was a showstopper for something. Secondly, I thought that modules would not be installed anywhere else than when building base components or when downloading sources. For a small project it is a nice and sneaky solution to have certain build steps second guessing what dependencies are missing and then installing them "on the fly", if not covertly. But for a large project such as CCTBX with several teams using that as a common framework this becomes messy I would argue. A quick and dirty fix for the above error might be just to put vcvarsall.bat in the PATH environment. But that is not a solution since bootstrap.py has been fashioned to use different versions of Visual C++, some of which are not invoked through a vcvarsall.bat file. I appreciate the need to install additional component modules for python. But would it not be more tidy to install these during the base step of the bootstrap process? Is there any reason why this cannot happen? Regards, Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234

5 5

simtbx - Pilatus example available
by James Holton 08 Apr '18

08 Apr '18

Took most of last week, but I finally got it to work. tst_nanoBragg_pilatus.py will create a Pilatus-like image, complete with bad pixels, window panes, thickness effects, module-to-module misalignment and 0.1 deg of phi rotation. It takes one optional argument, which is the image number in a sequence of 3600. I've processed this simulated data set with xia2 and XDS and found the anomalous signal has the right sign. This required some edits and additions to the simTBX code as well as FormatCBFMini.as_file() that I think fixed a beam center consistency bug. Now if you dxtbx.load() and FormatCBFMiniPilatus.as_file() you get the same beam center in both input and output headers. I also enhanced the header to make it look more like the ones in DIALS regression. A new "SIM.F000" value is now available for marking the "direct beam" on the image itself. I spent most of Friday and all of this morning trying to follow the check-in procedure we laid out during the DIALS workshop. Not sure how, but it suddenly started working just now. I believe I have checked in this code into the simtbx_sandbox branch of cctbx_project. Is Jenkins looking at this yet? Is anyone looking at this yet? -James Holton MAD Scientist

3 3

print dxtbx.model.Scan()
by James Holton 04 Apr '18

04 Apr '18

The following sequence of commands: % source build/setpaths.csh % libtbx.python >>> import dxtbx >>> from dxtbx import model >>> print dxtbx.model.Scan() seems to cause Python itself to core dump. Is this expected? show_stack(1): <stdin>(1) <module> libc backtrace (25 frames, most recent call last): /home/jamesh/projects/cctbx-dev/build/../base/bin/python2.7() [0x400659] /lib64/libc.so.6(__libc_start_main+0xfd) [0x34e961ed1d] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(Py_Main+0xc74) [0x7f93eb5c2b74] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyRun_AnyFileExFlags+0x4c) [0x7f93eb5ad78c] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyRun_InteractiveLoopFlags+0x4e) [0x7f93eb5ad04e] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyRun_InteractiveOneFlags+0x195) [0x7f93eb5ace35] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCode+0x32) [0x7f93eb58d4e2] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x88e) [0x7f93eb58d3ce] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x3ef0) [0x7f93eb589de0] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyFile_WriteObject+0x78) [0x7f93eb503b58] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(+0x958f2) [0x7f93eb5288f2] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyObject_Str+0xa) [0x7f93eb52879a] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(_PyObject_Str+0x7b) [0x7f93eb5286ab] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(+0xb5a1e) [0x7f93eb548a1e] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x43) [0x7f93eb584fd3] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f93eb4dc093] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(+0x5b6cf) [0x7f93eb4ee6cf] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f93eb4dc093] /home/jamesh/projects/cctbx-dev/build/lib/libboost_python.so(+0x2d088) [0x7f93e3f23088] /home/jamesh/projects/cctbx-dev/build/lib/libboost_python.so(boost::python::handle_exception_impl(boost::function0<void>)+0x43) [0x7f93e3f34013] /home/jamesh/projects/cctbx-dev/build/lib/libboost_python.so(+0x2fe08) [0x7f93e3f25e08] /home/jamesh/projects/cctbx-dev/build/lib/libboost_python.so(boost::python::objects::function::call(_object*, _object*) const+0xef) [0x7f93e3f251ff] /home/jamesh/projects/cctbx-dev/build/lib/dxtbx_model_ext.so(boost::python::objects::caller_py_function_impl<boost::python::detail::caller<std::string (*)(dxtbx::model::Scan const&), boost::python::default_call_policies, boost::mpl::vector2<std::string, dxtbx::model::Scan const&> > >::operator()(_object*, _object*)+0x83) [0x7f93db88a7a3] /home/jamesh/projects/cctbx-dev/build/lib/dxtbx_model_ext.so(dxtbx::model::boost_python::scan_to_string(dxtbx::model::Scan const&)+0x319) [0x7f93db871929] /lib64/libc.so.6() [0x34e9632510] Segmentation fault (Python and libc call stacks above)

2 1

Exceptions squashed by easy_mp (revenge of)
by Graeme.Winter＠Diamond.ac.uk 03 Apr '18

03 Apr '18

Folks, Following up on user reports again of errors within easy_mp - all that gets logged is “something went wrong” i.e. Using multiprocessing with 10 parallel job(s) Traceback (most recent call last): File "/home/user/bin/dials-installer/build/../modules/dials/command_line/integrate.py", line 613, in <module> halraiser(e) File "/home/user/bin/dials-installer/build/../modules/dials/command_line/integrate.py", line 611, in <module> script.run() File "/home/user/bin/dials-installer/build/../modules/dials/command_line/integrate.py", line 341, in run reflections = integrator.integrate() File "/home/user/bin/dials-installer/modules/dials/algorithms/integration/integrator.py", line 1214, in integrate self.reflections, _, time_info = processor.process() File "/home/user/bin/dials-installer/modules/dials/algorithms/integration/processor.py", line 271, in process preserve_exception_message = True) File "/home/user/bin/dials-installer/modules/dials/util/mp.py", line 171, in multi_node_parallel_map preserve_exception_message = preserve_exception_message) File "/home/user/bin/dials-installer/modules/dials/util/mp.py", line 53, in parallel_map preserve_exception_message = preserve_exception_message) File "/home/user/bin/dials-installer/modules/cctbx_project/libtbx/easy_mp.py", line 627, in parallel_map result = res() File "/home/user/bin/dials-installer/modules/cctbx_project/libtbx/scheduling/result.py", line 119, in __call__ self.traceback( exception = self.exception() ) File "/home/user/bin/dials-installer/modules/cctbx_project/libtbx/scheduling/stacktrace.py", line 115, in __call__ self.raise_handler( exception = exception ) File "/home/user/bin/dials-installer/modules/cctbx_project/libtbx/scheduling/mainthread.py", line 100, in poll value = target( *args, **kwargs ) File "/home/user/bin/dials-installer/modules/dials/util/mp.py", line 91, in __call__ preserve_exception_message = self.preserve_exception_message) File "/home/user/bin/dials-installer/modules/cctbx_project/libtbx/easy_mp.py", line 627, in parallel_map result = res() File "/home/user/bin/dials-installer/modules/cctbx_project/libtbx/scheduling/result.py", line 119, in __call__ self.traceback( exception = self.exception() ) File "/home/user/bin/dials-installer/modules/cctbx_project/libtbx/scheduling/stacktrace.py", line 86, in __call__ raise exception RuntimeError: Please report this error to dials-support(a)lists.sourceforge.net: exit code = -9 I forget why it was decided that keeping the proper stack trace was a bad thing, but could this be revisited? It would greatly help to see it in the output of the program (if as is the case here I do not have the user data) My email-fu is not strong enough to dig out the previous conversation Cheers Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 6

Re: [cctbxbb] [git/cctbx] master: New command: libtbx.install (914c74bf1)
by Pavel Afonine 13 Mar '18

13 Mar '18

Just out of curiosity: what's your definition of "real world"? Pavel On 3/13/18 03:49, CCTBX commit wrote: > Repository : ssh://g18-sc-serv-04.diamond.ac.uk/cctbx > On branch : master > > ------------------------------------------------------------------------ > > > commit 914c74bf195b5e5d4fa2c4145da3e5f89c7e5872 > Author: Markus Gerstel <markus.gerstel(a)diamond.ac.uk> > Date: Tue Mar 13 10:43:22 2018 +0000 > > New command: libtbx.install > > This downloads and configures from a select list of cctbx/libtbxish > modules in a similar manner to pip in the real world. This will not > update or install any libraries in the base build, but assuming > those libraries are present libtbx.install should work. > > > ------------------------------------------------------------------------ > > > 914c74bf195b5e5d4fa2c4145da3e5f89c7e5872 > libtbx/command_line/install.py | 118 > +++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 118 insertions(+) > > diff --git a/libtbx/command_line/install.py > b/libtbx/command_line/install.py > new file mode 100644 > index 000000000..47ad94eb5 > --- /dev/null > +++ b/libtbx/command_line/install.py > @@ -0,0 +1,118 @@ > +from __future__ import absolute_import, division, print_function > + > +import collections > +import os > +import sys > +from optparse import SUPPRESS_HELP, OptionParser > + > +import procrunner > + > +import libtbx.load_env > +from libtbx.auto_build.bootstrap import Toolbox > + > +# Basically 'pip' for selected libtbx/cctbx modules. > + > + > +def is_source_repository(path): > + return (path / '.git').isdir() or (path / '.svn').isdir() > + > +def run(args): > + parser = OptionParser(usage="libtbx.install [package]", > + description="Installs an additional cctbx > package") > + parser.add_option("-?", action="help", help=SUPPRESS_HELP) > + options, args = parser.parse_args(args) > + > + modules_directory = list(filter(lambda dir: not > is_source_repository(dir), libtbx.env.repository_paths)) > + if not modules_directory: > + sys.exit("No repository path candidate found. Can't install > modules without an installation root.") > + if len(modules_directory) > 1: > + print("More than one repository path candidate found.") > + installation_root = modules_directory[0] > + print("Using %s as installation root" % abs(installation_root)) > + > + packages_to_configure = set() > + > + errors = False > + for package in args: > + if (installation_root / package).isdir(): > + print("Skipping package %s: Directory already exists in > installation root" % package) > + errors = True > + continue > + if package not in warehouse: > + print("Skipping package %s: Never heard of this before" % package) > + errors = True > + continue > + > + downloaded = False > + for mech, call in mechanisms.items(): > + if mech in warehouse[package]: > + print("Attempting to obtain %s using %s..." % (package, > mech), end='') > + sys.stdout.flush() > + if call(abs(installation_root / package), > warehouse[package][mech]): > + assert (installation_root / package).isdir(), "Installation > failed" > + downloaded = True > + print("success") > + break > + else: > + assert not (installation_root / package).isdir(), "Install > mechanism %s did not fail cleanly" % mech > + print("failed") > + if not downloaded: > + print("Skipping package %s: Could not install" % package) > + errors = True > + continue > + > + packages_to_configure.add(package) > + > + if packages_to_configure: > + packages_to_configure = sorted(packages_to_configure) > + print("Configuring packages %s" % ", ".join(packages_to_configure)) > + os.chdir(abs(libtbx.env.build_path)) > + result = procrunner.run_process(['libtbx.configure'] + > packages_to_configure) > + if result['exitcode']: errors = True > + > + if errors: > + sys.exit(1) > + > +def install_git(location, source): > + try: > + result = procrunner.run_process(['git', 'clone', '--recursive', > source, location], print_stdout=False, print_stderr=False) > + if result['exitcode']: > + return False > + Toolbox.set_git_repository_config_to_rebase(os.path.join(location, > '.git', 'config')) > + return True > + except OSError: > + return False # git may not be installed > + > +def install_zip(location, source): > + os.mkdir(location) > + tempfile = os.path.join(location, '.tmp.zip') > + etagfile = os.path.join(location, '..tmp.zip.etag') > + def cleanup(): > + try: os.remove(tempfile) > + except OSError: pass > + try: os.remove(etagfile) > + except OSError: pass > + if Toolbox.download_to_file(source['url'], tempfile) <= 0: > + cleanup() > + os.rmdir(location) > + return False > + Toolbox.unzip(tempfile, location, trim_directory=source.get('trim', 0)) > + cleanup() > + return True > + > +mechanisms = collections.OrderedDict(( > + ('git-auth', install_git), > + ('git-anon', install_git), > + ('http-zip', install_zip), > +)) > + > +warehouse = { > + 'i19': { > + 'git-auth': '[email protected]:/xia2/i19', > + 'git-anon': 'https://github.com/xia2/i19.git', > + 'http-zip': { 'url': > 'https://github.com/xia2/i19/archive/master.zip', 'trim': 1 }, > + }, > +} > + > +if __name__ == '__main__': > + run(sys.argv[1:])

2 1

Re: [cctbxbb] [git/cctbx] master: rename test files, remove them from run_tests (23a4a6fe4)
by Pavel Afonine 05 Mar '18

05 Mar '18

Just to make sure: you are converting to use pytest this particular codes (fable), correct? Pavel P.S.: I'm allergic to pytest. On 3/3/18 07:46, CCTBX commit wrote: > This in preparation for pytestification.

5 8

Revisiting cctbx/iotbx/dxtbx dependencies
by Aaron Brewster 19 Feb '18

19 Feb '18

Hey folks, especially Rob and Luc. So after a quick change to iotbx/libtbx_config, it turns out there's a way to get a super minimum version of cctbx, without configuring dxtbx, and it takes advantage of existing mechanisms: - python bootstrap.py --builder=cctbx hot update - python modules/cctbx_project/libtbx/auto_build/install_base_packages.py openssl python python_compatibility certifi numpy cython png libsvm pytest pythonextra jinja2 junitxml biopython docutils sphinx pyopengl misc freetype matplotlib pillow gettext glib expat fontconfig render pixman tiff cairo gtk fonts wxpython - mkdir build; cd build - ../base/bin/python ../modules/cctbx_project/libtbx/configure.py cctbx iotbx - make Now, some details as to why this works. First, the cctbx builder in boostrap.py doesn't download cbflib so cbflib is never configured. Next, the base step of bootstrap.py is simply a call to install_base_packages.py with a list of required packages. You can just run it standalone and get the packages you need. Here, I ran "python bootstrap.py --builder=cctbx base", killed it after I saw the list of packages it was going to install, then ran install_base_packages manually after removing hdf5 and lz4 (which depends on hdf5) from the list. One could imagine removing other packages depending on your use case. With that, I could run configure.py directly. I can't use cctbx by itself as the cctbx/maptbx/SConscript file has a dependency on iotbx, but configuring cctbx and iotbx together seems to work. The key trick was to make iotbx only depend on dxtbx if dxtbx is explicitly configured. I used a keyword in libtbx_configure added by Luc a bit ago: optional_modules_only_if_explicit_request. I committed that change just now. Outstanding questions: 1. Is there interest in making this a builder? It would be straightforward to add a builder named cctbx_minimal which would just do the above steps 2. Is there further need for making cbflib and hdf5 optional in dxtbx? I purport there is not as the above procedure seems to satisfy the use case, but I can be persuaded otherwise. Thanks, -Aaron

4 5

Re: [cctbxbb] [git/cctbx] master: use larger tolerance (0b81ca799)
by Graeme.Winter＠diamond.ac.uk 11 Feb '18

11 Feb '18

Pavel, This commit did not fix test on OS X High Sierra libtbx.python "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/maptbx/tst_interpolation.py" [FAIL] 1.4s Time: 1.39 Return code: 1 OKs: 0 Standard error: Traceback (most recent call last): File "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/maptbx/tst_interpolation.py", line 128, in <module> exercise(f = f, gx = gx, gy = gy, gz = gz, function_type=function_type) File "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/maptbx/tst_interpolation.py", line 99, in exercise assert approx_equal(it, 0.0, 1.e-4) AssertionError cheers Graeme On 11 Feb 2018, at 00:14, CCTBX commit <diamondlightsource.jenkins(a)GMAIL.COM<mailto:[email protected]>> wrote: Repository : ssh://g18-sc-serv-03.diamond.ac.uk/cctbx On branch : master ________________________________ commit 0b81ca7992f9cf7730eb007aff6e8c30bd2ba71b Author: Pavel <pafonine(a)lbl.gov<mailto:[email protected]>> Date: Sat Feb 10 16:14:51 2018 -0800 use larger tolerance ________________________________ 0b81ca7992f9cf7730eb007aff6e8c30bd2ba71b cctbx/maptbx/tst_interpolation.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/cctbx/maptbx/tst_interpolation.py b/cctbx/maptbx/tst_interpolation.py index 918ba68fd..90dec8a46 100644 --- a/cctbx/maptbx/tst_interpolation.py +++ b/cctbx/maptbx/tst_interpolation.py @@ -96,15 +96,15 @@ def exercise(f,gx,gy,gz,function_type, eps = 1.e-6): if(function_type=="L"): cntr+=1 for it in [tl,gxl,gyl,gzl, tq,gxq,gyq,gzq, tt,gxt,gyt,gzt]: - assert approx_equal(it, 0.0) + assert approx_equal(it, 0.0, 1.e-4) if(function_type=="Q"): cntr+=1 for it in [tq,gxq,gyq,gzq, tt,gxt,gyt,gzt]: - assert approx_equal(it, 0.0) + assert approx_equal(it, 0.0, 1.e-4) if(function_type=="C"): cntr+=1 for it in [tt,gxt,gyt,gzt]: - assert approx_equal(it, 0.0) + assert approx_equal(it, 0.0, 1.e-4) if(abs(tl)>eps and abs(tq)>eps): assert tl >=tq if(function_type=="F"): cntr+=1 -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

Re: [cctbxbb] [Dials-support] indexing success rates
by Aaron Brewster 05 Feb '18

05 Feb '18

Right, typically if I don't know the cell, I'll index a bunch of images with no target, then use cluster.unit_cell on the results to get a consensus cell, then re-index with that cell. For dials.stills_process, the default of fft1d works pretty well generally, but to get the best results, indexing.method_list=fft1d,fft3d might be better. There, if the first method fails the second is tried. real_space_grid_search can also be used. Getting a specific image in a composite file (such as hdf5) isn't a use case I've hit often, so I don't have an easy solution there. It's possible an image range parameter to dials.import would be useful? -Aaron On Sat, Feb 3, 2018 at 1:02 AM, Graeme Winter <graeme.winter(a)gmail.com> wrote: > Hi Aaron > > (and DIALS) > > For stills / single images - should dials.index switch by magic to default > fft1d? > > Of is that the responsibility of e.g. xia2 / cctbx.xfel to worry about? > > It has to be said, more often than not, particularly for screening, we do > not know cell / symmetry a priori (at diamond anyways) > > A related question > > If I have a data block / strong pickle from 1000 stills, how can I index > e.g. image 672? Do I have to split this into 1000 separate things? > > @JH - thanks for this - I think some rigorous testing will be useful here. > > Cheers Graeme > > On 3 Feb 2018, at 08:37, Aaron Brewster <asbrewster(a)lbl.gov> wrote: > > Hi James, thanks. I looked at random image 15 and after trying a bunch of > stuff it indexes fine if I use indexing.method=fft3d. The default for > stills processing is fft1d, so I find this super interesting. > > For what it's worth, I pretty much never process stills without a target > cell. I generated ~30 random images using your script and indexed them, > then I got the unit cells with "dials.show *refined*.json | grep cell". > They were all about the same. So I re-indexed random image 15 using fft1d > but I also specified known_symmetry.cell=50,60,70,90,90,90 and it indexed > fine. > > So, either indexing.method=fft3d or known_symmetry.cell=50,60,70,90,90,90 > is sufficient to index random image 15. Certainly for experimental data I > would recommend using known_symmetry as the results are always better. > However, I'd like to look deeper at fft1d vs fft3d at some point. > > Cc'ing dials-support. > > Thanks! > -Aaron > > > > On Fri, Feb 2, 2018 at 1:35 PM, James Holton <jmholton(a)lbl.gov> wrote: > >> While testing the new simTBX diffraction image simulator Aaron and I >> found that some rather beautiful-looking images still won't index. >> Subsequently, I followed up with mosflm, which does succeed in these cases, >> even without prior knowledge of the cell or space group. >> >> I have now run this 1000 times, varying only the crystal orientation, and >> found 171 cases of dials.stills_process failing to index the simulated >> image whereas mosflm had no trouble, and 5 cases where mosflm failed and >> dials.stills_process succeeded. There were no overlapping cases where both >> programs failed. >> >> I have tarballed up the relevant files here: >> http://bl831.als.lbl.gov/~jamesh/bugreports/dials_index_020218.tgz >> >> The tarball contains a "runme.com" shell script for reproducing these >> results, and also the tst_nanoBragg_forindex.py jiffy for making a test >> image given a provided random-number seed. For example: >> >> libtbx.python ./tst_nanoBragg_forindex.py random 15 >> >> will create a file called noiseimage_001.cbf that dials.stills_process >> cannot index (at least in my hands). For those of you who don't have >> mosflm installed, the tarball incluses a copy, and the "autoindex.com" >> script should run on any linux system. >> >> Any ideas as to why there is this discrepancy? Feels like an opportunity >> for a non-trivial improvement. >> >> Something to ponder over the weekend, I suppose. Note that I recently >> checked in a few bug fixes to simTBX, but this analysis is unaffected by >> them, so you shouldn't have to update your build to reproduce this. >> >> Cheers, >> >> -James Holton >> MAD Scientist >> >> >> >> _______________________________________________ >> cctbxbb mailing list >> cctbxbb(a)phenix-online.org >> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> > ------------------------------------------------------------ > ------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot______ > _________________________________________ > DIALS-support mailing list > DIALS-support(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/dials-support > > >

2 1

indexing success rates
by James Holton 03 Feb '18

03 Feb '18

While testing the new simTBX diffraction image simulator Aaron and I found that some rather beautiful-looking images still won't index. Subsequently, I followed up with mosflm, which does succeed in these cases, even without prior knowledge of the cell or space group. I have now run this 1000 times, varying only the crystal orientation, and found 171 cases of dials.stills_process failing to index the simulated image whereas mosflm had no trouble, and 5 cases where mosflm failed and dials.stills_process succeeded. There were no overlapping cases where both programs failed. I have tarballed up the relevant files here: http://bl831.als.lbl.gov/~jamesh/bugreports/dials_index_020218.tgz The tarball contains a "runme.com" shell script for reproducing these results, and also the tst_nanoBragg_forindex.py jiffy for making a test image given a provided random-number seed. For example: libtbx.python ./tst_nanoBragg_forindex.py random 15 will create a file called noiseimage_001.cbf that dials.stills_process cannot index (at least in my hands). For those of you who don't have mosflm installed, the tarball incluses a copy, and the "autoindex.com" script should run on any linux system. Any ideas as to why there is this discrepancy? Feels like an opportunity for a non-trivial improvement. Something to ponder over the weekend, I suppose. Note that I recently checked in a few bug fixes to simTBX, but this analysis is unaffected by them, so you shouldn't have to update your build to reproduce this. Cheers, -James Holton MAD Scientist

2 1

Fwd: [git/cctbx] master: Allow compiling standalone statically linked executable of Phaser on Windows and MinGW platforms requires: static linking of some boost libraries, additional preprocessor defines for Windows build, additional library specified ...
by Graeme.Winter＠diamond.ac.uk 01 Feb '18

01 Feb '18

Tried sending before got rejected ... Begin forwarded message: From: Graeme Winter <graeme.winter(a)gmail.com<mailto:[email protected]>> Date: 1 February 2018 at 18:51:58 GMT To: cctbxbb(a)PHENIX-ONLINE.ORG<mailto:[email protected]> Cc: CCTBX-COMMIT(a)JISCMAIL.AC.UK<mailto:[email protected]> Subject: Re: [git/cctbx] master: Allow compiling standalone statically linked executable of Phaser on Windows and MinGW platforms requires: static linking of some boost libraries, additional preprocessor defines for Windows build, additional library specified for rstbx build (29fc06f71) Just after this commit all our builds died at diamond Is build bot happy? Curious if local issue Cheers Graeme On 1 Feb 2018, at 18:07, CCTBX commit <diamondlightsource.jenkins(a)GMAIL.COM<mailto:[email protected]>> wrote: Repository : ssh://g18-sc-serv-04.diamond.ac.uk/cctbx On branch : master ________________________________ commit 29fc06f714e1059cb9f4eaac35cc2024178237dc Author: Robert Oeffner <rdo20(a)cam.ac.uk<mailto:[email protected]>> Date: Thu Feb 1 18:07:49 2018 +0000 Allow compiling standalone statically linked executable of Phaser on Windows and MinGW platforms requires: static linking of some boost libraries, additional preprocessor defines for Windows build, additional library specified for rstbx build ________________________________ 29fc06f714e1059cb9f4eaac35cc2024178237dc boost_adaptbx/SConscript | 13 +++++++------ libtbx/SConscript | 4 +++- rstbx/SConscript | 2 +- 3 files changed, 11 insertions(+), 8 deletions(-) diff --git a/boost_adaptbx/SConscript b/boost_adaptbx/SConscript index 8696c1510..b8b58ce2d 100644 --- a/boost_adaptbx/SConscript +++ b/boost_adaptbx/SConscript @@ -32,8 +32,9 @@ print("BOOST_VERSION: %i" %env_etc.boost_version) # any issue. if (libtbx.env.build_options.enable_boost_threads and env_etc.boost_thread_support): - boost_thread_uses_winthreads = (sys.platform == 'win32' and - not env_etc.compiler.endswith('mingw')) + boost_thread_uses_winthreads = (sys.platform == 'win32') + # MinGW compiler 5.3.0 appears to compile boost_threads + # with the same condition as win32. # This environment is to be used latter to build Boost libraries # So we need a clean one without extra library linking. env_boost_shlibs_base = env_base.Copy() @@ -264,7 +265,7 @@ e.g. with "yum install python-dev" or "apt-get install python-dev". libs.append('rt') if boost_thread_uses_winthreads: libs.append('boost_chrono') - env_boost_threads.SharedLibrary( + env_boost_threads.StaticLibrary( target='#lib/boost_thread', source=boost_thread_src, LIBS=libs) @@ -280,7 +281,7 @@ e.g. with "yum install python-dev" or "apt-get install python-dev". # For Windows (c.f. point 2 above) env_boost_system.Append( CPPDEFINES={'BOOST_SYSTEM_DYN_LINK':1}) - env_boost_system.SharedLibrary( + env_boost_system.StaticLibrary( target='#lib/boost_system', source=boost_system_src) @@ -294,7 +295,7 @@ e.g. with "yum install python-dev" or "apt-get install python-dev". os.path.join(env_etc.boost_dist, 'libs', 'chrono', 'src', '*.cpp'))] # For Windows (c.f. point 2 above) env_boost_chrono.Append(CPPDEFINES={'BOOST_CHRONO_DYN_LINK':1}) - env_boost_chrono.SharedLibrary( + env_boost_chrono.StaticLibrary( target='#lib/boost_chrono', source=boost_chrono_src, LIBS=['boost_system']) @@ -308,7 +309,7 @@ e.g. with "yum install python-dev" or "apt-get install python-dev". env_etc.include_registry.append( env=env_boost_threads_test, paths=[env_etc.boost_adaptbx_include]) - env_boost_threads_test.SharedLibrary( + env_boost_threads_test.StaticLibrary( target="#lib/boost_adaptbx_boost_thread_test_ext", source="tests/boost_thread_test.cpp") diff --git a/libtbx/SConscript b/libtbx/SConscript index b4fcc5df9..b956956f2 100644 --- a/libtbx/SConscript +++ b/libtbx/SConscript @@ -364,7 +364,9 @@ For newer versions, please adjust this SConscript. "/D_SECURE_SCL=0", "/D_CRT_SECURE_NO_DEPRECATE", "/DNOMINMAX", - #"/DWIN32", + "/DWIN32", + "/DDLL_EXPORTS", + "/D_WIN32_WINNT=_WIN32_WINNT_WIN7", "/bigobj", # for heavy use of template libraries "/wd4996", # warning C4996: ... was declared deprecated "/wd4068", # warning C4068: unknown pragma diff --git a/rstbx/SConscript b/rstbx/SConscript index 422a9864c..76ba4d1c5 100644 --- a/rstbx/SConscript +++ b/rstbx/SConscript @@ -40,7 +40,7 @@ if (not env_etc.no_boost_python): Import("env_boost_python_ext") env_rstbx_boost_python_ext = env_boost_python_ext.Clone() env_rstbx_boost_python_ext.Prepend( - LIBS=["rstbx", "cctbx", "scitbx_boost_python"]) + LIBS=["rstbx", "ann", "cctbx", "scitbx_boost_python"]) env_rstbx_boost_python_ext.SharedLibrary( target="#lib/rstbx_ext", source="ext.cpp") env_rstbx_boost_python_ext.SharedLibrary( -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 2

Tidy up CCTBX
by Dr. Robert Oeffner 26 Jan '18

26 Jan '18

Hi all, I'm wondering if we could make efforts to refactor code modules in the cctbx_project so as making it have the minimum number of dependencies. This would be helpful to other users of the CCTBX that wishes to develop code on top of it. In the longer term it would also make it easier for us to deploy CCTBX as a Python wheel installer and by implication, spread the gospel of the wonders of the CCTBX. What is currently complicating this is the inclusion of the dxtbx module within the cctbx_project. It is being used by Dials. So my question is if it would be possible to move this project into the Dials code? If the dxtbx was just another small module it wouldn't be much of a problem. But the dxtbx depends on the cbflib and HDF5 components. If you want to build CCTBX from sources that's cumbersome. Why? Because CCTBX as deployed from CCI on http://cci.lbl.gov/cctbx_build/ is only built for a few select platforms. Building CCTBX on a different platform from those with bootstrap compels us to also build cbflib and HDF5 from scratch. Although this is not impossible it does increase the entry barrier for other potential users of the CCTBX who wish to incorporate the CCTBX in their programs: If all a user wants is to calculate allowed origins and symmetry sites for a given space group then an alternative to CCTBX might be preferable rather than having to massively expand their project with the seemingly unrelated HDF5 component. In other words, as it stands now the CCTBX is not straightforward for others to adopt. What do other people think? Regards, Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234

8 16

libtbx.configure crashes RE: new requirements for libtbx_refresh.py
by markus.gerstel＠diamond.ac.uk 23 Jan '18

23 Jan '18

Hi everyone, You may find that with the latest cctbx_project updates your libtbx.configure / make reconf crashes with (...) File "(..)/modules/annlib_adaptbx/libtbx_refresh.py", line 6 print message ^ SyntaxError: invalid syntax make: *** [reconf] Error 1 Solution: Changes are required in annlib_adaptbx, and you do not have the latest version on your system. Please run cd $(libtbx.find_in_repositories annlib_adaptbx)/.. rm -rf annlib_adaptbx git clone https://github.com/cctbx/annlib_adaptbx.git Background: This is related to my earlier email attached below. I couldn't figure out how to update the static tarball of annlib_adaptbx myself, and found it easier to just convert the repository including history. Said repository now lives at https://github.com/cctbx/annlib_adaptbx. Bootstrap has been updated to pull from this location. To avoid duplication the old repository copy we kept at https://github.com/dials/annlib_adaptbx for DIALS release control now points there, too. I note there is another copy at https://github.com/cctbx-xfel/annlib_adaptbx/network where you may also want to change the existing labels to the correct commit tree. Whoever maintains this - let me know if you want any help. Builds are still running but it looks like this is the only bit which may require intervention on your end. -Markus From: cctbxbb-bounces(a)phenix-online.org [mailto:[email protected]] On Behalf Of markus.gerstel(a)diamond.ac.uk Sent: 16 January 2018 09:29 To: cctbxbb(a)phenix-online.org Subject: [cctbxbb] new requirements for libtbx_refresh.py Hello everyone, There will soon be a requirement for all libtbx_refresh.py and run_tests.py of cctbx modules to contain the line from __future__ import absolute_import, division, print_function I have modified those that I have write access to (everything in cctbx_project, all default modules, dials*, labelit*, phenix*, xia2*), but there may be others that I am not aware of/do not use/do not have access to. (While you are at it ideally you would want to make sure these files are valid Python2 and 3 code, but that is not strictly required at this time.) For more information on the patch that requires this change see https://github.com/cctbx/cctbx_project/commit/ecd7192930961fde31c4ba29a049b… -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

GRC / cctbx catch up
by Graeme.Winter＠diamond.ac.uk 16 Jan '18

16 Jan '18

Folks, Who’s going to GRC on diffraction methods this summer? Would it be good to have a cctbx developer’s meeting on the Wednesday afternoon which is “free” - seems like a good opportunity? Cheers Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

new requirements for libtbx_refresh.py
by markus.gerstel＠diamond.ac.uk 16 Jan '18

16 Jan '18

Hello everyone, There will soon be a requirement for all libtbx_refresh.py and run_tests.py of cctbx modules to contain the line from __future__ import absolute_import, division, print_function I have modified those that I have write access to (everything in cctbx_project, all default modules, dials*, labelit*, phenix*, xia2*), but there may be others that I am not aware of/do not use/do not have access to. (While you are at it ideally you would want to make sure these files are valid Python2 and 3 code, but that is not strictly required at this time.) For more information on the patch that requires this change see https://github.com/cctbx/cctbx_project/commit/ecd7192930961fde31c4ba29a049b… -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

How to hack at the autobuild?
by Luc Bourhis 15 Dec '17

15 Dec '17

Hi, I have found a few issues with auto build on recent versions of Fedora. I need to edit files in libtbx/autobuild to fix them but then I want to test that running bootstrap works. But then bootstrap.py always pull from github. I don’t want to push my changes to github until it’s solid though. How do you guys approach that? Best wishes, Luc

3 2

What does get numpy to be install during bootstrap?
by Luc Bourhis 14 Dec '17

14 Dec '17

Hi, I noticed that bootstrap.py does indirectly result in the installation of numpy but a quick swipe of the code did not reveal to me what does get it installed. Since I am sure somebody knows… Best wishes, Luc

2 1

Fwd: [Dials-support] dials.stills_process crash report
by Graeme.Winter＠diamond.ac.uk 13 Dec '17

13 Dec '17

Folks Once again, lack of stack traces in easy_mp is a big annoyance I forget the reasons why printing the actual error was a bad thing, but I do remember a discussion some 18 mo past Not wishing to replay the entire transaction, but could we revisit this? Thanks Graeme Begin forwarded message: From: <Danny.Axford(a)Diamond.ac.uk<mailto:[email protected]>> Subject: [Dials-support] dials.stills_process crash report Date: 12 December 2017 at 17:43:41 GMT To: <dials-support(a)lists.sourceforge.net<mailto:[email protected]>> Cc: martin.appleby(a)diamond.ac.uk<mailto:[email protected]> Perhaps this means something to someone. This is the SACLA data again, it seems to run happily and then hit a problematic frame? Martin and I see the error at pretty much the same point in the dataset, we were both running on local machines rather than the cluster. Cheers, Danny Detector 1 RMSDs by panel: ---------------------------------------------------- | Panel | Nref | RMSD_X | RMSD_Y | RMSD_DeltaPsi | | id | | (px) | (px) | (deg) | ---------------------------------------------------- | 1 | 89 | 0.37442 | 0.48949 | 0.2192 | | 2 | 69 | 0.49627 | 0.4847 | 0.21385 | | 5 | 90 | 0.48267 | 0.59909 | 0.22839 | | 6 | 70 | 0.6094 | 0.55592 | 0.20944 | ---------------------------------------------------- Traceback (most recent call last): File "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20171212/build/../modules/dials/command_line/stills_process.py", line 870, in <module> halraiser(e) File "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20171212/build/../modules/dials/command_line/stills_process.py", line 868, in <module> script.run() File "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20171212/build/../modules/dials/command_line/stills_process.py", line 383, in run preserve_exception_message=True) File "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20171212/modules/cctbx_project/libtbx/easy_mp.py", line 623, in parallel_map result = res() File "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20171212/modules/cctbx_project/libtbx/scheduling/result.py", line 119, in __call__ self.traceback( exception = self.exception() ) File "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20171212/modules/cctbx_project/libtbx/scheduling/stacktrace.py", line 86, in __call__ raise exception RuntimeError: Please report this error to dials-support(a)lists.sourceforge.net<mailto:[email protected]>: exit code = -9 -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org<http://slashdot.org/>! http://sdm.link/slashdot _______________________________________________ DIALS-support mailing list DIALS-support(a)lists.sourceforge.net<mailto:[email protected]> https://lists.sourceforge.net/lists/listinfo/dials-support

3 2

Re: [cctbxbb] cctbxbb Digest, Vol 105, Issue 3
by Mooers, Blaine H.M. (HSC) 08 Dec '17

08 Dec '17

Hi Gianluca , You might try getting the sources for 2.7 from here: https://gist.github.com/0x414A/8b64178e69d9f8331938 Best regards, Blaine Blaine Mooers, Ph.D. Associate Professor Department of Biochemistry and Molecular Biology, College of Medicine University of Oklahoma Health Sciences Center 975 NE 10th Street, BRC 466 Oklahoma City, OK 73104-5419 Faculty webpage X-ray lab (LBSF) COBRE in Structural Biology SSRL UEC Facebook Page SSRL UEC SSRL SAC LCLS UEC EasyPyMOL Molecular Graphics Small Angle Scattering office: (405) 271-8300 lab: (405) 271-8313 e-mail: blaine-mooers(a)ouhsc.edu ________________________________________ From: cctbxbb-bounces(a)phenix-online.org [cctbxbb-bounces(a)phenix-online.org] on behalf of cctbxbb-request(a)phenix-online.org [cctbxbb-request(a)phenix-online.org] Sent: Friday, December 08, 2017 2:00 PM To: cctbxbb(a)phenix-online.org Subject: cctbxbb Digest, Vol 105, Issue 3 Send cctbxbb mailing list submissions to cctbxbb(a)phenix-online.org To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai… or, via email, send a message with subject or body 'help' to cctbxbb-request(a)phenix-online.org You can reach the person managing the list at cctbxbb-owner(a)phenix-online.org When replying, please edit your Subject line so it is more specific than "Re: Contents of cctbxbb digest..." Today's Topics: 1. Importing PyQt4 in litbx.python (Gianluca SANTONI) 2. Re: Importing PyQt4 in litbx.python (markus.gerstel(a)diamond.ac.uk) 3. Re: Importing PyQt4 in litbx.python (Gianluca SANTONI) ---------------------------------------------------------------------- Message: 1 Date: Fri, 8 Dec 2017 13:18:13 +0100 From: Gianluca SANTONI <gianluca.santoni(a)esrf.fr> To: cctbx mailing list <cctbxbb(a)phenix-online.org> Subject: [cctbxbb] Importing PyQt4 in litbx.python Message-ID: <09889a85-3126-af38-05f3-8af190cdfd07(a)esrf.fr> Content-Type: text/plain; charset=utf-8; format=flowed Dear all, I'm having some troubles recently in importing pyqt4 in libtbx.python under Debian8. It used to work with a previous build compiled from the sources, but I don't find a reasonable way to make it work from the bootstrap installation. More details: 1) If I do in libtbx.python sys.path.append('/usr/lib/python2.7/dist-packages/') and "from PyQt4 import QtCore", I get an unidentified symbol: PyUnicodeUCS4_AsLatin1String 2) If I try to compile cctbx as I did in the past "python ../libtbx/configure.py cctbx" no make file is produced. Maybe I'm missing something Thanks everybody for your help. Gian -- Gianluca Santoni, PhD ESRF structural biology group 71 avenue des Martyrs 38027 Grenoble cedex (France) ------------------------------ Message: 2 Date: Fri, 8 Dec 2017 12:40:13 +0000 From: <markus.gerstel(a)diamond.ac.uk> To: <cctbxbb(a)phenix-online.org> Subject: Re: [cctbxbb] Importing PyQt4 in litbx.python Message-ID: <f8d53c3e-c8d8-4941-9540-c9b80f95441d(a)email.android.com> Content-Type: text/plain; charset=utf-8 Hi, There is an issue open on UCS4 which is somewhat related: https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_cctbx_cctbx… We currently don't use UCS4, and your system pyqt4 does. You would have to install (and potentially, compile) it for the cctbx python directly. A 3 minute research from my mobile phone suggests that you might be able to download the sources as tar.gz files and then do something along the lines of cctbx.python -m pip install SIP cctbx.python -m pip install pyqt-source.tar.gz but this is untested, and I haven't used pyqt before. -Markus On 8 Dec 2017 12:19, Gianluca SANTONI <gianluca.santoni(a)esrf.fr> wrote: Dear all, I'm having some troubles recently in importing pyqt4 in libtbx.python under Debian8. It used to work with a previous build compiled from the sources, but I don't find a reasonable way to make it work from the bootstrap installation. More details: 1) If I do in libtbx.python sys.path.append('/usr/lib/python2.7/dist-packages/') and "from PyQt4 import QtCore", I get an unidentified symbol: PyUnicodeUCS4_AsLatin1String 2) If I try to compile cctbx as I did in the past "python ../libtbx/configure.py cctbx" no make file is produced. Maybe I'm missing something Thanks everybody for your help. Gian -- Gianluca Santoni, PhD ESRF structural biology group 71 avenue des Martyrs 38027 Grenoble cedex (France) _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai… -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom ------------------------------ Message: 3 Date: Fri, 8 Dec 2017 14:20:26 +0100 From: Gianluca SANTONI <gianluca.santoni(a)esrf.fr> To: cctbxbb(a)phenix-online.org Subject: Re: [cctbxbb] Importing PyQt4 in litbx.python Message-ID: <820c8b08-d380-dd46-e25a-e9ee9e4b076b(a)esrf.fr> Content-Type: text/plain; charset=utf-8; format=flowed Hi Markus, I could fix everyting by compiling both libraries from source with cctbx.python. Hope this helps others with similar problem too. cctbx.python -m pip install SIP cctbx.python -m pip install pyqt-source.tar.gz did not work. There is actually no PyPi package for sip in python2.7 Cheers Gian Gianluca Santoni, PhD ESRF structural biology group 71 avenue des Martyrs 38027 Grenoble cedex (France) On 08/12/2017 13:40, markus.gerstel(a)diamond.ac.uk wrote: > Hi, > > There is an issue open on UCS4 which is somewhat related: https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_cctbx_cctbx… > > We currently don't use UCS4, and your system pyqt4 does. You would have to install (and potentially, compile) it for the cctbx python directly. > > A 3 minute research from my mobile phone suggests that you might be able to download the sources as tar.gz files and then do something along the lines of > > cctbx.python -m pip install SIP > cctbx.python -m pip install pyqt-source.tar.gz > > but this is untested, and I haven't used pyqt before. > > -Markus > > On 8 Dec 2017 12:19, Gianluca SANTONI <gianluca.santoni(a)esrf.fr> wrote: > Dear all, > I'm having some troubles recently in importing pyqt4 in libtbx.python > under Debian8. > It used to work with a previous build compiled from the sources, but I > don't find a reasonable way to make it work from the bootstrap installation. > > More details: > > 1) If I do in libtbx.python > sys.path.append('/usr/lib/python2.7/dist-packages/') and "from PyQt4 > import QtCore", I get an > > unidentified symbol: PyUnicodeUCS4_AsLatin1String > > 2) If I try to compile cctbx as I did in the past "python > ../libtbx/configure.py cctbx" no make file is produced. Maybe I'm > missing something > > Thanks everybody for your help. > Gian > > -- > Gianluca Santoni, PhD > ESRF structural biology group > 71 avenue des Martyrs > 38027 Grenoble cedex (France) > > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai… > > ------------------------------ _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai… End of cctbxbb Digest, Vol 105, Issue 3 ***************************************

1 0

Re: [cctbxbb] Importing PyQt4 in litbx.python
by markus.gerstel＠diamond.ac.uk 08 Dec '17

08 Dec '17

Hi, There is an issue open on UCS4 which is somewhat related: https://github.com/cctbx/cctbx_project/issues/60 We currently don't use UCS4, and your system pyqt4 does. You would have to install (and potentially, compile) it for the cctbx python directly. A 3 minute research from my mobile phone suggests that you might be able to download the sources as tar.gz files and then do something along the lines of cctbx.python -m pip install SIP cctbx.python -m pip install pyqt-source.tar.gz but this is untested, and I haven't used pyqt before. -Markus On 8 Dec 2017 12:19, Gianluca SANTONI <gianluca.santoni(a)esrf.fr> wrote: Dear all, I'm having some troubles recently in importing pyqt4 in libtbx.python under Debian8. It used to work with a previous build compiled from the sources, but I don't find a reasonable way to make it work from the bootstrap installation. More details: 1) If I do in libtbx.python sys.path.append('/usr/lib/python2.7/dist-packages/') and "from PyQt4 import QtCore", I get an unidentified symbol: PyUnicodeUCS4_AsLatin1String 2) If I try to compile cctbx as I did in the past "python ../libtbx/configure.py cctbx" no make file is produced. Maybe I'm missing something Thanks everybody for your help. Gian -- Gianluca Santoni, PhD ESRF structural biology group 71 avenue des Martyrs 38027 Grenoble cedex (France) _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org http://phenix-online.org/mailman/listinfo/cctbxbb -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

Importing PyQt4 in litbx.python
by Gianluca SANTONI 08 Dec '17

08 Dec '17

Dear all, I'm having some troubles recently in importing pyqt4 in libtbx.python under Debian8. It used to work with a previous build compiled from the sources, but I don't find a reasonable way to make it work from the bootstrap installation. More details: 1) If I do in libtbx.python sys.path.append('/usr/lib/python2.7/dist-packages/') and "from PyQt4 import QtCore", I get an unidentified symbol: PyUnicodeUCS4_AsLatin1String 2) If I try to compile cctbx as I did in the past "python ../libtbx/configure.py cctbx" no make file is produced. Maybe I'm missing something Thanks everybody for your help. Gian -- Gianluca Santoni, PhD ESRF structural biology group 71 avenue des Martyrs 38027 Grenoble cedex (France)

1 0

cctbx.miller problem
by 李选选 05 Dec '17

05 Dec '17

> > Dear cctbx developers, > > I’m Xuanxuan, and I’m using cctbx to develop a crystallography-related algorithm. Recently, I found there is something wrong when I use miller package to generate miller indices for given space group and unit cell parameters. Maybe it’s my mistake, or it’s a bug. > > Here are some details. I just use the following code to generate a list of miller indices for a Fd-3m crystal. > > ms = miller.build_set( > crystal_symmetry=crystal.symmetry( > space_group_symbol='Fd-3m', > unit_cell=(5.4307,5.4307,5.4307,90.00,90.0,90.00)), > anomalous_flag=False, > d_min=0.4) > > And I found some reflections are missing, like 220, 311, which should exist according to the reflection conditions listed in this page http://img.chem.ucl.ac.uk/sgp/large/227az2.htm <http://img.chem.ucl.ac.uk/sgp/large/227az2.htm>. > > Looking forward to your response. > > Thanks, > Xuanxuan

5 4

rotate_translate_map function causes "splinching"
by Pearce, N.M. (Nick) 29 Nov '17

29 Nov '17

Hi all, I’m having a problem with the rotate_translate_map function from cctbx.maptbx. I have a 3D array (a mask of 0s and 1s, in this case) that I’m writing as a CCP4 map file. If I write it out “on the origin” (without running rotate_translate_map), then it’s fine (see attached “before.png”). However, if I apply the rotate_translate_map function and then write it out, odd features appear at the edge of the unit cell (see “after.png”). It is NOT the case that the map is going through the “periodic boundary conditions” of the unit cell (the shift is less than one unit cell, as shown by the structure which is transformed by the same amount). The minimal code is: ——————————— array = cctbx.maptbx.rotate_translate_map( unit_cell=self.unit_cell(), map_data=array, rotation_matrix=scitbx.matrix.rec([1,0,0,0,1,0,0,0,1], (3,3)).elems, translation_vector=(-1.0*scitbx.matrix.rec((X,Y,Z), (3,1))).elems ) # Write the map iotbx.ccp4_map.write_ccp4_map( file_name=f_name, unit_cell=self.unit_cell(), space_group=self.space_group(), map_data=array, labels=flex.std_string(['’]) ) ——————————— where (X,Y,Z) is a tuple for the origin shift. Is there something stupid that I’m missing? Has anyone else had the same problem? The only code for a rotate_translate_map function I could find was actually in maptbx/average_densities.h and I couldn’t see anything obviously “wrong”… though I’m not that familiar with C++. The maps and files can be found at https://www.dropbox.com/sh/xqrsk5lcn603416/AABiVO74g6A_jZMGn_DjxOmka?dl=0 all “grid” files are without translation, and all “ref” files are after translation. both are written out exactly the same (the iotbx.ccp4_map call above). I’m using the cctbx bundled with a ccp4 nightly from 2016-10-22. Thanks, Nick [cid:[email protected]][cid:[email protected]]

2 1

cctbx versions causing indentation crash?
by Magnus Andersson 10 Nov '17

10 Nov '17

Hi all, I have a script that runs fine on my old computer, but crashes on my new computer with the error message: IndentationError: unexpected indent This script is (amongst other things) calling the softwatre Coot such as: subprocess.call(["""coot --no-graphics<<end/n (handle-read-draw-molecule-with-recentre \""""+str(inf)+"""\" 1) (set-refinement-immediate-replacement 1) (set-refine-max-residues 40) (regularize-zone 0 \" \" """+str(res1-2)+""" """+str(res3)+"""\" \") (accept-regularizement) (regularize-zone 0 \" \" """+str(res3)+""" """+str(res2+2)+"""\" \") (accept-regularizement) (write-pdb-file 0 \""""+str(outf)+"""\") (coot-real-exit 0)"""], shell=True) which is ok on my old system. I am trying to find the most likely source of this error. Here are the differences: - OLD cctbx = uses Python 2.7.10, NEW cctbx 2.7.14 - OLD OSX = 10.10.5 (i.e. Yosemite), NEW OSX = 10.11.5 (i.e. El Capitan) - OLD Coot = vers 0.8, NEW Coot = vers 0.8.8 [old Coot install was from Fink but this shouldn’t matter I think] Perhaps someone has encountered something similar or has an educated guess on whether different cctbx versions can cause such errors? Thanks, Magnus #### Magnus Andersson, Docent, Ph.D. Email: magnus.andersson(a)scilifelab.se Phone: 0769469872 Mailing address: Science For Life Laboratory Alpha 6 Box 1031 171 21 Solna Visiting address: Tomtebodavägen 23A 171 65 Solna

2 2

scipy module
by Magnus Andersson 06 Nov '17

06 Nov '17

Hello, I am not normally using cctbx but I’m trying to run a script that imports modules as follows: import sys,io,os import subprocess import numpy import pickle import math import pylab #import pymol import time import scipy import iotbx.pdb import cctbx import mmtbx.model from scipy.optimize import fmin, fmin_cg, leastsq from scipy.interpolate import interp1d import datetime from numpy.random import random_sample from cctbx.array_family import flex from scitbx.math import superpose I have installed cctbx and run with cctbx.python. How can I import scipy into the cctbx python? Cheers, Magnus #### Magnus Andersson, Docent, Ph.D. Email: magnus.andersson(a)scilifelab.se Phone: 0769469872 Mailing address: Science For Life Laboratory Alpha 6 Box 1031 171 21 Solna Visiting address: Tomtebodavägen 23A 171 65 Solna

3 3

Re: [cctbxbb] [git/cctbx] master: libtbx.clean_clutter runs isort in supported locations (612869130)
by Aaron Brewster 26 Oct '17

26 Oct '17

Hi Markus, could isort be added as a dependency instead? If I understand this commit right, when running libtbx.clean_clutter, if .isort.cfg is present and isort is not installed, pip is invoked to install isort. l'm nervous about pip being imported anywhere in our code, especially to install things when running code unrelated to installation. It's important to keep installation of stuff separate from running of stuff. Otherwise, it's impossible to keep dependencies straight. -Aaron On Thu, Oct 26, 2017 at 2:29 AM, CCTBX Commit via DLS Jenkins < graeme.winter(a)gmail.com> wrote: > Repository : ssh://g18-sc-serv-04.diamond.ac.uk/cctbx > On branch : master > > ------------------------------ > > > commit 612869130a6210d49a573a3cff04416a9de67808 > Author: Markus Gerstel <markus.gerstel(a)diamond.ac.uk> > Date: Thu Oct 26 10:29:05 2017 +0100 > > libtbx.clean_clutter runs isort in supported locations > > > ------------------------------ > > > 612869130a6210d49a573a3cff04416a9de67808 > libtbx/command_line/clean_clutter.py | 26 ++++++++++++++++++++++++++ > 1 file changed, 26 insertions(+) > > diff --git a/libtbx/command_line/clean_clutter.py > b/libtbx/command_line/clean_clutter.py > index d49bf4c25..8fc2e53c8 100644 > --- a/libtbx/command_line/clean_clutter.py > +++ b/libtbx/command_line/clean_clutter.py > @@ -32,6 +32,24 @@ def clean_clutter_in(files, tabsize=8): > os.remove(tmpname) > raise > > +def isort(path): > + try: > + import mock > + from isort.main import main > + except ImportError: > + # Install package if necessary > + import pip > + pip.main(['install', 'isort', 'mock']) > + import mock > + from isort.main import main > + with mock.patch.object(sys, 'argv', ['isort', '-y', '-ac', '-vb']): > + oldcwd = os.getcwd() > + try: > + os.chdir(path) > + main() > + finally: > + os.chdir(oldcwd) > + > def run(): > opt_parser = (option_parser( > usage=""" > @@ -60,6 +78,7 @@ by running svn commit.""") > if co.committing and files: > opt_parser.show_help() > exit(1) > + run_isort_in_path = False > if co.committing: > try: > files = list(subversion.marked_for_commit()) > @@ -72,7 +91,14 @@ by running svn commit.""") > else: dir = files[0] > files = [ c.path for c in libtbx.file_clutter.gather([dir]) > if c.is_cluttered(flag_x=False) ] > + if os.path.exists(os.path.join(dir, '.isort.cfg')): > + run_isort_in_path = dir > clean_clutter_in(files, tabsize=co.tabsize) > + if run_isort_in_path: > + try: > + isort(run_isort_in_path) > + except Exception as e: > + print("Did not run isort (%s)" % str(e)) > > if (__name__ == "__main__"): > import sys >

2 2

Re: [cctbxbb] Scons for python3 released
by markus.gerstel＠diamond.ac.uk 20 Oct '17

20 Oct '17

That's 3 weeks from now. If we do a 3 week freeze for every DIALS release, plus for phenix releases etc, cctbx would be trapped in perma-frost :) Feel free to start on it now - if it causes issues I can always pick a stable point in the past to branch off from. -Markus On 10 Oct 2017 6:34 pm, Billy Poon <bkpoon(a)lbl.gov> wrote: Great! There's a DIALS release on October 31, so I can start making changes after that. -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org On Tue, Oct 10, 2017 at 1:54 AM, R. D. Oeffner <rdo20(a)cam.ac.uk<mailto:[email protected]>> wrote: http://scons.org/scons-300-is-available.html Perhaps start using this to easy upgrading cctbx to python 3 -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html<http://www.cimr.cam.ac.uk/investigators/read/index.html> tel: +44(0)1223 763234 _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org<mailto:[email protected]> http://phenix-online.org/mailman/listinfo/cctbxbb -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

5 6

Update to allsvn script?
by Graeme.Winter＠diamond.ac.uk 20 Oct '17

20 Oct '17

Folks, phenix_regression/allsvn assumes if you are using git that you are using git/svn These days probably not true ;o) Viz: cd cctbx_project; git svn rebase Unable to determine upstream SVN information from working tree history ccp4io: At revision 160. ccp4io_adaptbx: At revision 142. cd dials; git svn rebase Unable to determine upstream SVN information from working tree history someone on phenix-land want to have a look at this? Obviously not a blocker but is untidy! Thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 2

Re: [cctbxbb] [git/cctbx] master: Add --print_statement checking capability to libtbx.find_clutter (d499b973b)
by Aaron Brewster 19 Oct '17

19 Oct '17

Hi, should this have been from __future__ import print_function ? Thanks, -Aaron On Thu, Oct 19, 2017 at 1:33 PM, CCTBX Commit via DLS Jenkins < graeme.winter(a)gmail.com> wrote: > Repository : ssh://g18-sc-serv-04.diamond.ac.uk/cctbx > On branch : master > > ------------------------------ > > > commit d499b973be4c0443e607fbad13e495bb450938a3 > Author: Markus Gerstel <markus.gerstel(a)diamond.ac.uk> > Date: Thu Oct 19 21:33:54 2017 +0100 > > Add --print_statement checking capability to libtbx.find_clutter > > > ------------------------------ > > > d499b973be4c0443e607fbad13e495bb450938a3 > libtbx/command_line/find_clutter.py | 12 +++++++++++- > libtbx/file_clutter.py | 23 +++++++++++++++++++---- > 2 files changed, 30 insertions(+), 5 deletions(-) > > diff --git a/libtbx/command_line/find_clutter.py > b/libtbx/command_line/find_clutter.py > index 27baa6d7b..66f4b46eb 100644 > --- a/libtbx/command_line/find_clutter.py > +++ b/libtbx/command_line/find_clutter.py > @@ -13,6 +13,7 @@ def run(args): > only_dos = False > only_future = False > flag_absolute_import = False > + flag_print_statement = False > # > paths = [] > for arg in args: > @@ -34,6 +35,8 @@ def run(args): > only_future = True > elif (arg == "--absolute_import"): > flag_absolute_import = True > + elif (arg == "--print_statement"): > + flag_print_statement = True > else: > paths.append(arg) > if (len(paths) == 0): paths = ["."] > @@ -45,9 +48,12 @@ def run(args): > n_too_many_from_future_import_division = 0 > n_missing_from_future_import_absolute_import = 0 > n_too_many_from_future_import_absolute_import = 0 > + n_missing_from_future_import_print_statement = 0 > + n_too_many_from_future_import_print_statement = 0 > n_bad_indentation = 0 > for info in gather(paths=paths, find_unused_imports=not flag_ni, > - find_bad_indentation=flag_indentation, flag_absolute_import=flag_ > absolute_import): > + find_bad_indentation=flag_indentation, flag_absolute_import=flag_ > absolute_import, > + flag_print_statement=flag_print_statement): > if (info.is_cluttered(flag_x=flag_x)): > n_is_cluttered += 1 > if (info.n_bare_excepts > 0): > @@ -62,6 +68,10 @@ def run(args): > n_missing_from_future_import_absolute_import += 1 > elif info.n_from_future_import_absolute_import > 1: > n_too_many_from_future_import_absolute_import += 1 > + if info.n_from_future_import_print_statement == 0: > + n_missing_from_future_import_print_statement += 1 > + elif info.n_from_future_import_print_statement > 1: > + n_too_many_from_future_import_print_statement += 1 > if (info.bad_indentation is not None) and (flag_indentation) : > n_bad_indentation += 1 > info.show( > diff --git a/libtbx/file_clutter.py b/libtbx/file_clutter.py > index 8af972751..768f06491 100644 > --- a/libtbx/file_clutter.py > +++ b/libtbx/file_clutter.py > @@ -11,9 +11,12 @@ class file_clutter(object): > '^ from [ ]+ __future__ [ ]+ import [ \w,]+ division', re.VERBOSE) > from_future_import_absolute_import_pat = re.compile( > '^ from [ ]+ __future__ [ ]+ import [ \w,]+ absolute_import', > re.VERBOSE) > + from_future_import_print_statement_pat = re.compile( > + '^ from [ ]+ __future__ [ ]+ import [ \w,]+ print_statement', > re.VERBOSE) > > def __init__(self, path, find_unused_imports=False, > - find_bad_indentation=True, flag_absolute_import=False): > + find_bad_indentation=True, flag_absolute_import=False, > + flag_print_statement=False): > self.path = path > self.is_executable = os.access(path, os.X_OK) > self.dos_format = False > @@ -25,6 +28,8 @@ class file_clutter(object): > self.n_from_future_import_division = None > self.flag_absolute_import = flag_absolute_import > self.n_from_future_import_absolute_import = None > + self.flag_print_statement = flag_print_statement > + self.n_from_future_import_print_statement = None > self.bad_indentation = None > self.file_should_be_empty = False > bytes = open(path, "rb").read() > @@ -45,6 +50,7 @@ class file_clutter(object): > if (path.endswith(".py")): > self.n_from_future_import_division = 0 > self.n_from_future_import_absolute_import = 0 > + self.n_from_future_import_print_statement = 0 > py_lines = bytes.splitlines() > self.file_should_be_empty = True > for line in py_lines: > @@ -54,6 +60,8 @@ class file_clutter(object): > self.n_from_future_import_division += 1 > if self.from_future_import_absolute_import_pat.search(line): > self.n_from_future_import_absolute_import += 1 > + if self.from_future_import_print_statement_pat.search(line): > + self.n_from_future_import_print_statement += 1 > ls = line.strip() > if ( ls.startswith("except") > and ls[6:].strip().startswith(":") > @@ -103,7 +111,8 @@ class file_clutter(object): > if (self.has_unused_imports()): > sapp("unused imports=%d" % len(self.unused_imports)) > if self.file_should_be_empty: > - if self.n_from_future_import_division == 0 and > self.n_from_future_import_absolute_import == 0: > + if self.n_from_future_import_division == 0 and > self.n_from_future_import_absolute_import == 0 and \ > + self.n_from_future_import_print_statement: > sapp("file is empty, should be 0 byte file") > else: > sapp("file contains only 'from __future__ import' and should be > empty instead") > @@ -116,6 +125,11 @@ class file_clutter(object): > sapp("missing 'from __future__ import absolute_import'") > elif self.n_from_future_import_absolute_import > 1: > sapp("more than one appearance of 'from __future__ import > absolute_import'") > + if self.flag_print_statement and not self.file_should_be_empty: > + if self.n_from_future_import_print_statement == 0: > + sapp("missing 'from __future__ import print_statement'") > + elif self.n_from_future_import_print_statement > 1: > + sapp("more than one appearance of 'from __future__ import > print_statement'") > if (self.bad_indentation is not None) and (flag_indentation) : > n_tab, n_space = self.bad_indentation > sapp("non-standard indentation: %d space, %d tab" % (n_space, > n_tab)) > @@ -144,11 +158,12 @@ def is_text_file(file_name): > if (name.endswith(extension)): return True > return False > > -def gather(paths, find_unused_imports=False, find_bad_indentation=False, > flag_absolute_import=False): > +def gather(paths, find_unused_imports=False, find_bad_indentation=False, > flag_absolute_import=False, flag_print_statement=False): > clutter = [] > def capp(): > clutter.append(file_clutter(path, find_unused_imports, > - find_bad_indentation=find_bad_indentation, > flag_absolute_import=flag_absolute_import)) > + find_bad_indentation=find_bad_indentation, > flag_absolute_import=flag_absolute_import, > + flag_print_statement=flag_print_statement)) > for path in paths: > if (not os.path.exists(path)): > print >> sys.stderr, "No such file or directory:", path >

2 1

Scons for python3 released
by R. D. Oeffner 10 Oct '17

10 Oct '17

http://scons.org/scons-300-is-available.html Perhaps start using this to easy upgrading cctbx to python 3 -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234

2 1

Re: [cctbxbb] [git/cctbx] master: Comment out simage test for moment as no one interested (apparently) in fixing test (767a45560)
by Nicholas Sauter 05 Oct '17

05 Oct '17

Graeme, Not true that we are not interested in fixing it. Billy just diagnosed the problem as of yesterday. I will provide a fix. Nick Nicholas K. Sauter, Ph. D. Senior Scientist, Molecular Biophysics & Integrated Bioimaging Division Lawrence Berkeley National Laboratory 1 Cyclotron Rd., Bldg. 33R0345 Berkeley, CA 94720 (510) 486-5713 On Wed, Oct 4, 2017 at 10:57 PM, CCTBX Commit via DLS Jenkins < graeme.winter(a)gmail.com> wrote: > Repository : ssh://g18-sc-serv-04.diamond.ac.uk/cctbx > On branch : master > > ------------------------------ > > > commit 767a45560e5f5816d77e9f82e3f846a724967e81 > Author: graeme-winter <graeme.winter(a)gmail.com> > Date: Thu Oct 5 06:57:25 2017 +0100 > > Comment out simage test for moment as no one interested (apparently) > in fixing test > > > ------------------------------ > > > 767a45560e5f5816d77e9f82e3f846a724967e81 > rstbx/run_tests.py | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/rstbx/run_tests.py b/rstbx/run_tests.py > index ea9de29e1..abaf74b4c 100644 > --- a/rstbx/run_tests.py > +++ b/rstbx/run_tests.py > @@ -19,7 +19,7 @@ def run(): > print "OK" > > tst_list = ( > - "$D/simage/tst.py", > + #"$D/simage/tst.py", > "$D/dps_core/tst_iotbx_converter.py", > "$D/diffraction/tst_predict.py", > "$D/diffraction/tst_ewald_sphere.py", >

1 0

SQLite extension library for calculating histograms
by R.D. Oeffner 04 Oct '17

04 Oct '17

Hi, Although peripheral to most developers using the CCTBX I would like to announce an extension library for SQLite that allows creating histogram tables, tables of average values of histogram bins as well as tables of ratios between histogram bins from existing SQLite database files. https://github.com/Oeffner/SQLiteHistograms In our group we often use SQLite databases for statistical analysis of results. This extension library permits you to create histograms of an existing table with one line of SQL. Regards, Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234

2 1

Compiling CCTBX with newer GCC on Mac
by Lance Westerhoff 04 Oct '17

04 Oct '17

Hello All- How does one introduce an alternate gcc compiler for CCTBX on Mac? I must be missing it in the documentation. I don’t want to use the old one in /usr/bin (and instead I want to you my newer one in /opt/gcc). I’ve tried all of the standard methods used by other build systems (change the $PATH, set $CXX and $CC, etc) but to no avail. These environment variables seem to be disregarded. We do use this compiler for building CCTBX on linux without too much trouble, so I think the support should be there. Specifically: mac-build64:build jenkins$ env | grep CXX CXX=/opt/gcc/current/bin/g++ mac-build64:build jenkins$ env | grep CC CC=/opt/gcc/current/bin/gcc mac-build64:build jenkins$ which gcc /opt/gcc/current/bin/gcc mac-build64:build jenkins$ g++ -v Using built-in specs. COLLECT_GCC=g++ COLLECT_LTO_WRAPPER=/opt/gcc/gcc-6.3.0-08092017/libexec/gcc/x86_64-apple-darwin11.4.2/6.3.0/lto-wrapper Target: x86_64-apple-darwin11.4.2 Configured with: ../gcc-6.3.0/configure --prefix=/opt/gcc/gcc-6.3.0-08092017 --enable-languages=c,c++,fortran,objc --with-gmp=/opt/gcc/gcc-6.3.0-08092017 --with-mpfr=/opt/gcc/gcc-6.3.0-08092017 --with-mpc=/opt/gcc/gcc-6.3.0-08092017 --enable-libgomp --disable-multilib Thread model: posix gcc version 6.3.0 (GCC) mac-build64:build jenkins$ /opt/python/current/bin/python ../modules/cctbx_project/libtbx/configure.py cctbx Python: 2.7.9 "/opt/python/current/bin/python" Compiler: default .... This script is only applicable to Linux - exiting. Done. mac-build64:build jenkins$ make ./bin/libtbx.scons -j "`./bin/libtbx.show_number_of_processors`" scons: Reading SConscript files ... On MacOS, using clang 4.2.0 Xcode 4.6.3 Build version 4H1503 ... Thanks for your help! -Lance ____________________ Lance M. Westerhoff, Ph.D. President and General Manager QuantumBio Inc. WWW: http://www.quantumbioinc.com Email: lance(a)quantumbioinc.com

3 7

Should we enable boost threads in bootstrap?
by Graeme.Winter＠diamond.ac.uk 28 Sep '17

28 Sep '17

Afternoon all, Should we do this? Any opinions? Could be useful for threads in a semi-portable way... Thanks & cheerio Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

7 16

Should we enable boost threads in bootstrap?
by Tristan Croll 27 Sep '17

27 Sep '17

Sorry for not replying directly to the existing thread, but I've only just subscribed to cctbxbb. Since I started the discussion of the Global Interpreter Lock (GIL) yesterday, I thought I should give a quick run-down of why it's important and where/how it can be released. In brief, the much-maligned GIL exists because much of the Python API proper (in particular, reference counting) is not thread-safe. Its purpose is to ensure that only one thread can ever be using Python objects at any given time. In practice this means that a naive implementation of Python threads gives you the worst of all possible worlds: parallel logic, but slower-than-single-threaded performance. Python regularly tries to swap between all threads it has running, but only the one that currently holds the GIL can go forward. So if one thread runs a method that takes 10 seconds without releasing the GIL, *all* threads will hang for 10 seconds. To be clear, this *doesn't* prevent you from using C++ threads, as long as those threads are not acting on Python objects. The thing is, in most production code the heavy-duty computation is *not* done on Python objects - it's done in C++ on objects that Python simply holds a pointer to. All such functions can safely release the GIL as long as they reacquire it before returning to Python. In the context of the example above, that means all other threads are able to continue on doing their own thing while that 10-second function is running. So why use threads rather than multiprocessing? Three key reasons: - threads are trivial to set up and run in the same way on Linux, macOS and Windows - threads share memory by default, making life much easier when they need to communicate regularly - It turns out that OpenCL/CUDA do not play at all well with forked processes. A real-world example: in ISOLDE I'm currently bringing together two key packages with their own Python APIs: ChimeraX for the molecular graphics, OpenMM for MD simulation. Since this is an interactive application, speed is of the essence. Graphics performance needs to be independent of simulation performance (to allow the user to rotate/translate etc. smoothly no matter how fast the simulation runs) so parallelism is mandatory. There is constant back-and-forth communication needed (the simulation needs to update coordinates; interactions need to be sent back to the simulation), which easier/faster in shared memory. The simulation is running on a GPU. My initial implementation used Python's multiprocessing module applying os.fork() - this worked in Linux under the very specific circumstances where the GPU had not previously been used for OpenCL or CUDA by the master process, but failed on the Mac and is of course impossible on Linux. Switching from multiprocessing to threading (with no other changes to my code) gave me an implementation that works equally well on Linux and Mac with no OS-specific code, and should work just as well on Windows once I get around to doing a build. The performance is effectively equal to what I was getting from multiprocessing, since all the major libraries I'm using (ChimeraX, OpenMM, NumPy) release the GIL in all their C++ calls. At the moment, though, CCTBX functions don't - but there's no reason why they can't. See https://wiki.python.org/moin/boost.python/HowTo#Multithreading_Support_for_…. A simple toy example using Python threads to speed up a heavyish computation: from multiprocessing.pool import ThreadPool import numpy from math import ceil from time import sleep def f(a, target, start_i, end_i): arr = a[start_i:end_i] target[start_i:end_i] = numpy.exp(numpy.cos(arr)+numpy.sin(arr)) def test_threads(a, num_threads): l = len(a) ret = numpy.empty(l, a.dtype) stride = int(ceil(l/num_threads)) with ThreadPool(processes=num_threads) as p: for i in range(num_threads): start = stride*i end = stride*(i+1) if end > l: end = l p.apply_async(f, (a, ret, start, end)) p.close() p.join() return ret a = numpy.random.rand(50000000) target = numpy.empty(len(a), a.dtype) %timeit f(a, target, 0, len(a)) 2.35 s ± 102 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) %timeit test_threads(a, 1): 2.55 s ± 30.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) %timeit test_threads(a, 2): 1.5 s ± 5.69 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) %timeit test_threads(a, 3): 1.2 s ± 111 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

1 1

rstbx test is failing without indication in test_all_parallel
by Oleg Sobolev 22 Sep '17

22 Sep '17

Dear colleagues, Accidentally I found out that this command: rstbx.simage.solver d_min=5 lattice_symmetry=P422 intensity_symmetry=P4 index_and_integrate=True multiprocessing=True finishes with traceback. It is run as part of phenix_regression/misc/tst_rstbx.csh The second problem is that test_all_parallel script fails to detect an actual failure. It would be great if somebody who is in charge of rstbx and testing could look at this. Best regards, Oleg Sobolev.

7 9

Massive commit just now
by Graeme.Winter＠diamond.ac.uk 08 Sep '17

08 Sep '17

Folks, Earlier a branch which was fairly long lived was merged in perhaps a non-ideal way, apparently replaying every commit from around July to now. While this is confusing and has (a little) messed with history, it has not actually broken anything... However, if you have a big branch you want to merge, I think a pull request makes this a little more obvious Just some background to what happened... Cheers Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 3

rna3p error
by Haiguang Liu 30 Aug '17

30 Aug '17

Dear All, I recently run into an error while processing DNA molecules using pdb_interpreter. The error message is as the following: File "/Users/hgliu/test/cctbx/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 5419, in __init__ restraints_loading_flags=restraints_loading_flags) File "/Users/hgliu/test/cctbx/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 3221, in __init__ log=log, File "/Users/hgliu/test/cctbx/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2507, in __init__ m_j=mm, File "/Users/hgliu/test/cctbx/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 1693, in get_lib_link return mon_lib_srv.link_link_id_dict["rna3p"] KeyError: 'rna3p' After searching through the threads, I found the same error has been reported, but no solution posted. Any suggestion is appreciated. Best, haiguang

2 2

Berkeley servers are back in service
by Billy Poon 03 Aug '17

03 Aug '17

Hi everybody, The servers at Berkeley are now available. Please let us know of any issues. Thanks! -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

1 0

Emergency shutdown
by Nigel Moriarty 02 Aug '17

02 Aug '17

Folks Phenix servers are shutting down due to a grass fire in the area. Cheers Nigel --- Nigel W. Moriarty Building 33R0349, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709 Email : NWMoriarty(a)LBL.gov Fax : 510-486-5909 Web : CCI.LBL.gov

1 0

Changing boost versions
by Nicholas Devenish 01 Aug '17

01 Aug '17

Hi All, Graeme's thread reminded me to ask about something I've been struggling with; is there a built-in/easy way to specify a different installation of boost to use? The build seems to construct lib boost_python manually, for some reason, so I don't know how easy it is to disentangle. Thanks, Nick

3 2

Pillow has landed
by markus.gerstel＠diamond.ac.uk 25 Jul '17

25 Jul '17

Hello everyone, Bootstrap now installs Pillow as the new replacement library of the Python Imaging Library (PIL), which has been deprecated/abandonware for a couple of years now. You can update your local installation by running libtbx.python -m pip install Pillow I ran tests locally, and they appeared to run fine. Although this is pretty much a drop-in replacement there may be some import statements which need updating. If you do come across problems you may find this to be of use: https://pillow.readthedocs.io/en/4.2.x/porting.html -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

smtbx.refine not working correctly?
by Graeme.Winter＠diamond.ac.uk 20 Jul '17

20 Jul '17

Not sure who to ask... Tried smtbx.refine from latest phenix & dials builds, both give [gw56@cs03r-sc-serv-16 Refine]$ smtbx.refine lcy.ins lcy.hkl lcy_smtbx.res Traceback (most recent call last): File "/dls_sw/apps/dials/dials-v1-6-4/build/../modules/cctbx_project/smtbx/command_line/refine.py", line 152, in <module> run(args, options) File "/dls_sw/apps/dials/dials-v1-6-4/build/../modules/cctbx_project/smtbx/command_line/refine.py", line 67, in run strictly_shelxl=False) File "/dls_sw/apps/dials/dials-v1-6-4/modules/cctbx_project/smtbx/refinement/__init__.py", line 10, in from_shelx return _(cls, *args, **kwds) File "/dls_sw/apps/dials/dials-v1-6-4/modules/cctbx_project/iotbx/shelx/__init__.py", line 59, in smtbx_refinement_model_from builder.weighting_scheme, AttributeError: 'smtbx_builder' object has no attribute 'weighting_scheme' [gw56@cs03r-sc-serv-16 Refine]$ module load phenix/1.12 Setting up phenix environment and PATH... [gw56@cs03r-sc-serv-16 Refine]$ smtbx.refine lcy.ins lcy.hkl lcy_smtbx.res Traceback (most recent call last): File "/dls_sw/apps/phenix/1.12/phenix-1.12-2829/build/../modules/cctbx_project/smtbx/command_line/refine.py", line 152, in <module> run(args, options) File "/dls_sw/apps/phenix/1.12/phenix-1.12-2829/build/../modules/cctbx_project/smtbx/command_line/refine.py", line 67, in run strictly_shelxl=False) File "/dls_sw/apps/phenix/1.12/phenix-1.12-2829/modules/cctbx_project/smtbx/refinement/__init__.py", line 10, in from_shelx return _(cls, *args, **kwds) File "/dls_sw/apps/phenix/1.12/phenix-1.12-2829/modules/cctbx_project/iotbx/shelx/__init__.py", line 59, in smtbx_refinement_model_from builder.weighting_scheme, AttributeError: 'smtbx_builder' object has no attribute 'weighting_scheme' who's best to ask about this? Thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 3

New error appeared recently, do not know who it belongs to...
by Graeme.Winter＠diamond.ac.uk 17 Jul '17

17 Jul '17

Exhibit A: libtbx.python "/Users/graeme/svn/cctbx/modules/cctbx_project/rstbx/simage/tst.py" rstbx.simage.explore_completeness d_min=10 rstbx.simage.explore_completeness d_min=10 intensity_symmetry=P4 use_symmetry=True multiprocessing=True rstbx.simage.solver d_min=5 rstbx.simage.solver d_min=5 lattice_symmetry=R32:R intensity_symmetry=R3:R rstbx.simage.solver d_min=5 lattice_symmetry=R32:R intensity_symmetry=P1 rstbx.simage.solver d_min=5 lattice_symmetry=P422 intensity_symmetry=P4 index_and_integrate=True multiprocessing=True Traceback (most recent call last): File "/Users/graeme/svn/cctbx/modules/cctbx_project/rstbx/simage/tst.py", line 273, in <module> run(args=sys.argv[1:]) File "/Users/graeme/svn/cctbx/modules/cctbx_project/rstbx/simage/tst.py", line 269, in run exercise_solver() File "/Users/graeme/svn/cctbx/modules/cctbx_project/rstbx/simage/tst.py", line 255, in exercise_solver "index_and_integrate=True", "multiprocessing=True"]) File "/Users/graeme/svn/cctbx/modules/cctbx_project/rstbx/simage/tst.py", line 213, in run command=cmd, stdout_splitlines=False).raise_if_errors().stdout_buffer File "/Users/graeme/svn/cctbx/modules/cctbx_project/libtbx/easy_run.py", line 39, in raise_if_errors raise Error(msg) RuntimeError: child process stderr output: command: 'rstbx.simage.solver d_min=5 lattice_symmetry=P422 intensity_symmetry=P4 index_and_integrate=True multiprocessing=True' Traceback (most recent call last): File "/Users/graeme/svn/cctbx/build/../modules/cctbx_project/rstbx/command_line/simage.solver.py", line 5, in <module> run(args=sys.argv[1:]) File "/Users/graeme/svn/cctbx/modules/cctbx_project/rstbx/simage/solver.py", line 1161, in run return run_fresh(args) File "/Users/graeme/svn/cctbx/modules/cctbx_project/rstbx/simage/solver.py", line 1145, in run_fresh process(work_params, i_calc) File "/Users/graeme/svn/cctbx/modules/cctbx_project/rstbx/simage/solver.py", line 1088, in process process_core(work_params, i_calc.p1_anom, reindexing_assistant, image_mdls) File "/Users/graeme/svn/cctbx/modules/cctbx_project/rstbx/simage/solver.py", line 985, in process_core work_params, reindexing_assistant, image_mdls, usables) File "/Users/graeme/svn/cctbx/modules/cctbx_project/rstbx/simage/solver.py", line 883, in build_image_cluster raise RuntimeError("Insufficient connectivity between images.") RuntimeError: Insufficient connectivity between images. usr+sys time: 0.78 seconds, ticks: 777467, micro-seconds/tick: 1.003 wall clock time: 6.99 seconds OS X & RHEL6 Any volunteers? Cheers Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 5

Re: [cctbxbb] duplicate spam
by Aaron Brewster 10 Jul '17

10 Jul '17

More commit spam incoming. This is from merging a bunch of commits from an LCLS beamtime into the main branch. Sorry for the flood. -Aaron On Tue, Jun 27, 2017 at 12:30 AM, Markus Gerstel < markus.gerstel(a)diamond.ac.uk> wrote: > Due to rewriting git history there are now a number of duplicate mails > coming through here. > > Apologies. > > > > -Markus > > > > -- > > This e-mail and any attachments may contain confidential, copyright and or > privileged material, and are for the use of the intended addressee only. If > you are not the intended addressee or an authorised recipient of the > addressee please notify us of receipt by returning the e-mail and do not > use, copy, retain, distribute or disclose the information in or attached to > the e-mail. > Any opinions expressed within this e-mail are those of the individual and > not necessarily of Diamond Light Source Ltd. > Diamond Light Source Ltd. cannot guarantee that this e-mail or any > attachments are free from viruses and we cannot accept liability for any > damage which you may sustain as a result of software viruses which may be > transmitted in or with the message. > Diamond Light Source Limited (company no. 4375679). Registered in England > and Wales with its registered office at Diamond House, Harwell Science and > Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom > >

2 2

Building cctbx & friends with intel compilers
by Graeme.Winter＠diamond.ac.uk 29 Jun '17

29 Jun '17

Greetings all, Do we have a recipe for compiling cctbx etc with icc? To pick a random example, this version: icc (ICC) 17.0.4 20170411 Copyright (C) 1985-2017 Intel Corporation. All rights reserved. Thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 2