- cctbxbb - phenix-online.org

Failing tests on python3 builds
by richard.gildea＠diamond.ac.uk 31 Jul '19

31 Jul '19

Hi, Whilst looking into test failures reported against a pull request I currently have open, I noticed that there are several (9) failing tests on the Python 3 build. It appears that Markus reported these 3 weeks ago - is anyone looking into these failures? Having builds permanently reporting failure isn't particularly helpful as it discourages people from looking at the results, and will likely lead to further Python 3 regressions being introduced into the codebase. https://dev.azure.com/cctbx/cctbx_project/_build/results?buildId=240 https://github.com/cctbx/cctbx_project/issues Cheers, Richard Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 4

"duplicate" cif categories
by wtempel 07 Jul '19

07 Jul '19

Hi, wwpdb onedep recently rejected a file that I had manipulated (but not validated, I admit) in IOTBX. The error was "duplicate categories", or a similar complaint. I had added _reflns.* items related to data processing to a phenix.refine cif file that already included a _reflns.B_iso_Wilson_estimate. Merely running sort() on the relevant block made the error go away, but who knows what side effects there may have been. Does the CCTBX ecosystem provide some higher-level management of cif categories? Thanks. Wolfram

1 0

align density maps
by Peter Schmidtke 03 Jul '19

03 Jul '19

2 3

pickling nan values
by Robert Oeffner 14 Jun '19

14 Jun '19

Hi all, It seems that a flex.double array containing "nan" values doesn't survive pickling as in: oeffner@grove:~$ cctbx.python Python 2.7.15 (default, Sep 11 2018, 23:48:45) [GCC 4.4.7 20120313 (Red Hat 4.4.7-18)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> >>> from cctbx.array_family import flex >>> import pickle >>> flex_nans = flex.double(3, float('nan')) >>> list(flex_nans) [nan, nan, nan] >>> list(pickle.loads( pickle.dumps(flex_nans))) [0.0, 0.0, 0.0] >>> >>> If the flex double is cast into a python list then it does survive pickling as in: >>> >>> list(pickle.loads( pickle.dumps(list( flex_nans )))) [nan, nan, nan] >>> I would like to use nan values to indicate missing or invalid data values. Would anyone have a suggestion how to go about this? Many thanks, Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus The Keith Peters Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234

2 1

Fwd: Conda/DIALS 1.14.5 source build with --use-conda does not work
by Nicholas Devenish 28 May '19

28 May '19

(Forwarding this manually as apparently my Diamond address isn't on cctbxbb) Hi All, On 24 May 2019, at 14:12, David Waterman - UKRI STFC < david.waterman(a)stfc.ac.uk<mailto:[email protected]>> wrote: > In the process, I discovered that the source builds we point to from > https://dials.github.io/installation.html don't work with --use-conda > because the paths to various packages are out of date. This seems to be to be a fundamental problem with the conda implementation? Let’s try to reproduce an old build from, say, 1st march: the point in cctbx_project at which dials 1.14 split off. This is https://github.com/cctbx/cctbx_project/commit/35652e06564da7ff989500a6d929b… . Try reproducing this point: git clone git(a)github.com<mailto:[email protected]>:cctbx/cctbx_project.git modules/cctbx_project ( cd modules/cctbx_project/ && git checkout 35652e06564da7ff989500a6d929bc595f69040f ) ln -s modules/cctbx_project/libtbx/auto_build/bootstrap.py . ./bootstrap.py base --use-conda —> many "An HTTP error occurred when trying to retrieve this URL.” This means that with the current conda implementation it’s impossible to guarantee reproducibility for any build beyond the past few days, without lots and lots of manual work reconstructing the versioning out of the file list dump. Because the version constraints aren’t kept with the repository, this means that a source distribution is not nearly enough to recreate any given release. This seems like a problem? Nick

3 2

Python3 / scons / non-conda builds
by Graeme.Winter＠Diamond.ac.uk 13 May '19

13 May '19

Dear All, Back in the mists of time, one could build cctbx modules without bootstrap or conda by having the right modules available yourself - a so-called baseless install which I think is still important in X-fel land? Anyhow, trying this with Python3 suggests that we have some special constraints which are perhaps too tight regarding module locations - for example scons - Grey-Area p3_compile :( $ pip3 install scons Collecting scons Downloading https://files.pythonhosted.org/packages/65/41/ac58638421414acc64e26fc120c81… (831kB) 100% |████████████████████████████████| 839kB 14.3MB/s Installing collected packages: scons Successfully installed scons-3.0.5 You are using pip version 19.0.1, however version 19.1.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command. Grey-Area p3_compile :) $ make ./bin/libtbx.scons -j "`./bin/libtbx.show_number_of_processors`" ModuleNotFoundError: No module named 'SCons' During handling of the above exception, another exception occurred: Sorry: SCons is not available. A possible solution is to unpack a SCons distribution in one of these directories: "/Users/graeme/svn/scr" "/Users/graeme/svn/scr/cctbx_project" SCons distributions are available at this location: http://www.scons.org/ It may be necessary to rename the unpacked distribution, e.g.: mv scons-0.96.1 scons make: *** [default] Error 1 Grey-Area p3_compile :( $ which scons /Library/Frameworks/Python.framework/Versions/3.7/bin/scons Should we require that scons is installed as part of cctbx when it is already installed on the system? If we *do* have to have this constraint, maybe that should be checked at configure time (where it complains about six and future) Thoughts? In other news - the syntax changing branch referenced in https://github.com/cctbx/cctbx_project/pull/325 now compiles correctly on Python3 (note: not run, just compile but this means that it’s purpose is done) for the modules which it intends to touch and has no conflicts right now… Best wishes Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 2

Overloading Python builtin names with local variables
by Graeme.Winter＠Diamond.ac.uk 03 May '19

03 May '19

Morning all, As a side-effect of looking at some of the Python3 refactoring I stumbled across a few places where e.g. range is used as a variable name. While this is fine and legal, when moving the code to Python3 xrange => range and from builtins import range is a back-port of Python3 range to Python2 which allows the existing behaviour of xrange generator to be maintained This is fine except for where range is a variable name and we then get a “variable referenced before assignment” type error As a general statement I would say overloading Python reserved names with local variables is untidy at best, and can easily cause confusion (as well as real problems as identified above) - I have therefore taken the liberty of adding a tool to libtbx which can be used to find such variables to give people a chance to avoid them: Grey-Area dxtbx :) [master] $ libtbx.find_reserved_names format/ Checking format/ format/FormatSMVRigakuSaturn.py:239 "format" format = self._scan_factory.format("SMV") format/FormatTIFFRayonixESRF.py:36 "format" format = {LITTLE_ENDIAN: "<", BIG_ENDIAN: ">"}[order] format/FormatTIFFRayonixESRF.py:29 "bytes" width, height, depth, order, bytes = FormatTIFFRayonix.get_tiff_header( format/FormatCBFMini.py:231 "format" format = self._scan_factory.format("CBF") format/FormatTIFFRayonix.py:176 "format" format = self._scan_factory.format("TIFF") format/FormatTIFFRayonix.py:31 "bytes" width, height, depth, order, bytes = FormatTIFF.get_tiff_header(image_file) format/FormatTIFFRayonix.py:74 "bytes" width, height, depth, order, bytes = FormatTIFF.get_tiff_header(image_file) format/FormatSMVRigakuEiger.py:243 "format" format = self._scan_factory.format("SMV") format/FormatSMVADSC.py:212 "format" format = self._scan_factory.format("SMV") format/Registry.py:88 "format" for format in sorted(self._formats, key=lambda x: x.__name__): format/FormatSMVJHSim.py:141 "format" format = self._scan_factory.format("SMV") format/FormatSMVRigakuPilatus.py:188 "format" format = self._scan_factory.format("SMV") format/FormatSMVADSCSN928.py:47 "format" format = self._scan_factory.format("SMV") format/FormatRAXISIVSpring8.py:134 "format" format = self._scan_factory.format("RAXIS") format/FormatTIFFRayonixSPring8.py:31 "bytes" width, height, depth, order, bytes = FormatTIFFRayonix.get_tiff_header( format/FormatSMVADSCmlfsom.py:37 "format" format = self._scan_factory.format("SMV") format/FormatTIFFBruker.py:174 "format" format = self._scan_factory.format("TIFF") format/FormatTIFFBruker.py:31 "bytes" width, height, depth, order, bytes = FormatTIFF.get_tiff_header(image_file) format/FormatTIFFBruker.py:71 "bytes" width, height, depth, order, bytes = FormatTIFF.get_tiff_header(image_file) format/FormatCBFMiniADSCHF4M.py:26 "format" for format in ["%a_%b_%d_%H:%M:%S_%Y"]: format/FormatCBFMiniADSCHF4M.py:169 "format" format = self._scan_factory.format("CBF") format/FormatSMVADSCNoDateStamp.py:54 "format" format = self._scan_factory.format("SMV") format/FormatSMVRigakuSaturnNoTS.py:228 "format" format = self._scan_factory.format("SMV") format/FormatPYunspecified.py:152 "format" format = "" format/FormatSMVRigakuA200.py:234 "format" format = self._scan_factory.format("SMV") format/FormatCBFMiniEigerPhotonFactory.py:131 "format" format = self._scan_factory.format("CBF") format/FormatSMVTimePix_SU.py:97 "format" format = self._scan_factory.format("SMV") format/FormatCBFMiniPilatus.py:83 "format" format = self._scan_factory.format("CBF") format/FormatCBFMiniPilatusHelpers.py:22 "format" for format in ["%Y-%b-%dT%H:%M:%S", "%Y-%m-%dT%H:%M:%S", "%Y/%b/%d %H:%M:%S"]: format/FormatSMVADSCSNAPSID19.py:91 "format" format = self._scan_factory.format("SMV") format/FormatCBFMiniEiger.py:216 "format" format = self._scan_factory.format("CBF") format/FormatSMVNOIR.py:218 "format" format = self._scan_factory.format("SMV") format/FormatSMVCMOS1.py:197 "format" format = self._scan_factory.format("SMV") format/FormatRAXIS.py:238 "format" format = self._scan_factory.format("RAXIS") is an example where “format” is scattered all over the place but is also a useful function - replacing with fmt will mean the same and avoid this overloading, as an example. This uses AST parsing to work through the code and find variable assignments - doing this for dials found lots of examples of “file” and “type” being used. How much people care about this is a local issue, but I thought the tool would be useful All the best Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

modulo operation on flex.double
by Robert Oeffner 12 Apr '19

12 Apr '19

Hi, I have a flex.double array of phase values between -360 and 360 degrees I would like to cast them all in one go to lie within 0 to 360. Does anybody know if there is a nice way of doing this with flex arrays? The modulo operator doesn't seem to be supported for flex arrays unlike the +, -, * and the / operators. This would otherwise be handy as I could then convert the array in one line of code like phases = phases % 360.0 If not I guess I'll have to numpy the problem :-) Many thanks, Rob

5 5

Re: [cctbxbb] cannot compile Phenix
by R. D. Oeffner 07 Apr '19

07 Apr '19

>From the messages on the cctbx github mailing list it seems that Dials people have been extricating dxtbx from cctbx_project. Rob -----Original Message----- From: Pavel Afonine Sent: Saturday, April 6, 2019 7:07 PM To: phenixdev Subject: cannot compile Phenix Hi, anyone knows why current Phenix code is broken: python bootstrap.py --builder=phenix --sfuser=afonine --cciuser=afonine update python bootstrap.py --builder=phenix --sfuser=afonine --cciuser=afonine build --nproc=12 leads to (...) AttributeError: 'empty' object has no attribute 'dxtbx_include': File "/Users/pafonine/phenix/build/SConstruct", line 31: SConscript("spotfinder/SConscript") File "/Users/pafonine/phenix/modules/scons/engine/SCons/Script/SConscript.py", line 607: return method(*args, **kw) File "/Users/pafonine/phenix/modules/scons/engine/SCons/Script/SConscript.py", line 544: return _SConscript(self.fs, *files, **subst_kw) File "/Users/pafonine/phenix/modules/scons/engine/SCons/Script/SConscript.py", line 253: call_stack[-1].globals) File "/Users/pafonine/phenix/modules/cctbx_project/spotfinder/SConscript", line 22: env_etc.dxtbx_include, usr+sys time: 1.94 seconds wall clock time: 4.51 seconds Process failed with return code 2 Any idea for a workaround? Pavel --- This email has been checked for viruses by AVG. https://www.avg.com

2 3

jupyter notebook
by wtempel 03 Apr '19

03 Apr '19

Hi all, how does one configure, and run inside a jupyter notebook, a cctbx-enabled kernel, beginning with a downloaded [cctbx build]( http://cci.lbl.gov/cctbx_build/)? [This page]( https://medium.com/@sljack1992/making-a-custom-ipython-notebook-kernel-c59e…) seems to describe the first part, but may no longer apply to more recent cctbx builds. Thanks. Wolfram Tempel

2 1