- cctbxbb - phenix-online.org

Monthly "how far are we from moving to git" message
by Graeme.Winter＠diamond.ac.uk 21 Oct '16

21 Oct '16

Good afternoon / morning / evening (depending on your locale) I'm just running a bootstrap build and having to check out the whole of cctbx one ... file ... at ... a ... time from svn & it reminded me we were going to move over to git at some point - ISTR was determined by the PHENIX release. Just wondering how this is going & how long it is likely to be until we can get this ball rolling :) Thanks & best wishes Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 6

sginfo and sgtbx
by Pascal 21 Oct '16

21 Oct '16

Hi, Just out of curiosity, sginfo website mentions that it has been superseeded by sgtbx. Is that a brand new implementation? I don't recognize sginfo code at all. Also, anyone can confirm that sgtbx is tightly embedded is cctbx and cannot be used standalone? Regards, Pascal

4 6

flex.int.as_double() performance
by richard.gildea＠diamond.ac.uk 11 Oct '16

11 Oct '16

Hi, Whilst looking into performance of various DIALS routines recently, I noticed that the flex.int.as_double() conversion takes roughly 3x as long as the equivalent functionality in numpy: (the choice of a 2527x2463 array is simply because this is the size of the raw data array for a Pilatus 6M detector) import time from dials.array_family import flex n = 100 a = flex.int(flex.grid(2527, 2463)) t0 = time.time() for i in range(n): b = a.as_double() t1 = time.time() t = t1-t0 print "flex.int.as_double(): %.2fs (%.3fs/call)" %(t, t/n) import numpy as np a = np.random.randint(100, size=a.all()) t0 = time.time() for i in range(n): b = a.astype(np.float64) t1 = time.time() t = t1-t0 print "numpy.astype(np.float64): %.2fs (%.3fs/call)" %(t, t/n) $ libtbx.python time_as_double.py flex.int.as_double(): 5.76s (0.058s/call) numpy.astype(np.float64): 1.86s (0.019s/call) This is the current version of the as_double function: static flex_double as_double(f_t const& a) { shared_plain<double> result(a.size(), init_functor_null<double>()); for(std::size_t i=0;i<a.size();i++) result[i] = a[i]; return flex_double(result, a.accessor()); } Replacing the above code with the following, I see a significant runtime improvement, now flex.int.as_double() is marginally quicker than the equivalent numpy routine: static flex_double as_double(f_t const& a) { shared_plain<double> result(a.begin(), a.end()); return flex_double(result, a.accessor()); } $ libtbx.python time_as_double.py flex.int.as_double(): 1.64s (0.016s/call) numpy.astype(np.float64): 1.82s (0.018s/call) Is there any obvious reason why making this change would be A Bad Thing? In the context of dials spot-finding, this change results in a 22s saving (out of a total 172s with the original code) when running dials.find_spots on a 540 image Pilatus 6M dataset using a single processor, so it is definitely a noticeable speed-up. All timings done on a macbook pro. Cheers, Richard Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 3

"import pickle"
by markus.gerstel＠diamond.ac.uk 23 Sep '16

23 Sep '16

Hi everyone, I note that we are using "import pickle" on some 57 occasions. Is there a particular reason we don't import cPickle? For example: --- libtbx/env_config.py (revision 25451) +++ libtbx/env_config.py (working copy) @@ -7,7 +7,10 @@ from libtbx import adopt_init_args import platform import shutil -import pickle +try: + import cPickle as pickle +except: + import pickle from cStringIO import StringIO import re import sys, os This should still work on all systems without issue. According to python docs the only difference is that cPickle can't be subclassed. If it's alright with you I would go through those imports and update them to use cPickle where possible. -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 2

bootstrap / svn method / say reason not anonymous && http
by Graeme.Winter＠diamond.ac.uk 21 Sep '16

21 Sep '16

Hello folks, In bootstrap.py is there any logical reason why we can’t use %sfmethod for anonymous as well as authenticated access to the cctbx_project sources i.e. in # Core CCTBX repositories # These must all provide anonymous access. class cctbx_module(SourceModule): module = 'cctbx_project' anonymous = ['svn','svn://svn.code.sf.net/p/cctbx/code/trunk'] authenticated = ['svn', '%(sfmethod)s://%(sfuser)[email protected]/p/cctbx/code/trunk'] also respect %(sfmethod) in place of svn as transport layer. Will help people behind firewalls. Appreciate that this will also need changing when we move to git anyhow… but meantimes there are people out there this would help. Not keen to make this change myself without checking first…! Thanks & best wishes Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

rpath problem on Mac system
by Haiguang Liu 21 Sep '16

21 Sep '16

Dear CCTBX developing team, Is there already a solution for the run path problem in the new Mac OS? The System Integrity Protection <https://support.apple.com/en-us/HT204899> does not allow relative run path. I temporarily disabled this feature to run the programs that requires boost (for example). If there is already a solution, please let me know. Thanks, Haiguang

3 4

Python 2.4 compatibility
by richard.gildea＠diamond.ac.uk 16 Sep '16

16 Sep '16

This commit uses the "with" statement, which was introduced in Python 2.5, to libtbx/__init__.py, preventing installation of DIALS from sources on systems with Python 2.4: https://sourceforge.net/p/cctbx/code/25101/ Could this code be changed to remove the use of the "with" statement? Cheers, Richard Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 5

failing test with CCP4 7.0.18
by markus.gerstel＠diamond.ac.uk 16 Sep '16

16 Sep '16

Hi, The test libtbx.python $(libtbx.find_in_repositories cctbx/development/run_shelx.py) P31 works with CCP4 7.0.17, but fails when run with SHELXL from CCP4 7.0.18. The test apparently involves creating a small random crystal structure, calculating the structure factors and solving the structure using SHELXL. The updated SHELXL version creates deterministic, but slightly different anisotropic temperature factors. I am not sure if this is a problem with the test or with the SHELXL update. Could anybody have a look into that? -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

4 4

Re: [cctbxbb] [Cctbx-cvs] SF.net SVN: cctbx:[25333] trunk/libtbx/env_config.py
by markus.gerstel＠diamond.ac.uk 09 Sep '16

09 Sep '16

Hi, I just spent some time tracking software crashes to this change. Is setting the default to en_US really appropriate and what we want? In particular it affects the output of downstream, external software we run from within python. What is the unicode issue you hint at in the commit message? -Markus Dr Markus Gerstel MBCS Postdoctoral Research Associate Tel: +44 1235 778698 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE -----Original Message----- From: bkpoon(a)users.sourceforge.net [mailto:[email protected]] Sent: 07 September 2016 00:54 To: cctbx-cvs(a)lists.sourceforge.net Subject: [Cctbx-cvs] SF.net SVN: cctbx:[25333] trunk/libtbx/env_config.py Revision: 25333 http://sourceforge.net/p/cctbx/code/25333 Author: bkpoon Date: 2016-09-06 23:54:29 +0000 (Tue, 06 Sep 2016) Log Message: ----------- Unicode support: set LC_ALL in dispatchers to the one in the user's environment (if available, and supports UTF-8), otherwise use the default setting of en_US.UTF-8; fixes unicode issue with python in Linux (e.g. os.path functions do not work correctly with unicode if LC_ALL=C Modified Paths: -------------- trunk/libtbx/env_config.py Modified: trunk/libtbx/env_config.py =================================================================== --- trunk/libtbx/env_config.py 2016-09-06 21:15:34 UTC (rev 25332) +++ trunk/libtbx/env_config.py 2016-09-06 23:54:29 UTC (rev 25333) @@ -945,6 +945,15 @@ def write_bin_sh_dispatcher(self, source_file, target_file, source_is_python_exe=False): + + # determine LC_ALL from environment (Python UTF-8 compatibility in Linux) + LC_ALL = os.environ.get('LC_ALL') # user setting + if (LC_ALL is not None): + if ( ('UTF-8' not in LC_ALL) and ('utf8' not in LC_ALL) ): + LC_ALL = None + if (LC_ALL is None): + LC_ALL = 'en_US.UTF-8' # default + f = target_file.open("w") if (source_file is not None): print >> f, '#! /bin/sh' @@ -975,7 +984,7 @@ print >> f, '#' print >> f, _SHELLREALPATH_CODE print >> f, 'unset PYTHONHOME' - print >> f, 'LC_ALL=C' + print >> f, 'LC_ALL=' + LC_ALL print >> f, 'export LC_ALL' print >> f, 'LIBTBX_BUILD="$(shellrealpath "$0" && cd "$(dirname "$RESULT")/.." && pwd)"' print >> f, 'export LIBTBX_BUILD' This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. ------------------------------------------------------------------------------ _______________________________________________ Cctbx-cvs mailing list Cctbx-cvs(a)lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/cctbx-cvs -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 3

Inserting F000 into reflection (mtg) file
by Nicholas Pearce 22 Aug '16

22 Aug '16

Hi Everyone, I’ve been looking through the cctbx/phenix scripts and although there’s a script for estimating F000 (phenix.f000), I can’t find any way of inserting it into an mtz file to make it accessible for easy map generation. Is this implemented anywhere/what do people use as the best way of doing this? If the insertion of F000 into an mtz file has been purposefully avoided, I presume it's done because we don't want the possibility of someone refining a model where F000 is used in refinement. Unless there’s another reason? Thanks, Nick

2 1

Git
by Graeme.Winter＠diamond.ac.uk 01 Aug '16

01 Aug '16

Folks, Can we revisit the idea of moving to git for cctbx? Thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

7 18

Structure Factor for 000
by Gareth.Nisbet＠diamond.ac.uk 28 Jul '16

28 Jul '16

Is it possible to get the structure_factors function to give the structure factor for the 000 reflection? I appreciate this might be a slightly odd question. Gareth -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 3

Berkeley Server Downtime (24 hours, 7/22 - 7/23)
by Billy Poon 20 Jul '16

20 Jul '16

Dear all, On Friday, July 22, our servers will be shut down starting at 5:00 pm Pacific for scheduled maintenance in our building. We expect that the servers will be available the next day (7/23) by 5:00 pm Pacific at the latest. This will affect, - The Phenix web site (http://phenix-online.org/) - The DIALS West web site (http://dials.lbl.gov/) - The CCI web site (http://cci.lbl.gov/) - The Structural Biology Technology Portal (http://technology.sbkb.org/) Bug reports and emails requesting help should be queued and we will receive them once the servers are back online. We apologize for any inconvenience. -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

1 0

Re: [cctbxbb] Git
by R. D. Oeffner 02 Jul '16

02 Jul '16

How often is this git repo of cctbx copied from the svn repo of cctbx? Rob -----Original Message----- From: cctbxbb-request(a)phenix-online.org Sent: Friday, July 1, 2016 8:00 PM To: cctbxbb(a)phenix-online.org Subject: cctbxbb Digest, Vol 88, Issue 1 Send cctbxbb mailing list submissions to cctbxbb(a)phenix-online.org To subscribe or unsubscribe via the World Wide Web, visit http://phenix-online.org/mailman/listinfo/cctbxbb or, via email, send a message with subject or body 'help' to cctbxbb-request(a)phenix-online.org You can reach the person managing the list at cctbxbb-owner(a)phenix-online.org When replying, please edit your Subject line so it is more specific than "Re: Contents of cctbxbb digest..." Today's Topics: 1. Re: Git (markus.gerstel(a)diamond.ac.uk) ---------------------------------------------------------------------- Message: 1 Date: Fri, 1 Jul 2016 12:34:56 +0000 From: <markus.gerstel(a)diamond.ac.uk> To: <cctbxbb(a)phenix-online.org> Subject: Re: [cctbxbb] Git Message-ID: <415F7F573406BB498256C68158D45674666211BB@exchmbx01> Content-Type: text/plain; charset="utf-8" Hello everyone, Following up the earlier discussion, do have we reached a consensus? Also, I think timeframe has not been discussed, but I assume this will boil down to ?after phenix release? anyway. In the meantime I have contacted (nearly) all cctbx developers and obtained their github user-IDs - alongside some enthusiastic* feedback. I will mass-invite you all soonishly to a cctbx organisation with a preliminary git copy of the svn history to play with. Best wishes -Markus Markus Gerstel MBCS Postdoctoral Research Associate Tel: +44 1235 778698 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE * The best comment I received was ?What?s CCTBX?? ? and that was talking to the correct person :) (name withheld) -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

Installation failure after package updates
by markus.gerstel＠diamond.ac.uk 27 Jun '16

27 Jun '16

Following the latest package updates we are now observing installer failures. In particular the configure step appears to fail with a missing libffi.so.6 Any ideas how to rectify this, or if this error can be ignored safely? -Markus Running with --verbose: ========================================================================== DIALS Installation version: dev20160610 machine type: intel-linux-2.6-x86_64 OS version: 2.6.32-573.18.1.el6.x86_64 destination: /dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610 # of processors: 24 ========================================================================= Configuring DIALS components... /dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/build ['/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/base/bin/python', '/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/libtbx/configure.py', '--current_working_directory', '/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/build', 'dials', 'xia2', 'iota', 'prime'] : Python: 2.7.8 "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/base/bin/python" : Compiler: default : Build mode: release : Warning level: 0 : Precompiled Headers: False : Static libraries: False : Static exe: False : Scan Boost headers: False : Write full flex_fwd.h files: False : Build Boost.Python extensions: True : Define BOOST_PYTHON_NO_PY_SIGNATURES: False : Define BOOST_PYTHON_BOOL_INT_STRICT: True : Enable OpenMP if possible: False : Boost threads enabled: False : Enable CUDA: False : Use opt_resources if available: False : Use environment flags: False : Enable C++11: False : Force true division: True : command_version_suffix: None : Relocatable paths anchored at: /dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/build : Top-down list of all modules involved: : prime "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/prime" : iota "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/iota" : xia2 "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/xia2" : mmtbx "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/mmtbx" : dials "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/dials" : xfel "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/xfel" : cma_es "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/cma_es" : gltbx "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/gltbx" : crys3d "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/crys3d" : rstbx "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/rstbx" : spotfinder "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/spotfinder" : annlib+adaptbx "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/annlib" : "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/annlib_adaptbx" : dxtbx "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/dxtbx" : iotbx "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/iotbx" : cbflib+adaptbx "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cbflib" : "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/cbflib_adaptbx" : ccp4io+adaptbx "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/ccp4io" : "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/ccp4io_adaptbx" : smtbx "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/smtbx" : ucif "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/ucif" : cctbx "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/cctbx" : chiltbx "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/chiltbx" : scitbx "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/scitbx" : fable "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/fable" : omptbx "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/omptbx" : boost+adaptbx "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/boost" : "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/boost_adaptbx" : tbxx "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/tbxx" : libtbx "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/libtbx" : Creating files in build directory: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/build" : dispatcher_include_template.sh : dispatcher_head.sh : setpaths.sh : setpaths.csh : setpaths_all.sh : setpaths_all.csh : setpaths_debug.sh : unsetpaths.sh : setpaths_debug.csh : unsetpaths.csh : SConstruct : Makefile : run_tests.csh : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/libtbx/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/boost_adaptbx/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/fable/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/scitbx/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/cctbx/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/smtbx/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/cbflib_adaptbx/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/iotbx/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/dxtbx/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/spotfinder/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/rstbx/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/crys3d/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/gltbx/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/xfel/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/xfel/ui/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/dials/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/mmtbx/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/xia2/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/iota/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/prime/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/boost_adaptbx/libtbx_refresh.py" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/scitbx/libtbx_refresh.py" : Generating C++ header files in: : "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/build/include/scitbx/array_family" : Generating C++ files in: : "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/build/include/scitbx/array_family/boost_python" : Using fable to convert scitbx/lbfgs.f : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/cctbx/libtbx_refresh.py" : Generating C++ files in: : "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/build/cctbx/eltbx" : Generating C++ files in: : "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/build/include/cctbx/boost_python" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/smtbx/libtbx_refresh.py" : Generating C++ files in: : "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/build/include/smtbx/boost_python" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/cbflib_adaptbx/libtbx_refresh.py" : Copying to lib: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cbflib/pycbf/pycbf.py" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/iotbx/libtbx_refresh.py" : Using fable to convert iotbx/pdb/hybrid_36_f.f : Writing: iotbx/pdb/hybrid_36_fem.cpp : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/annlib_adaptbx/libtbx_refresh.py" : Generating C++ files for self-inclusive approximate nearest neighbor : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/crys3d/libtbx_refresh.py" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/gltbx/libtbx_refresh.py" : Generating C++ files in: : "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/build/gltbx" : Info: Re-using existing font cpp files. : Hint: Remove gltbx/FONTS_UCS_DONE_FLAG_FILE : to force generation of new font files. : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/xfel/libtbx_refresh.py" : Creating cxi_user directory : Creating cxi_user/__init__.py : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/dials/libtbx_refresh.py" : DIALS 1.dev.360-ge6510c8 : Identifying autocompletable commands: dials.analyse_background dials.combine_found_spots dials.find_spots dials.index dials.integrate dials.reciprocal_lattice_viewer dials.refine dials.reindex dials.rl_png : Installing autocompletion script into: setpaths_all.sh setpaths_debug.sh setpaths.sh : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/mmtbx/libtbx_refresh.py" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/xia2/libtbx_refresh.py" : XIA2 0.4.0.381-g9b3a992 : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/build/exe_dev" : Regenerating module files in /dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/base : /dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/base/etc/pango not present, could not regenerate pango files : generating gtk.immodules file : /dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/base/bin/gtk-query-immodules-2.0: error while loading shared libraries: libffi.so.6: cannot open shared object file: No such file or directory : Traceback (most recent call last): : File "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/libtbx/configure.py", line 22, in <module> : if not run(): : File "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/libtbx/configure.py", line 17, in run : libtbx.env_config.cold_start(sys.argv) : File "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/libtbx/env_config.py", line 2264, in cold_start : env.refresh() : File "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/libtbx/env_config.py", line 1537, in refresh : regenerate_module_files.run(libtbx.env.under_base('.'), only_if_needed=True) : File "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/libtbx/auto_build/regenerate_module_files.py", line 154, in run : fix_gtk(base_dir, out) : File "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/libtbx/auto_build/regenerate_module_files.py", line 72, in fix_gtk : log=out) : File "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/libtbx/auto_build/installer_utils.py", line 81, in call : raise RuntimeError("Call to '%s' failed with exit code %d" % (args, rc)) : RuntimeError: Call to '/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/base/bin/gtk-query-immodules-2.0 /dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/base/lib/gtk-2.0/2.10.0/immodules/*.so> /dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/base/etc/gtk-2.0/gtk.immodules' failed with exit code 127 Traceback (most recent call last): File "./bin/install.py", line 97, in <module> installer(sys.argv[1:]).install() File "/scratch/jenkins_slave/workspace/dials_install_dls_module/dials-installer-dev/lib/libtbx/auto_build/install_distribution.py", line 197, in install self.install_from_binary() File "/scratch/jenkins_slave/workspace/dials_install_dls_module/dials-installer-dev/lib/libtbx/auto_build/install_distribution.py", line 318, in install_from_binary self.reconfigure(log=log) File "/scratch/jenkins_slave/workspace/dials_install_dls_module/dials-installer-dev/lib/libtbx/auto_build/install_distribution.py", line 484, in reconfigure call(args=args, log=log, verbose=self.options.verbose) File "/scratch/jenkins_slave/workspace/dials_install_dls_module/dials-installer-dev/lib/libtbx/auto_build/installer_utils.py", line 81, in call raise RuntimeError("Call to '%s' failed with exit code %d" % (args, rc)) RuntimeError: Call to '/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/base/bin/python /dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/libtbx/configure.py --current_working_directory /dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/build dials xia2 iota prime' failed with exit code 1 Markus Gerstel MBCS Postdoctoral Research Associate Tel: +44 1235 778698 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 6

Re: [cctbxbb] error
by markus.gerstel＠diamond.ac.uk 23 Jun '16

23 Jun '16

Hi James, CCing cctbxbb for further support I can reproduce at least similar errors with both DIALS 1.2.5 and current nightly builds. I think this may be a matplotlib 1.3.1 problem. A minimal reproducer is: $ dials.python > import matplotlib.pyplot as plt > plt.imshow([[1]]) Gtk-Message... > (press ctrl+D) Segmentation fault When compiled on the same machine I do not get the Gtk-Messages, but I do get the segmentation fault. Markus Gerstel MBCS Postdoctoral Research Associate Tel: +44 1235 778698 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE From: Parkhurst, James (DLSLtd,RAL,LSCI) Sent: 21 June 2016 16:58 To: Gerstel, Markus (DLSLtd,RAL,LSCI) Subject: FW: error Hi Markus, I've been getting my summer student to do the cctbx_introduction tutorials in dials_scratch. He's using a version of dials downloaded from the website and when he runs the introduction script he gets errors about gdk-pixbuf giving paths to /scratch/jenkins_slave/workspace/. This seems like there is a jenkins path being set in the distributed installers. Would you have any idea how I can fix this? I'm downloading the linux installer of DIALS now so I can test this myself. Thanks, James ________________________________ [sse87232@ws188 cctbx_introduction_1]$ dials.python cctbx_introduction_1_test.py ximg2701_00001.cbf (cctbx_introduction_1_test.py:6243): GdkPixbuf-WARNING **: Cannot open pixbuf loader module file '/scratch/jenkins_slave/workspace/dials_release_build/compilationtarget/native/label/dials-ws133/build_dials/base/etc/gtk-2.0/gdk-pixbuf.loaders': No such file or directory (cctbx_introduction_1_test.py:6243): GdkPixbuf-WARNING **: Cannot open pixbuf loader module file '/scratch/jenkins_slave/workspace/dials_release_build/compilationtarget/native/label/dials-ws133/build_dials/base/etc/gtk-2.0/gdk-pixbuf.loaders': No such file or directory (cctbx_introduction_1_test.py:6243): GdkPixbuf-WARNING **: Cannot open pixbuf loader module file '/scratch/jenkins_slave/workspace/dials_release_build/compilationtarget/native/label/dials-ws133/build_dials/base/etc/gtk-2.0/gdk-pixbuf.loaders': No such file or directory (cctbx_introduction_1_test.py:6243): GdkPixbuf-WARNING **: Cannot open pixbuf loader module file '/scratch/jenkins_slave/workspace/dials_release_build/compilationtarget/native/label/dials-ws133/build_dials/base/etc/gtk-2.0/gdk-pixbuf.loaders': No such file or directory (cctbx_introduction_1_test.py:6243): GdkPixbuf-WARNING **: Cannot open pixbuf loader module file '/scratch/jenkins_slave/workspace/dials_release_build/compilationtarget/native/label/dials-ws133/build_dials/base/etc/gtk-2.0/gdk-pixbuf.loaders': No such file or directory (cctbx_introduction_1_test.py:6243): GdkPixbuf-WARNING **: Cannot open pixbuf loader module file '/scratch/jenkins_slave/workspace/dials_release_build/compilationtarget/native/label/dials-ws133/build_dials/base/etc/gtk-2.0/gdk-pixbuf.loaders': No such file or directory (cctbx_introduction_1_test.py:6243): GdkPixbuf-WARNING **: Cannot open pixbuf loader module file '/scratch/jenkins_slave/workspace/dials_release_build/compilationtarget/native/label/dials-ws133/build_dials/base/etc/gtk-2.0/gdk-pixbuf.loaders': No such file or directory Segmentation fault (core dumped) [sse87232@ws188 cctbx_introduction_1]$ which dials.python /dls/science/users/sse87232/dials/dials-v1-2-4/build/bin/dials.python -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

4 6

past contributors
by markus.gerstel＠diamond.ac.uk 16 Jun '16

16 Jun '16

I have now contacted everyone who ever contributed to cctbx in the past to get name/email details in preparation for an svn->git conversion. Everyone except these three: Erik McKee (emckee) Vishal Verma (vermav) ydr (y0udar) Anyone know how to get into contact with them? -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

Not wishing to raise this again but...
by Graeme.Winter＠diamond.ac.uk 16 Jun '16

16 Jun '16

... is there any enthusiasm out there for moving cctbx to git? The more I work with git the more I wish this existed properly for cctbx. I've raised this a bunch of times and always it is "not right now" however "a good idea" - this'll be the last time I ask... Could we move to git, please? I note the bootstrap script for Windows now has a git dependency even! Am happy to help if I can with anything though I think it should be pretty simple... was for DIALS & xia2 :) Thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

8 23

Not wishing to raise this again but...
by R.D. Oeffner 15 Jun '16

15 Jun '16

Personally I think moving the cctbx to git is a great idea as it would make it easier and safer when developing and testing new features. Given so many teams are using the cctbx this seems like a compelling reason for it to be moved. We had a chat a while ago with Berkeley about this but I've forgotten what the consensus was. Rob > > Message: 1 > Date: Tue, 14 Jun 2016 12:10:03 +0000 > From: <Graeme.Winter(a)diamond.ac.uk> > To: <cctbxbb(a)phenix-online.org> > Subject: Re: [cctbxbb] Not wishing to raise this again but... > Message-ID: <13999B6FC038EB44B067771E3CFBD091BC327D8E@exchmbx03> > Content-Type: text/plain; charset="us-ascii" > > Hi All, > > This sounds like it is heading in a pragmatic direction - I would find > it very helpful to gauge feeling for or against this. To make this > simple I have attempted (starting from a completely ignorant position) > to put together one of those online survey things: > > https://www.surveymonkey.co.uk/r/GXPY5JW > > If you could pick an option that would be great, explaining why would > be even better, or hitting reply-all to this thread would also work > (as has previously done so) I'll collate answers over the next few > days & feed back when we have (say) 10-20 or a few days have passed > >> From experience with moving xia2 & DIALS it was surprisingly >> straightforward (an opinion) > > Thanks & cheerio Graeme > > -----Original Message----- > From: cctbxbb-bounces(a)phenix-online.org > [mailto:[email protected]] On Behalf Of > markus.gerstel(a)diamond.ac.uk > Sent: 14 June 2016 12:18 > To: cctbxbb(a)phenix-online.org > Subject: Re: [cctbxbb] Not wishing to raise this again but... > > Hi Pavel, > > I volunteered to do this some time ago, just like for DIALS, and would > still be happy to do this. > We can do the move slowly, so allow users (developers) with a git > repository while development still happens in SVN for say a couple of > weeks, and I would be happy to run the git support for that time and > for the aftermath of the move. > > Best wishes > -Markus > > -----Original Message----- > From: cctbxbb-bounces(a)phenix-online.org > [mailto:[email protected]] On Behalf Of Pavel Afonine > Sent: 14 June 2016 12:11 > To: cctbx mailing list > Subject: Re: [cctbxbb] Not wishing to raise this again but... > > Hey there! > > Well I guess I say something even though I don't have much to say > really. I've been contributing to CCTBX for the past 13 years, so I > feel like this conversation is not too irrelevant to me to say the > least. > > I don't have strong opinion about staying with svn vs moving to git. I > routinely contribute to CCTBX (which is SVN-based) and I routinely > contribute to my other activity - Q|R (Quantum Refinement, which is > hosted by GitHub). I'm comfortable with both environments as I'm using > only the minimal functionality of both: check in, check out, diff.. > period. I'm not interested in branching/etc sophistication; this is > just because the way I structure my workflow.. > > My current understanding is that moving CCTBX to use git boils down to: > - actually someone doing it; AND > - taking good proper responsible care of the aftermath (if any). > > So far no-one volunteered to embrace this challenge, as far as I know. > > In summary I think we should move to git, at least to follow the > vector of progress. > > All the best, > Pavel >

3 2

bootstrap.py build on Ubuntu
by David Waterman 14 Jun '16

14 Jun '16

Hi folks I recently tried building cctbx+dials on Ubuntu 15.04 following the instructions here: http://dials.github.io/documentation/installation_developer.html This failed during installation of pango-1.16.1. Looking at pango_install_log, I see the command that failed was as follows: gcc -DHAVE_CONFIG_H -I. -I. -I../.. -DSYSCONFDIR=\"/home/fcx32934/sw/dials_bootstrap_test/base/etc\" -DLIBDIR=\"/home/fcx32934/sw/dials_bootstrap_test/base/lib\" -DG_DISABLE_CAST_CHECKS -I../.. -DG_DISABLE_DEPRECATED -I/home/fcx32934/sw/dials_bootstrap_test/base/include -I/home/fcx32934/sw/dials_bootstrap_test/base/include/freetype2 -g -O2 -Wall -MT fribidi.lo -MD -MP -MF .deps/fribidi.Tpo -c fribidi.c -fPIC -DPIC -o .libs/fribidi.o In file included from fribidi.h:31:0, from fribidi.c:28: fribidi_config.h:1:18: fatal error: glib.h: No such file or directory The file glib.h appears to be in base/include/glib-2.0/, however this directory was not explicitly included in the command above, only its parent. This suggests a configuration failure in pango to me. Taking a look at base_tmp/pango-1.16.1/config.log, I see what look like the relevant lines: configure:22227: checking for GLIB configure:22235: $PKG_CONFIG --exists --print-errors "$GLIB_MODULES" configure:22238: $? = 0 configure:22253: $PKG_CONFIG --exists --print-errors "$GLIB_MODULES" configure:22256: $? = 0 configure:22304: result: yes but this doesn't tell me very much. Does anyone have any suggestions as to how I might proceed? Many thanks -- David

3 10

Nightly test server down?
by Luc Bourhis 14 Jun '16

14 Jun '16

Hi, http://cci-vm-6.lbl.gov:8010/waterfall does not answer anymore from where I stand (Paris). Best wishes, Luc

3 2

flex.histogram: add weights?
by Graeme.Winter＠diamond.ac.uk 14 Jun '16

14 Jun '16

Hi Folks I would like to be able to compute a weighted flex histogram i.e. instead of adding 1 for every element in array add weight[i] for array[i] instead (ie. generalized) Has anyone done this already? Don't want to break anything, will require the histogram itself being non-integral... Thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 3

Project Scientist positions for computational methods
by Nicholas Sauter 13 Jun '16

13 Jun '16

Two Project Scientist positions are available in my group in the *Molecular Biophysics and Integrated Bioimaging* <http://biosciences.lbl.gov/divisions/mbib/> (MBIB) Division at Lawrence Berkeley National Lab. *Bioimaging Project Scientist* Position Summary: We have an exciting opportunity for a Bioimaging Project Scientist who will apply neuromorphic computing techniques to structural biology problems in X-ray diffraction, CryoEM, and CryoET. This will be a cross-disciplinary effort between MBIB and the Computational Research (CRD) Division at Lawrence Berkeley National Laboratory. The research group will include investigators Nick Sauter (X-ray Crystallography), Karen Davies (CryoEM/ET), and Chao Yang (Scalable Solvers). Specific Responsibilities: - Investigate how to apply deep learning algorithms to specific data processing and data interpretation problems in structural biology. - Implement convolutional neural network (CNN) code on neuromorphic computing hardware such as the IBM TrueNorth chip. Example problems include the identification of positive diffraction events in X-ray free-electron laser diffraction, conformational classification in CryoEM single particle reconstruction, and the identification of 3D sections for CryoET subtomogram averaging. - Use tools such as MatConvNet, CAFFE, THEANO or TensorFlow to construct and train CNNs that can be used to optimally classify experimental data. Required Qualifications: - Research experience in Structural Biology, with expertise in allied fields such as Bioinformatics, Mathematics, Computer Science, or Engineering (Ph.D. + 3 years or equivalent) - Strong background in image analysis and machine learning, including neural networks. - Working knowledge of image classification and/or other image processing techniques. - Proficient programming skills in MATLAB and/or Python. - Working knowledge of performance optimization for scientific codes. *Algorithm Developer Project Scientist* Position Summary: We are seeking an algorithm developer for the increasingly complex analysis of large diffraction datasets in structural biology. Current projects utilize XFEL crystallography and spectroscopy to investigate the photosynthetic mechanism of water splitting and to probe other metalloenzyme reactions. We also wish to test whether diffuse scattering can reveal correlated atomic motions in crystals. Many problems remain to be solved, including the details of how to optimally merge datasets from thousands of crystals. Our software development projects (including packages such as DIALS <http://dials.lbl.gov/> and cctbx.xfel <http://cci.lbl.gov/xfel>) have been highlighted in several high-impact publications listed HERE <http://biosciences.lbl.gov/profiles/nicholas-sauter>. Specific Responsibilities: - Continue work on the DIALS data integration suite. - Test new algorithms for optimally characterizing and merging data from serial crystallography experiments. - Analyze anomalous scattering from metalloenzymes using XFEL diffraction. - Investigate whether diffuse scattering can reveal correlated atomic motions in crystals. Required Qualifications: - Expertise in one or more computational techniques including, but not limited to, ray tracing to test underlying physical models of the diffraction, Bayesian approaches for refining model parameters, macromolecular modeling and refinement, neural networks for interpreting image features, as well as signal processing and denoising methods. - Extensive data analysis experience in crystallography or from more general bioimaging backgrounds are welcome. - We particularly encourage strong mathematical intuition and a track record of bringing new ideas and tools to fruition, as evidenced through written publication and clear presentation. Candidates should send an expression of interest, CV, and list of three references to Nick Sauter, nksauter(a)lbl.gov*. *Further details are posted at https://lbl.taleo.net/careersection/2/jobdetail.ftl?lang=en&job=82554. Nicholas K. Sauter, Ph. D. Computer Staff Scientist, Molecular Biophysics and Integrated Bioimaging Division Lawrence Berkeley National Laboratory 1 Cyclotron Rd., Bldg. 33R0345 Berkeley, CA 94720

1 0

Not wishing to raise this again but...
by R. D. Oeffner 13 Jun '16

13 Jun '16

Switching to git for the cctbx would be very liberating for developing and testing code I believe. If bisecting git hash values seems cumbersome one can always tag each commit with the number of previous commits made to the branch used for the build. These numbers are sequential and obviously straightforward to use for bisecting. We do this currently for the phaser git repo: https://git.csx.cam.ac.uk/x/cimr-phaser/phaser.git Rob Date: Mon, 13 Jun 2016 21:24:58 +0000 From: <richard.gildea(a)diamond.ac.uk> To: <cctbxbb(a)phenix-online.org> Subject: Re: [cctbxbb] Not wishing to raise this again but... Message-ID: <1BB183A24316FD4DA62A5F8B952D4291CC715321@exchmbx03> Content-Type: text/plain; charset="windows-1252" At least according to Google Trends, svn has been on a downward trend since about 2009/10, whereas git is still firmly on an upward trend (and overtook svn in popularity long ago): <https://www.google.co.uk/trends/explore#q=git%20commit%2C%20svn%20commit&cm…> https://www.google.co.uk/trends/explore#q=git%20commit%2C%20svn%20commit&cm… <https://www.google.co.uk/trends/explore#q=git%20commit%2C%20svn%20commit&cm…><https://www.google.co.uk/trends/explore#q=git%20commit%2C%20svn%20commit&cm…><https://www.google.co.uk/trends/explore#q=git%20commit%2C%20svn%20commit&cm…> [cid:162fa876-cc25-6335-73af-714b0957d71e] I think that avoiding a move to git is simply delaying the inevitable (and if the cctbx didn't ever move with the times then we'd still be stuck with CVS). In my opinion, the potential benefits to using git far outweigh the (slightly) added complexity. <https://www.google.co.uk/trends/explore#q=git%20commit%2C%20svn%20commit&cm…><https://www.google.co.uk/trends/explore#q=git%20commit%2C%20svn%20commit&cm…>Cheers, Richard ________________________________ From: cctbxbb-bounces(a)phenix-online.org [cctbxbb-bounces(a)phenix-online.org] on behalf of Nicholas Sauter [nksauter(a)lbl.gov] Sent: 13 June 2016 22:05 To: cctbx mailing list Subject: Re: [cctbxbb] Not wishing to raise this again but... I'd vote against the switch to git. Simple things, like svn update [remote directory] seem too complicated. As for slightly involved tasks like code bisection to locate the past commit that broke the test, well....I've never been able to figure this one out so I am pretty lost trying to debug DIALS. I do not wish to make things difficult for the maintenance of cctbx.xfel. Nick Nicholas K. Sauter, Ph. D. Computer Staff Scientist, Molecular Biophysics and Integrated Bioimaging Division Lawrence Berkeley National Laboratory 1 Cyclotron Rd., Bldg. 33R0345 Berkeley, CA 94720 (510) 486-5713 On Mon, Jun 13, 2016 at 8:52 AM, Luc Bourhis <luc_j_bourhis(a)mac.com<mailto:[email protected]>> wrote: On 13 Jun 2016, at 17:09, Graeme.Winter(a)Diamond.ac.uk<mailto:[email protected]> wrote: ? is there any enthusiasm out there for moving cctbx to git? The more I work with git the more I wish this existed properly for cctbx. I?ve raised this a bunch of times and always it is ?not right now? however ?a good idea? ? this?ll be the last time I ask? Could we move to git, please? I note the bootstrap script for Windows now has a git dependency even! +1 ? and time to share some thoughts! It would be great if we could get the cctbx up and running by just cloning, configuring and building without needing to download anything else. That could be done by adding SCons, Boost, and ccp4io as submodules in the cctbx repository. And I reckon ccp4io_adaptbx should be added as a subtree since it is developed by the same people as the cctbx. Just an idea to think about as I haven?t not worked out the details at all. Best wishes, Luc _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org<mailto:[email protected]> http://phenix-online.org/mailman/listinfo/cctbxbb -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

XQuartz 2.7.9 issue for OS X
by Billy Poon 11 Jun '16

11 Jun '16

Hi all, There seems to be an issue with XQuartz 2.7.9 on OS X when trying to render an OpenGL window (e.g. Coot, PyMOL, Phenix GLViewer) over SSH using X forwarding (i.e. ssh -Y). You will get an error that looks like X Error of failed request: BadValue (integer parameter out of range for operation) Major opcode of failed request: 150 (GLX) Minor opcode of failed request: 3 (X_GLXCreateContext) It seems like indirect rendering of a GLX context does not work properly on 2.7.9. Downgrading to XQuartz 2.7.8 works for me. -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

1 0

Who is using bootstrap.py on Windows?
by Robert Oeffner 10 Jun '16

10 Jun '16

Hi all, I would like to know if there are teams other than Phenix and Dials-west in Berkeley doing regular builds on Windows using cctbx_project/libtbx/auto_build/bootstrap.py. I plan to make some simplifications of this script using the ssh client that is part of Git for Windows rather than the one found in the PuTTY suite which we use currently. Many thanks, Robert

2 1

Berkeley Server Downtime (6/3/2016)
by Billy Poon 01 Jun '16

01 Jun '16

Dear all, On Friday, June 3, our servers will be shutdown starting at 6:30 am Pacific for scheduled maintenance in our building (7 - 8 am). This will affect, - The Phenix web site (http://phenix-online.org/) - The DIALS West web site (http://dials.lbl.gov/) - The CCI web site (http://cci.lbl.gov/) - The Structural Biology Technology Portal (http://technology.sbkb.org/) We expect that the servers will be available by 8:30 am Pacific at the latest. Bug reports and emails requesting help should be queued and we will receive them once the servers are back online. We apologize for any inconvenience. -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

1 0

The curious incident of the test at install time
by R. D. Oeffner 13 May '16

13 May '16

Hi Markus, Those changes were made by me and Oleg. I just tested a fresh installation of phenix-dev-2407-intel-linux-2.6-centos5 which is built with cctbx_project revision 24442. The test of cctbx_project/cctbx/regression/tst_grm_pickling.py passed without problems. I'm not aware of why this test is failing on dials after installation. Since the test is to do with the phenix restraints manager perhaps this test is not critical to dials. Perhaps the phenix restraints manager code together with this test should be moved into the phenix repository but that is not my call. Rob -----Original Message----- From: cctbxbb-request(a)phenix-online.org Sent: Friday, May 13, 2016 20:00 To: cctbxbb(a)phenix-online.org Subject: cctbxbb Digest, Vol 86, Issue 7 Message: 2 Date: Fri, 13 May 2016 09:47:18 +0000 From: <markus.gerstel(a)diamond.ac.uk> To: <cctbxbb(a)phenix-online.org> Subject: [cctbxbb] The curious incident of the test at install time Message-ID: <415F7F573406BB498256C68158D45674612998A2(a)EXCHMBX01.fed.cclrc.ac.uk> Content-Type: text/plain; charset=iso-8859-1 Hi everyone, we have observed the test cctbx_project/cctbx/regression/tst_grm_pickling.py to be failing, however only on an installation installed from the installer. It appears to run successfully on our bootstrapped developer build. I presume that the test requires some external data that we do not distribute? The relevant change occurred between r24437 and r24441, and is almost certainly r24439, which enabled the failing bit of the test. Stacktrace below. -Markus $ libtbx.python /dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py Time pickling/unpickling: 0.0187, 0.0097 Traceback (most recent call last): File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py", line 198, in <module> exercise_all(sys.argv[1:]) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py", line 195, in exercise_all test_nucleic_acid(mon_lib_srv, ener_lib) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py", line 177, in test_nucleic_acid log=null_out()) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 5455, in run log=log) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 5005, in __init__ restraints_loading_flags=restraints_loading_flags, File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2956, in __init__ log=log, File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2343, in __init__ m_j=mm) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 1568, in get_lib_link return mon_lib_srv.link_link_id_dict["rna3p"] KeyError: 'rna3p' -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom ------------------------------ Message: 3 Date: Fri, 13 May 2016 09:50:03 +0000 From: <Graeme.Winter(a)diamond.ac.uk> To: <cctbxbb(a)phenix-online.org> Subject: Re: [cctbxbb] The curious incident of the test at install time Message-ID: <13999B6FC038EB44B067771E3CFBD091BC2E3E7A(a)EXCHMBX03.fed.cclrc.ac.uk> Content-Type: text/plain; charset="us-ascii" Markus I got this error this morning in the regular update / compile shuffle and it's on my to-nag list, also feel like I have seen this before... Cheerio G -----Original Message----- From: cctbxbb-bounces(a)phenix-online.org [mailto:[email protected]] On Behalf Of markus.gerstel(a)diamond.ac.uk Sent: 13 May 2016 10:47 To: cctbxbb(a)phenix-online.org Subject: [cctbxbb] The curious incident of the test at install time Hi everyone, we have observed the test cctbx_project/cctbx/regression/tst_grm_pickling.py to be failing, however only on an installation installed from the installer. It appears to run successfully on our bootstrapped developer build. I presume that the test requires some external data that we do not distribute? The relevant change occurred between r24437 and r24441, and is almost certainly r24439, which enabled the failing bit of the test. Stacktrace below. -Markus $ libtbx.python /dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py Time pickling/unpickling: 0.0187, 0.0097 Traceback (most recent call last): File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py", line 198, in <module> exercise_all(sys.argv[1:]) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py", line 195, in exercise_all test_nucleic_acid(mon_lib_srv, ener_lib) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py", line 177, in test_nucleic_acid log=null_out()) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 5455, in run log=log) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 5005, in __init__ restraints_loading_flags=restraints_loading_flags, File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2956, in __init__ log=log, File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2343, in __init__ m_j=mm) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 1568, in get_lib_link return mon_lib_srv.link_link_id_dict["rna3p"] KeyError: 'rna3p' -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org http://phenix-online.org/mailman/listinfo/cctbxbb ------------------------------ _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org http://phenix-online.org/mailman/listinfo/cctbxbb End of cctbxbb Digest, Vol 86, Issue 7 **************************************

1 0

The curious incident of the test at install time
by markus.gerstel＠diamond.ac.uk 13 May '16

13 May '16

Hi everyone, we have observed the test cctbx_project/cctbx/regression/tst_grm_pickling.py to be failing, however only on an installation installed from the installer. It appears to run successfully on our bootstrapped developer build. I presume that the test requires some external data that we do not distribute? The relevant change occurred between r24437 and r24441, and is almost certainly r24439, which enabled the failing bit of the test. Stacktrace below. -Markus $ libtbx.python /dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py Time pickling/unpickling: 0.0187, 0.0097 Traceback (most recent call last): File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py", line 198, in <module> exercise_all(sys.argv[1:]) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py", line 195, in exercise_all test_nucleic_acid(mon_lib_srv, ener_lib) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py", line 177, in test_nucleic_acid log=null_out()) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 5455, in run log=log) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 5005, in __init__ restraints_loading_flags=restraints_loading_flags, File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2956, in __init__ log=log, File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2343, in __init__ m_j=mm) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 1568, in get_lib_link return mon_lib_srv.link_link_id_dict["rna3p"] KeyError: 'rna3p' -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

easy_mp stack traces
by Graeme.Winter＠diamond.ac.uk 13 May '16

13 May '16

Dear Gabor, some time back Markus added the capability to retain stack traces through easy_mp execution. These were reverted which now means that by default the stack traces are lost :( I don't have a clear understanding what the exact design of the newly built scheduling/stacktrace system is, perhaps you could explain this since we’re in the dark here. We are however pretty certain that the corner cases that lead to you modifying the code do not apply to us with DIALS or xia2. Could you please advise how we can now enable the stacktrace to be preserved globally, in all cases, and for all dials.*/xia2.* programs? Just today this has come up twice, and is a support issue for us, see e.g. https://github.com/dials/dials/issues/139 where we are simply flying blind. Thanks in advance and best wishes, Graeme & xia2 / DIALS team -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

Installation with distutils?
by Luc Bourhis 09 May '16

09 May '16

Hi fellow cctbx developers, unless I am missing something, nobody has written a “setup.py” to install the cctbx with distutils in Python standard site-packages? If so, I am planning to contribute one as this would make it easier for us at Bruker. I am just asking in case I would duplicate existing work. Best wishes, Luc

3 4

problem when using Python to convert from numpy array to flex.int array
by Jun Aishima 04 May '16

04 May '16

I have used the bootstrap method to build cctbx with my own Python (in a virtual environment). I setup my virtual environment, then source $CCTBX/build/setpaths.sh (bash). I am currently trying to turn a detector image (uint16) read in with fabio, which results in a numpy ndarray, and then turn that into a scitbx flex array. Since I've had problems doing this in my system, I've written a little script that reproduces the problem: ********** import fabio file=fabio.open('testcrystal_1_001.img') # adsc detector image file.data[0] from scitbx.array_family import flex flex.int(file.data[3071]) # this is a 3072x3072 image. For some reason, row 0 causes no problems. ********** At the end of this email is the stack trace I get. Has anybody had similar issues? I first saw this when I was trying to convert the entire image into a flex array, but now I've seen it with only one row [3071], while still working for [0:1500], so it doesn't seem like a simple memory issue. Any help would be appreciated. Jun Aishima ********** show_stack(1): <stdin>(1) <module> libc backtrace (23 frames, most recent call last): /home/aishimaj/spotter/bin/python() [0x414199] /lib64/libc.so.6(__libc_start_main+0xfd) [0x366d21ed5d] /home/aishimaj/spotter/bin/python(Py_Main+0xb22) [0x4150b2] /home/aishimaj/spotter/bin/python(PyRun_AnyFileExFlags+0x6b) [0x4ce82b] /home/aishimaj/spotter/bin/python(PyRun_InteractiveLoopFlags+0x4e) [0x4ce01e] /home/aishimaj/spotter/bin/python(PyRun_InteractiveOneFlags+0x1ff) [0x4cdd9f] /home/aishimaj/spotter/bin/python(PyEval_EvalCode+0x32) [0x4ad362] /home/aishimaj/spotter/bin/python(PyEval_EvalCodeEx+0x877) [0x4ad257] /home/aishimaj/spotter/bin/python(PyEval_EvalFrameEx+0x6a0b) [0x4aa8bb] /home/aishimaj/spotter/bin/python(PyObject_Call+0x47) [0x419ba7] /home/aishimaj/spotter/bin/python() [0x46df9f] /home/aishimaj/spotter/bin/python() [0x47623f] /home/aishimaj/spotter/bin/python(PyObject_Call+0x47) [0x419ba7] /home/aishimaj/spotter/bin/python() [0x42296f] /home/aishimaj/spotter/bin/python(PyObject_Call+0x47) [0x419ba7] /beamline/phenix/bootstrap/build/lib/libboost_python.so(+0x2d088) [0x7f5561101088] /beamline/phenix/bootstrap/build/lib/libboost_python.so(boost::python::handle_exception_impl(boost::function0<void>)+0x43) [0x7f5561112013] /beamline/phenix/bootstrap/build/lib/libboost_python.so(+0x2fe08) [0x7f5561103e08] /beamline/phenix/bootstrap/build/lib/libboost_python.so(boost::python::objects::function::call(_object*, _object*) const+0xef) [0x7f55611031ff] /beamline/phenix/bootstrap/build/lib/scitbx_array_family_flex_ext.so(boost::python::objects::signature_py_function_impl<boost::python::detail::caller<scitbx::af::versa<int, scitbx::af::flex_grid<scitbx::af::small<long, 10ul> > >* (*)(boost::python::numeric::array const&), boost::python::detail::constructor_policy<boost::python::default_call_policies>, boost::mpl::vector2<scitbx::af::versa<int, scitbx::af::flex_grid<scitbx::af::small<long, 10ul> > >*, boost::python::numeric::array const&> >, boost::mpl::v_item<void, boost::mpl::v_item<boost::python::api::object, boost::mpl::v_mask<boost::mpl::vector2<scitbx::af::versa<int, scitbx::af::flex_grid<scitbx::af::small<long, 10ul> > >*, boost::python::numeric::array const&>, 1>, 1>, 1> >::operator()(_object*, _object*)+0x52) [0x7f555ed68452] /beamline/phenix/bootstrap/build/lib/scitbx_array_family_flex_ext.so(scitbx::af::boost_python::flex_int_from_numpy_array(boost::python::numeric::array const&)+0x29) [0x7f555f039f79] /beamline/phenix/bootstrap/build/lib/scitbx_array_family_flex_ext.so(scitbx::af::versa<int, scitbx::af::flex_grid<scitbx::af::small<long, 10ul> > > scitbx::af::boost_python::versa_flex_from_numpy_array<int>(boost::python::numeric::array const&)+0x1c87) [0x7f555f0415c7] /lib64/libc.so.6() [0x366d2326a0] Segmentation fault (Python and libc call stacks above) This crash may be due to a problem in any imported Python module, including modules which are not part of the cctbx project. To disable the traps leading to this message, define these environment variables (e.g. assign the value 1): BOOST_ADAPTBX_FPE_DEFAULT BOOST_ADAPTBX_SIGNALS_DEFAULT This will NOT solve the problem, just mask it, but may allow you to proceed in case it is not critical. Jun Aishima | Post Doc - MX | Australian Synchrotron p: (03) 8540 4114 | f: (03) 8540 4200 jun.aishima(a)synchrotron.org.au<mailto:[email protected]> | www.synchrotron.org.au 800 Blackburn Road, Clayton, Victoria 3168

3 6

Algorithm developer postdoc opening at Berkeley National Lab
by Nicholas Sauter 28 Apr '16

28 Apr '16

Berkeley Lab's Molecular Biophysics and Integrated Bioimaging Division <http://biosciences.lbl.gov/divisions/mbib/> is looking for a Computational Biologist Postdoctoral Fellow to work in the Sauter group. We are seeking an algorithm developer for the increasingly complex analysis of large diffraction datasets in structural biology. Current projects utilize XFEL crystallography and spectroscopy to investigate the photosynthetic mechanism of water splitting and to probe other metalloenzyme reactions. We also wish to test whether diffuse scattering can reveal correlated atomic motions in crystals. Many problems remain to be solved, including the details of how to optimally merge datasets from thousands of crystals. Our software development projects (including packages such as DIALS <http://dials.lbl.gov> and cctbx.xfel <http://cci.lbl.gov/xfel>) have been highlighted in several high-impact publications listed HERE <http://biosciences.lbl.gov/profiles/nicholas-sauter>. Candidates should have expertise in one or more computational techniques including, but not limited to, ray tracing to test underlying physical models of the diffraction, Bayesian approaches for refining model parameters, macromolecular modeling and refinement, neural networks for interpreting image features, as well as signal processing and denoising methods. Extensive data analysis experience in crystallography or from more general bioimaging backgrounds are welcome. We particularly encourage strong mathematical intuition and a track record of bringing new ideas and tools to fruition, as evidenced through written publication and clear presentation. Candidates should send an expression of interest, CV, and list of three references to *Nick Sauter, nksauter(a)lbl.gov <nksauter(a)lbl.gov>.* Further details are posted at https://lbl.taleo.net/careersection/2/jobdetail.ftl?lang=en&job=82354. Nicholas K. Sauter, Ph. D. Computer Staff Scientist, Molecular Biophysics and Integrated Bioimaging Division Lawrence Berkeley National Laboratory 1 Cyclotron Rd., Bldg. 33R0345 Berkeley, CA 94720

1 0

Re: [cctbxbb] new fail on windows
by R. D. Oeffner 25 Apr '16

25 Apr '16

Markus, I'd say that this is a wrong take on the current state of affairs regarding supporting Windows. I believe line 22 in easy_mp.py you refer to is supposed to be one of the last warnings one would get if one were to use some of the functions there with Windows in an unsupported manner, say if objects can't be pickled. I don't think it means this file was written with non-windows support intentionally. I appreciate people code on different platforms but the way of working shouldn't be aloof denial of the existence of Windows. In the short term until you get a Windows PC perhaps Billy Poon might grant you access to one of the virtual Windows systems he has set up for testing. Otherwise I'm happy to run tests for you every now and then on my PC. Regards, Rob -----Original Message----- From: cctbxbb-request(a)phenix-online.org Sent: Saturday, April 23, 2016 20:00 To: cctbxbb(a)phenix-online.org Subject: cctbxbb Digest, Vol 85, Issue 23 Send cctbxbb mailing list submissions to cctbxbb(a)phenix-online.org To subscribe or unsubscribe via the World Wide Web, visit http://phenix-online.org/mailman/listinfo/cctbxbb or, via email, send a message with subject or body 'help' to cctbxbb-request(a)phenix-online.org You can reach the person managing the list at cctbxbb-owner(a)phenix-online.org When replying, please edit your Subject line so it is more specific than "Re: Contents of cctbxbb digest..." Today's Topics: 1. Re: new fail on windows (markus.gerstel(a)diamond.ac.uk) ---------------------------------------------------------------------- Message: 1 Date: Fri, 22 Apr 2016 21:26:38 +0000 From: <markus.gerstel(a)diamond.ac.uk> To: <cctbxbb(a)phenix-online.org> Subject: Re: [cctbxbb] new fail on windows Message-ID: <415F7F573406BB498256C68158D456746125CE10(a)EXCHMBX01.fed.cclrc.ac.uk> Content-Type: text/plain; charset=iso-8859-1 Hi Nigel, Should be resolved now. It is true that we do not currently test on windows. In this particular case it turns out neither does anyone else, cf. https://sourceforge.net/p/cctbx/code/HEAD/tree/trunk/libtbx/easy_mp.py line 22 which causes the other easy_mp tests to be skipped on Windows. This will make it kind of difficult to support windows software building on top of it, so at some point this needs to be addressed properly. Best wishes & good weekend -Markus ________________________________ From: cctbxbb-bounces(a)phenix-online.org [cctbxbb-bounces(a)phenix-online.org] on behalf of Nigel Moriarty [nwmoriarty(a)lbl.gov] Sent: Friday, April 22, 2016 17:39 To: cctbx mailing list Subject: Re: [cctbxbb] new fail on windows On Fri, Apr 22, 2016 at 1:35 AM, <markus.gerstel(a)diamond.ac.uk<mailto:[email protected]>> wrote: Hopefully this is fixed with r24353. Afraid not. Is it true that you are not testing this windows feature on a windows machine? Regarding junit, I'll talk with Aaron when he returns. Cheers Nigel --- Nigel W. Moriarty Building 33R0349, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709 Email : NWMoriarty(a)LBL.gov Fax : 510-486-5909 Web : CCI.LBL.gov<http://CCI.LBL.gov> -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom ------------------------------ _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org http://phenix-online.org/mailman/listinfo/cctbxbb End of cctbxbb Digest, Vol 85, Issue 23 ***************************************

2 1

new fail on windows
by Nigel Moriarty 23 Apr '16

23 Apr '16

******************************************************************************** ERROR: 1 TEST FAILURES. PLEASE FIX BEFORE COMMITTING CODE. ******************************************************************************** Standard error: Traceback (most recent call last): File "c:\users\builder\slave\cctbx-nightly-intel-windows\modules\cctbx_project\libtbx\tst_easy_mp.py", line 210, in <module> run(args=sys.argv[1:]) File "c:\users\builder\slave\cctbx-nightly-intel-windows\modules\cctbx_project\libtbx\tst_easy_mp.py", line 205, in run check_if_stacktrace_is_propagated_properly(method='multiprocessing') File "c:\users\builder\slave\cctbx-nightly-intel-windows\modules\cctbx_project\libtbx\tst_easy_mp.py", line 187, in check_if_stacktrace_is_propagated_properly assert isinstance(e, exceptions.ZeroDivisionError) or "ZeroDivisionError" in str(e), "Exception type mismatch: %s" % str(e) AssertionError: Exception type mismatch: Can't pickle <function may_divide_by_zero at 0x01C06470>: it's not found as __main__.may_divide_by_zero Traceback (most recent call last): File "<string>", line 1, in <module> File "C:\Users\builder\slave\cctbx-nightly-intel-windows\base\bin\python\lib\multiprocessing\forking.py", line 381, in main self = load(from_parent) File "C:\Users\builder\slave\cctbx-nightly-intel-windows\base\bin\python\lib\pickle.py", line 1378, in load return Unpickler(file).load() File "C:\Users\builder\slave\cctbx-nightly-intel-windows\base\bin\python\lib\pickle.py", line 858, in load dispatch[key](self) File "C:\Users\builder\slave\cctbx-nightly-intel-windows\base\bin\python\lib\pickle.py", line 880, in load_eof raise EOFError EOFError Cheers Nigel --- Nigel W. Moriarty Building 33R0349, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709 Email : NWMoriarty(a)LBL.gov Fax : 510-486-5909 Web : CCI.LBL.gov

3 4

Re: [cctbxbb] cctbxbb Digest, Vol 85, Issue 21
by R.D. Oeffner 22 Apr '16

22 Apr '16

Hi Markus, I was gonna ask why don't you guys set up a windows machine and test for the cctbx crash on that? The VC++9.0 compiler is free and if you use the python that comes in a phenix for windows installation the necessary modules are all present. Rob On 22/04/2016 09:35, cctbxbb-request(a)phenix-online.org wrote: > Send cctbxbb mailing list submissions to > cctbxbb(a)phenix-online.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://phenix-online.org/mailman/listinfo/cctbxbb > or, via email, send a message with subject or body 'help' to > cctbxbb-request(a)phenix-online.org > > You can reach the person managing the list at > cctbxbb-owner(a)phenix-online.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of cctbxbb digest..." > > > Today's Topics: > > 1. Re: new fail on windows (markus.gerstel(a)diamond.ac.uk) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 22 Apr 2016 08:35:42 +0000 > From: <markus.gerstel(a)diamond.ac.uk> > To: <cctbxbb(a)phenix-online.org> > Subject: Re: [cctbxbb] new fail on windows > Message-ID: > <415F7F573406BB498256C68158D456746125C23B(a)EXCHMBX01.fed.cclrc.ac.uk> > Content-Type: text/plain; charset="utf-8" > > Hi Nigel, > > Hopefully this is fixed with r24353. > > One question: Do you think it would be possible to install > https://pypi.python.org/pypi/junit-xml on your build machines, run > libtbx.run_tests_parallel with the output_junit_xml=True option, and > collect the output.xml files somewhere where we can download them? > > At the moment we do monitor the buildbot status - but only in a very > coarse way. We only see test failures on a high level, see attached > screenshot. As you can see there are a number of things broken, some > of them for quite some time, so because on that level there is always > something guaranteed to fail the overall status we see is always set > to ?warning?. I don?t think any of us bothers anymore looking into the > details, a number of errors are about things we can?t do anything > about anyway. > > However, if you could give us a collection of output.xml?s we could, > instead of taking the overall build status, use the test results and > set up a DIALS-East internal mail on the buildbot test status. Then we > would pick up these things earlier. > > Best wishes, > -Markus > > From: cctbxbb-bounces(a)phenix-online.org > [mailto:[email protected]] On Behalf Of Nigel Moriarty > Sent: 21 April 2016 17:41 > To: cctbx mailing list > Subject: [cctbxbb] new fail on windows > > > ******************************************************************************** > > ERROR: 1 TEST FAILURES. PLEASE FIX BEFORE COMMITTING CODE. > > ******************************************************************************** > > > > Standard error: > > Traceback (most recent call last): > > File > "c:\users\builder\slave\cctbx-nightly-intel-windows\modules\cctbx_project\libtbx\tst_easy_mp.py", > line 210, in <module> > > run(args=sys.argv[1:]) > > File > "c:\users\builder\slave\cctbx-nightly-intel-windows\modules\cctbx_project\libtbx\tst_easy_mp.py", > line 205, in run > > > check_if_stacktrace_is_propagated_properly(method='multiprocessing') > > File > "c:\users\builder\slave\cctbx-nightly-intel-windows\modules\cctbx_project\libtbx\tst_easy_mp.py", > line 187, in check_if_stacktrace_is_propagated_properly > > assert isinstance(e, exceptions.ZeroDivisionError) or > "ZeroDivisionError" in str(e), "Exception type mismatch: %s" % str(e) > > AssertionError: Exception type mismatch: Can't pickle <function > may_divide_by_zero at 0x01C06470>: it's not found as > __main__.may_divide_by_zero > > Traceback (most recent call last): > > File "<string>", line 1, in <module> > > File > "C:\Users\builder\slave\cctbx-nightly-intel-windows\base\bin\python\lib\multiprocessing\forking.py", > line 381, in main > > self = load(from_parent) > > File > "C:\Users\builder\slave\cctbx-nightly-intel-windows\base\bin\python\lib\pickle.py", > line 1378, in load > > return Unpickler(file).load() > > File > "C:\Users\builder\slave\cctbx-nightly-intel-windows\base\bin\python\lib\pickle.py", > line 858, in load > > dispatch[key](self) > > File > "C:\Users\builder\slave\cctbx-nightly-intel-windows\base\bin\python\lib\pickle.py", > line 880, in load_eof > > raise EOFError > > EOFError > Cheers > > Nigel > > --- > Nigel W. Moriarty > Building 33R0349, Molecular Biophysics and Integrated Bioimaging > Lawrence Berkeley National Laboratory > Berkeley, CA 94720-8235 > Phone : 510-486-5709 Email : > NWMoriarty(a)LBL.gov<mailto:[email protected]> > Fax : 510-486-5909 Web : CCI.LBL.gov<http://CCI.LBL.gov> > > -- > This e-mail and any attachments may contain confidential, copyright > and or privileged material, and are for the use of the intended > addressee only. If you are not the intended addressee or an authorised > recipient of the addressee please notify us of receipt by returning > the e-mail and do not use, copy, retain, distribute or disclose the > information in or attached to the e-mail. > Any opinions expressed within this e-mail are those of the individual > and not necessarily of Diamond Light Source Ltd. > Diamond Light Source Ltd. cannot guarantee that this e-mail or any > attachments are free from viruses and we cannot accept liability for > any damage which you may sustain as a result of software viruses which > may be transmitted in or with the message. > Diamond Light Source Limited (company no. 4375679). Registered in > England and Wales with its registered office at Diamond House, Harwell > Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United > Kingdom >

2 1

Trapping NaN or not trapping NaN
by Luc Bourhis 13 Apr '16

13 Apr '16

Richard and I have been hammering at our build system and code to try to solve several crashes on OS X with Xcode 7.3. Several of them came from the fact that we trap NaN whereas Xcode 7.3 aggressive optimisations produce code that may create a NaN but still deliver the correct final result. A trivial example came from cctbx/xray/conversions.h. We have if (f_sq > 0) f = std::sqrt(f_sq); else f = 0; With the version of clang shipping with Xcode 7.3, both branches are evaluated in parallel before selecting the result based on the test f_sq > 0. This is done so as not to stall the pipeline and this can result in much faster code indeed. But that means that every time f_sq < 0, a NaN will be produced, and in the context of our tests, this means we crash. Richard and I avoided the elephant in the room, stop to trap NaN, and as a result came with uglier and uglier fixes. Moreover, we are now at the mercy of the compiler applying such an optimisation to another file at the next update of Xcode, which would then force us to fix the compilation of said file in the relevant SConscript. It’s a loosing game. Thus I reckon that at least on OS X, we should stop trapping NaN. That will leave us with the true clang regression for which I will try to submit reduced cases to the clang bugzilla. Best wishes, Luc

5 7

Boost broken on mac?
by markus.gerstel＠diamond.ac.uk 12 Apr '16

12 Apr '16

Dials west is currently observing test failures on Mac related to Boost. To exclude any errors on our side I have run things twice clean from bootstrap: One of a number of affected tests: libtbx.python cctbx/xray/boost_python/tst_xray.py show_stack(1): /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/cctbx_project/cctbx/xray/boost_python/tst_xray.py(381) exercise_conversions show_stack(2): /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/cctbx_project/cctbx/xray/boost_python/tst_xray.py(1993) run show_stack(3): /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/cctbx_project/cctbx/xray/boost_python/tst_xray.py(2009) <module> libc backtrace (29 frames, most recent call last): 30 Python 0x0000000100000d71 Python + 3441 29 Python 0x0000000100000e58 Python + 3672 28 Python 0x00000001000ef06f Py_Main + 3135 27 Python 0x00000001000d78be PyRun_SimpleFileExFlags + 702 26 Python 0x00000001000d7d84 PyRun_FileExFlags + 164 25 Python 0x00000001000aea06 PyEval_EvalCode + 54 24 Python 0x00000001000af244 PyEval_EvalCodeEx + 2100 23 Python 0x00000001000b25c8 PyEval_EvalFrameEx + 12280 22 Python 0x00000001000b6ab6 fast_function + 118 21 Python 0x00000001000af244 PyEval_EvalCodeEx + 2100 20 Python 0x00000001000b25c8 PyEval_EvalFrameEx + 12280 19 Python 0x00000001000b6ab6 fast_function + 118 18 Python 0x00000001000af244 PyEval_EvalCodeEx + 2100 17 Python 0x00000001000b4198 PyEval_EvalFrameEx + 19400 16 Python 0x0000000100011023 PyObject_Call + 99 15 Python 0x000000010006ad8b type_call + 347 14 Python 0x00000001000708af slot_tp_init + 175 13 Python 0x0000000100011023 PyObject_Call + 99 12 Python 0x000000010001f0d8 instancemethod_call + 232 11 Python 0x0000000100011023 PyObject_Call + 99 10 libboost_python.dylib 0x0000000101888663 boost::python::objects::function_call(_object*, _object*, _object*) + 83 9 libboost_python.dylib 0x000000010188ce71 boost::python::handle_exception_impl(boost::function0<void>) + 81 8 libboost_python.dylib 0x00000001018889ca boost::detail::function::void_function_ref_invoker0<boost::python::objects::(anonymous namespace)::bind_return, void>::invoke(boost::detail::function::function_buffer&) + 26 7 libboost_python.dylib 0x000000010188693a boost::python::objects::function::call(_object*, _object*) const + 874 6 cctbx_xray_ext.so 0x0000000107c8e1c5 boost::python::detail::caller_arity<3u>::impl<void (*)(_object*, scitbx::af::const_ref<double, scitbx::af::trivial_accessor> const&, scitbx::af::const_ref<double, scitbx::af::trivial_accessor> const&), boost::python::default_call_policies, boost::mpl::vector4<void, _object*, scitbx::af::const_ref<double, scitbx::af::trivial_accessor> const&, scitbx::af::const_ref<double, scitbx::af::trivial_accessor> const&> >::operator()(_object*, _object*) + 229 5 cctbx_xray_ext.so 0x0000000107c91f44 boost::python::objects::make_holder<2>::apply<boost::python::objects::value_holder<cctbx::xray::array_f_sq_as_f<cctbx::xray::f_sq_as_f_xtal_3_7, double> >, boost::mpl::joint_view<boost::python::detail::drop1<boost::python::detail::type_list<scitbx::af::const_ref<double, scitbx::af::trivial_accessor> const&, scitbx::af::const_ref<double, scitbx::af::trivial_accessor> const&, boost::python::optional<double const&, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_>, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_> >, boost::python::optional<double const&, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_> > >::execute(_object*, scitbx::af::const_ref<double, scitbx::af::trivial_accessor> const&, scitbx::af::const_ref<double, scitbx::af::trivial_accessor> const&) + 100 4 ??? 0x0000000000000002 0x0 + 2 3 libsystem_platform.dylib 0x00007fff8caabeaa _sigtramp + 26 2 boost_python_meta_ext.so 0x0000000101841370 initboost_python_meta_ext + 0 Floating-point error (Python and libc call stacks above) This crash may be due to a problem in any imported Python module, including modules which are not part of the cctbx project. To disable the traps leading to this message, define these environment variables (e.g. assign the value 1): BOOST_ADAPTBX_FPE_DEFAULT BOOST_ADAPTBX_SIGNALS_DEFAULT This will NOT solve the problem, just mask it, but may allow you to proceed in case it is not critical. Any ideas? Best wishes, -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

6 35

cctbx can't build without iotbx
by Luc Bourhis 06 Apr '16

06 Apr '16

Hi all, am I getting mad or % libtbx.configure —only cctbx % libtbx.scons will result in a linking error because of the line marked with <<<<< in cctbx/maptbx/SConscript ? if (not env_etc.no_boost_python): Import("env_cctbx_boost_python_ext") env_bpl = env_cctbx_boost_python_ext.Clone() env_etc.include_registry.append( env=env_bpl, paths=env_etc.cctbx_common_includes) env_bpl.Prepend(LIBS=["asymmetric_map", "iotbx_xplor", "cctbx_sgtbx_asu”, <<<<<<<<<<<<<<<<<<<<<<<<<<<<< "cctbx", "omptbx"]) env_bpl.SharedLibrary( target="#lib/cctbx_asymmetric_map_ext", source=["asymmetric_map_ext.cpp"]) This code was introduced in Dec 2013!!! The only workaround % libtbx.configure cctbx iotbx So much for the modularisation with the libtbx_config’s… Best wishes, Luc

2 1

Re: [cctbxbb] sequential revision number from git repo
by markus.gerstel＠diamond.ac.uk 06 Apr '16

06 Apr '16

Hi Johan, No, not really. .gitversion is not in the repository and only kept (and updated) locally. Basically the logic is: Can I determine the current version number (ie do I have git installed and the repository present) - if so, write .gitversion. Otherwise, attempt to read .gitversion and use the information therein (this covers the route via the installer). Hence, if you download the zip file frm github you will not have a .gitversion file. But really in this case we can assume that the user knows what they're doing. -Markus On 5 Apr 2016 16:34, "Hattne, Johan" <hattnej(a)janelia.hhmi.org> wrote: Would this not mean that .gitversion is updated on every commit, at least if as in our case, it not only records the release version but also the current commit? @Rob: Apologies for my liberal use of “hook”. What our build system does is that it generates what essentially amounts to Markus’s .gitversion whenever we run “make dist” (using “git describe” or some such, it need not involve the network). There is little need for this file when compiling from a sources under git control, because it can be done on the fly. // Cheers; Johan > On Apr 5, 2016, at 09:47, markus.gerstel(a)diamond.ac.uk wrote: > > Hi Johan, > > We generate a file named '.gitversion' that lives in the main directory. That way the version number survives being packed up in an installer. > If you download the code via the "Download ZIP" button on Github there is no way (as far as I know) to recover any version information, so we fall back to a default of "DIALS 1.dev" ("DIALS 1.1" for the release branch) > > -Markus > > -----Original Message----- > From: cctbxbb-bounces(a)phenix-online.org [mailto:[email protected]] On Behalf Of Hattne, Johan > Sent: 05 April 2016 14:21 > To: cctbx mailing list > Subject: Re: [cctbxbb] sequential revision number from git repo > > Hi Markus; > > I didn’t actually read the code, but it looks like we’re doing something similar over here, and I find the the tricky bit is how to deal with any kind of releases that don’t have all the little git bits. We sort that out by having hooks in the “make dist” equivalent, which basically generate a version file that is included in the release tarball. > > However, it was recently pointed out to us that that all falls apart if someone uses the "Download ZIP” button from the code tab, because then the hooks are not run. I haven’t even attempted to RTFM around this, and if you or anybody else here has a solution I won’t have to. > > Do you by any chance? > > // Cheers; Johan > > >> On Apr 5, 2016, at 08:55, markus.gerstel(a)diamond.ac.uk wrote: >> >> Hi Rob, >> >> This is close to what we do with DIALS. >> If you run dials.version you may get e.g. >> DIALS 1.dev.132-g7042805 >> So this is the 132nd commit after the commit we tagged "1.dev", which >> happened when we branched for the 1.1 stable release so we could keep >> the version numbers sufficiently different >> >> If you run DIALS 1.1, for example the version from CCP4, you might get >> DIALS 1.1.3-gb5528a5-release The 3rd commit on the 1.1 release branch. >> >> Same with xia2: >> XIA2 0.4.0.301-g7f16ab6 >> >> In xia2 we also add the branch name if it is not 'master', so the version distributed with DIALS 1.1 is: >> XIA2 0.4.0.289-g2a1e557-dials-1.1 >> >> We kept the commit ID in - and I would recommend to keep it with cctbx, too. >> The reason for this is: >> - The number on its own does not uniquely identify a commit, as you >> might be on a different branch (although we do put in the branch >> names, too) >> - Because then I can go to >> https://github.com/dials/dials/commit/b5528a5 and see everything about >> that commit >> - Because then I can do git checkout b55528a5 and instantly have that particular code version on my machine to fiddle with things. >> >> To see the magic happen, check >> https://github.com/dials/dials/blob/master/util/version.py or >> https://github.com/xia2/xia2/blob/master/XIA2Version.py >> >> -Markus >> >> >> -----Original Message----- >> From: cctbxbb-bounces(a)phenix-online.org >> [mailto:[email protected]] On Behalf Of R.D. Oeffner >> Sent: 05 April 2016 13:38 >> To: cctbxbb(a)phenix-online.org >> Subject: [cctbxbb] sequential revision number from git repo >> >> Hi, >> >> We are slowly moving phaser svn to git. One issue that we would like to address is if there is a practical way of conveying a linear commit history from the git repo. >> >> One approach is to just count the number of revisions that was made to the master branch in the git repo with "git rev-list --count HEAD" >> and then pretend that number is a revision number. This enables us to tell users "This bug was fixed after revision 6456 of phaser" rather than "This bug was fixed after commit 29b0a1c... of phaser". Is there any pitfalls with this approach? >> >> What workarounds do other people using git come up with? >> >> >> >> Rob >> >> >> -- >> Robert Oeffner, Ph.D. >> Research Associate, The Read Group >> Department of Haematology, >> Cambridge Institute for Medical Research University of Cambridge >> Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road >> Cambridge CB2 0XY >> >> www.cimr.cam.ac.uk/investigators/read/index.html<http://www.cimr.cam.ac.uk/investigators/read/index.html> >> tel: +44(0)1223 763234 >> mobile: +44(0)7712 887162 >> _______________________________________________ >> cctbxbb mailing list >> cctbxbb(a)phenix-online.org >> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> -- >> This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. >> Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. >> Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. >> Diamond Light Source Limited (company no. 4375679). Registered in >> England and Wales with its registered office at Diamond House, Harwell >> Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United >> Kingdom >> >> >> _______________________________________________ >> cctbxbb mailing list >> cctbxbb(a)phenix-online.org >> http://phenix-online.org/mailman/listinfo/cctbxbb > > Research Specialist @ Gonen Lab ____________________________________________________ > Janelia Research Campus * 19700 Helix Drive Ashburn, VA 20147 * +1 (571) 209-4000 extension 3376 > > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org > http://phenix-online.org/mailman/listinfo/cctbxbb > > -- > This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. > Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. > Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. > Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org > http://phenix-online.org/mailman/listinfo/cctbxbb Research Specialist @ Gonen Lab ____________________________________________________ Janelia Research Campus * 19700 Helix Drive Ashburn, VA 20147 * +1 (571) 209-4000 extension 3376 _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org http://phenix-online.org/mailman/listinfo/cctbxbb

3 3

CCTBX.XFEL problems with synchrotron stills
by Schulz, Eike-Christian 06 Apr '16

06 Apr '16

Dear all, I am new to serial crystallography and trying to process still images from a synchrotron source (Pilatus 6M detector) using CCTBX. Obviously the process is not glitch-free and I was hoping to get some help here. The indexing routine starts, indexes the images correctly and and produces a log file but no .pickle files are created in the results/000/ directory as described in the wiki entry. I assume that there is a software module that is not properly loaded during the process since I get following error message during indexing: ---------BEGIN Integrate one frame 75 warsaw0048_00075.pickle EFFECTIVE TILING 0 0 2282 2282 Indexing error: No module named cxi_user Is there a way to find out if all dependencies are matched ? I suppose CCTBX.XFEL depends on some (specific) python libraries … ? Or am I completely wrong and the error is something different ? With best regards Eike

3 4

sequential revision number from git repo
by R.D. Oeffner 05 Apr '16

05 Apr '16

Hi, We are slowly moving phaser svn to git. One issue that we would like to address is if there is a practical way of conveying a linear commit history from the git repo. One approach is to just count the number of revisions that was made to the master branch in the git repo with "git rev-list --count HEAD" and then pretend that number is a revision number. This enables us to tell users "This bug was fixed after revision 6456 of phaser" rather than "This bug was fixed after commit 29b0a1c... of phaser". Is there any pitfalls with this approach? What workarounds do other people using git come up with? Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234 mobile: +44(0)7712 887162

3 6

changes to cctbx we have in ccp4
by Marcin Wojdyr 04 Apr '16

04 Apr '16

Hi All, we have a bunch of changes to cctbx in ccp4. If any of them are of general interest I can push them to svn. They'd need some cleanup before applying, so I'm not attaching any patches for now. - option to use external Boost. It's the same what debian and gentoo distros are doing when packaging cctbx. We change a few lines in the scons scripts and rm cctbx_project/boost/* (but the boost directory itself must be present). We could add an option for it (--use-system-boost ?). - removing -funroll-loops - this option must be a historical leftover. Generally it's better to let compiler decide because unrolling loops may make the program slower as well as faster. I tested it on Phaser a couple years ago. The only noticeable effect of this flag was much bigger binary. - extend --use-environment-flags to handle also CC and CXX. With most of makefiles / build scripts, if the user has multiple compilers installed he selects one with CC=gcc-mp-5 etc. Then if the build script really needs to differentiate between compilers it tries to determine what kind of compiler it is. In cctbx it's the other way around - the --compiler flag tells what kind of compiler it is and the build script guesses executable names. For ccp4 I added in two places in libtbx/SConscript: + if libtbx.env.build_options.use_environment_flags: + if 'CC' in os.environ: cc = os.environ['CC'] + if 'CXX' in os.environ: cxx = os.environ['CXX'] so for _some_ platform/compiler combinations it's possible to use compilers with non-default names. Useful for systems with multiple GCC versions. - probe and reduce are called in mmtbx through phenix wrappers (phenix.probe and phenix.reduce). We prefer to call them directly, but I guess it's not to be changed in cctbx. But after changing the name I also had to remove error checking in mmtbx/validation/clashscore.py that checks return code from phenix.reduce. Reduce itself quite often returns non-zero, it's just that the phenix.reduce wrapper ignores it and always returns 0 (probably not intentionally). - cctbx_project/spotfinder/SConscript - on Windows we have to link dxtbxspotfinder with spotfinder. That's the only such problem when compiling cctbx. OTOH it apparently works for everyone else. I don't know why. It must have something to do with us using MinGW. - A couple of #ifdefs for compiling with MinGW (annlib and gltbx) Marcin

2 2

Berkeley Server Downtime (3/30/2016)
by Billy Poon 30 Mar '16

30 Mar '16

Dear all, On Wednesday, March 30, our servers will be shutdown starting at 7:00 am Pacific for scheduled maintenance in our server room. This will affect, - The Phenix web site (http://phenix-online.org/) - The DIALS West web site (http://dials.lbl.gov/) - The CCI web site (http://cci.lbl.gov/) - The Structural Biology Technology Portal (http://technology.sbkb.org/) We expect that the servers will be available by 5 pm Pacific at the latest. Bug reports and emails requesting help should be queued and we will receive them once the servers are back online. We apologize for any inconvenience. -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

1 1

new parallel_map implementation
by Gabor Bunkoczi 29 Mar '16

29 Mar '16

Hi, I have switched over libtbx.easy_mp.parallel_map to use new code that is not susceptible to deadlock when large amounts of data are returned from a worker process. This also means that parameters added to parallel_map to prevent exactly this are no longer necessary (but will be respected), although unlikely to influence performance so much to warrant changing existing code. I have run all tests, but unsure how well this gets tested by regression test, so please let me know if you come across something that is amiss. BW, Gabor

1 0

Troublesome Hall Symbols in CIF
by Luc Bourhis 21 Mar '16

21 Mar '16

Hi, Consider the following two Hall symbols: (1) -C 2y (x,y,-x+z) (2) -I 2y They represent the same setting of the same spacegroup, so in a perfect world the fact that the sgtbx comes with (1) from some list of operators should not matter. Unfortunately, (1) is rejected by CheckCIF right now. Reminder: every single small molecule publication in Acta Cryst. must pass CheckCIF. On the contrary, CheckCIF accepts (2). Yes, I know, this is a bug in CheckCIF. But the IUCR is very slow at fixing bug in that kerberos of a program and we do therefore put our Olex2 users in trouble when they use the smtbx to refine their structure, as opposed to ShelXL, which does output (2). Besides, (2) is clearly more pleasing to the human mind than (1). Thus I reckon there is definitively a case to make a change in the cctbx here. The question is at which level? I don’t think we want to touch the sgtbx. I am of the opinion of patching iotbx.cif. Would that cause any trouble in Phenix and consort? Any remark or suggestion? Best wishes, Luc

3 2

New top-level module: fast_linalg
by Luc Bourhis 07 Mar '16

07 Mar '16

Hi, this is an advanced warning for the introduction of a new top-level module named fast_linalg. In a couple of days, I will push it in one go as 25 commits but don’t be scared, nothing is changed outside the top directory fast_linalg, which also means it is not configured by default. Thus there cannot be any disruption but I thought it would be polite to warn all cctbx developers. The module features the automatic installation and integration of OpenBLAS which I have used to speed up smtbx refinement. I will commit that work later. Best wishes, Luc J. Bourhis

2 2

Dead link for old builds on cctbx_build page
by Luc Bourhis 06 Mar '16

06 Mar '16

Hi, on http://cci.lbl.gov/cctbx_build/, in the phrase “ Old code bundles prior to 2015 may be accessed here.”, the word “here" is a link toward http://cci.lbl.gov/cctbx_legacy_build but this is a dead link. Is this a typo or are those builds truly gone? Best wishes, Luc

2 1

libtbx.easy_mp semantics
by markus.gerstel＠diamond.ac.uk 23 Feb '16

23 Feb '16

Hello everyone, Could someone please explain the libtbx.easy_mp semantics to me? Currently, if you run easy_mp with multiple processes, objects are 'reset' back to their original state, ie. the state of the object in the main thread is retained, changes in the processing threads are dropped. However if you run easy_mp with the number of processes set to 1, object states are modified, because the main thread is the executing thread. How it this supposed to work? For a more detailed test please run $ libtbx.python libtbx/tst_easy_mp_state.py The source code explains the test in more detail. This behaviour currently causes dials.integrate to fail on bzipped input files. Given that this code is used in many places and this particular feature is not obvious, it may explain other anecdotal emergent behaviour, too. -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 3

FW: Problems with cctbx - Explore symmetry
by robert.mcmeeking＠stfc.ac.uk 19 Feb '16

19 Feb '16

Dear Colleagues I may not have sent my previous message to the most appropriate email address. I now forward including the "phenix-online.org" one. Is there anyone out there who can point me in the right direction? Best Regards Bob From: McMeeking, Robert (STFC,DL,SC) Sent: 13 February 2016 12:18 To: cctbx(a)cci.lbl.gov Cc: McMeeking, Robert (STFC,DL,SC) Subject: Problems with cctbx - Explore symmetry Hello from the UK I am currently experiencing problems with the Explore symmetry web service - http://cci.lbl.gov/cctbx/explore_symmetry.html. When dealing with a particular Hall Symbol there is a cctbx runtime error (Malformed matrix symbol). The symbol in question is: -R 3 2"c It represents a setting for Space Group R -3 c :H (167) I am practically certain this worked fine in the past. I suspect the problem may be related to parsing the double quote symbol. If I replace with a single quote it manages to run OK, but the output information is a non-standard setting for Space Group 165. By way of background: I call the web service from my own CGI script and extract the symmetry operators that I need to display the crystal structure in question. The Hall symbol used is from the Crystallographic Open Database CIF (COD 2100992). I noticed the problem very recently, but because I cache information it is not clear when the problem first appeared. I only need to go through this process when explicit symmetry operators are not present in the CIF in question. I have not, as yet, installed the cctbx system on our server. Inoticed the problem when using our CrystalWorks system to attempt to display entries from the Crystallography365 Project http://cds.dl.ac.uk/cgi-bin/rfm/crystalworks_365?Feb17 Best Regards Bob Dr R.F. McMeeking Honorary Scientist STFC Chemical Database Service at Daresbury

4 8