- cctbxbb - phenix-online.org

incorrect url for downloading fasta files
by R.D. Oeffner 28 Jun '17

28 Jun '17

hi, It would appear that the URL in cctbx_projectiotbxpdbfetch.py for fetching fasta files has been changed by RCSB. So fetch.py therefore downloads garbage when trying to download a fasta file. The correct URL is not documented by RCSB but can be inferred from the RCSB web site when downloading a fasta file. I've updated the URL in fetch.py accordingly. Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234

2 1

please read before your next commit
by markus.gerstel＠diamond.ac.uk 27 Jun '17

27 Jun '17

Good morning, recently a couple of large files were erroneously committed to the cctbx repository and then deleted thereafter. Git always keeps a full history, so to avoid cruft aggregating over time we decided to cut those files out of the repository history. Unfortunately this means everyone will have to do a couple of steps: 1. Please ensure you have run libtbx.git_avoid_merges on all of your development machines 2. Run git pull --rebase the next time you update your repository instead of git pull - again, do this on all development machines. 3. If you keep any local branches you may need to rebase them onto master (git fetch; git checkout $branchname; git rebase origin/master) If you don't follow these instructions you may end up recommitting those large files, and we have to start again. Thanks -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

Housekeeping / roadmap for future
by Graeme.Winter＠diamond.ac.uk 21 Jun '17

21 Jun '17

Hello all, Appreciate a phenix release coming shortly & not suggesting anything any time soon However we were discussing earlier what the road map is for cctbx - making it ready for the future - and I was wondering what is already on the table? Is there a road map somewhere? I recall Billy mentioning Python3 recently, as an example... Thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

4 5

Re: [cctbxbb] bootstrap.py on Windows
by R. D. Oeffner 20 Jun '17

20 Jun '17

Instructions for building on windows are here: https://github.com/cctbx/cctbx_project/wiki Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234 -----Original Message----- From: R. D. Oeffner Sent: Tuesday, June 20, 2017 10:37 PM To: horst.puschmann(a)gmail.com ; cctbxbb(a)phenix-online.org Cc: oleg(a)olexsys.org Subject: Re: [cctbxbb] bootstrap.py on Windows Dear Horst, There's a writeup of instructions on how to do CCTBX builds from bootstrap.py on Windows. No authentication is needed for building CCTBX on any platform. Please follow these instructions and get in touch if you have further problems. CCTBX and other projects depending on CCTBX is tested through regression test nightly on LBL machines on all platforms supported. See http://cci-vm-6.lbl.gov:8010/one_line_per_build for log files of these tests. Regards, Robert -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234 -----Original Message----- From: markus.gerstel(a)diamond.ac.uk Sent: Tuesday, June 20, 2017 9:25 PM To: horst.puschmann(a)gmail.com ; cctbxbb(a)phenix-online.org Cc: oleg(a)olexsys.org Subject: Re: [cctbxbb] bootstrap.py on Windows Hello Horst, I think the automated tests aren't really covering the bootstrap process. While we extensively test bootstrap at our place, we do not run any tests on Windows. I don't know if LBL runs Windows tests, but if they do they probably run it within buildbot, which means it is not run the same way you run it. Best wishes -Markus ________________________________ From: cctbxbb-bounces(a)phenix-online.org [cctbxbb-bounces(a)phenix-online.org] on behalf of Horst Puschmann [horst.puschmann(a)gmail.com] Sent: Tuesday, April 04, 2017 09:59 To: cctbxbb(a)phenix-online.org Cc: Oleg Dolomanov (oleg(a)olexsys.org) Subject: [cctbxbb] bootstrap.py on Windows Hello I am trying to install the cctbx on a Windows 7 64 bit machine using the following bootstrap.py file: https://raw.githubusercontent.com/cctbx/cctbx_project/master/libtbx/auto_bu… The script will fail, unless I comment out lines 1101 and 1102: #if self.isPlatformWindows(): #tarurl, arxname, dirpath = MODULES.get_module(module)().get_tarauthenticated(auth=self.get_auth()) If it tries to execute the commented lines, the error will be "KeyError: 'cciuser' in line 590" After commenting this out, it will fail in line 271, with "AttributeError: 'module' object has no attribute '_create_unverified_context'" I can get round that with disabling line 247: if sys.platform == "win32": (i.e. change this to if sys.platform == "xxx":) After that, things start downloading. All appears well until the same thing happens again in another bootstrap.py file in \modules\cctbx_project\libtbx\auto_build\bootstrap.py If I disable *that*, it goes further, but fails finally with ===== Running in build: run configure.py Traceback (most recent call last): File "bootstrap.py", line 2113, in <module> run() File "bootstrap.py", line 2108, in run enable_shared=options.enable_shared, File "bootstrap.py", line 1036, in run i.run() File "bootstrap.py", line 189, in run raise RuntimeError("Could not run %s: File not found" % executable) RuntimeError: Could not run base\bin\python: File not found I guess the automatic tests haven't picked this up, because no authentication is needed, maybe? Greetings Horst -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org http://phenix-online.org/mailman/listinfo/cctbxbb --- This email has been checked for viruses by AVG. http://www.avg.com

1 0

bootstrap.py on Windows
by Horst Puschmann 20 Jun '17

20 Jun '17

Hello I am trying to install the cctbx on a Windows 7 64 bit machine using the following bootstrap.py file: https://raw.githubusercontent.com/cctbx/cctbx_project/ master/libtbx/auto_build/bootstrap.py The script will fail, unless I comment out lines 1101 and 1102: #if self.isPlatformWindows(): #tarurl, arxname, dirpath = MODULES.get_module(module)().get_tarauthenticated(auth=self.get_auth()) If it tries to execute the commented lines, the error will be "KeyError: 'cciuser' in line 590" After commenting this out, it will fail in line 271, with "AttributeError: 'module' object has no attribute '_create_unverified_context'" I can get round that with disabling line 247: if sys.platform == "win32": (i.e. change this to if sys.platform == "xxx":) After that, things start downloading. All appears well until the same thing happens again in another bootstrap.py file in \modules\cctbx_project\libtbx\auto_build\bootstrap.py If I disable *that*, it goes further, but fails finally with ===== Running in build: run configure.py Traceback (most recent call last): File "bootstrap.py", line 2113, in <module> run() File "bootstrap.py", line 2108, in run enable_shared=options.enable_shared, File "bootstrap.py", line 1036, in run i.run() File "bootstrap.py", line 189, in run raise RuntimeError("Could not run %s: File not found" % executable) RuntimeError: Could not run base\bin\python: File not found I guess the automatic tests haven't picked this up, because no authentication is needed, maybe? Greetings Horst

3 2

bioinformatics.SE
by Marcin Wojdyr 16 Jun '17

16 Jun '17

Hello, just to let you know, a few weeks ago StackExchange started a Bioinformatics Q&A site. Like Biostars, it's mostly about sequences, but sometimes people pop up with questions that could be answered using cctbx, for example a few minutes ago: https://bioinformatics.stackexchange.com/questions/783/how-can-we-find-the-…

1 0

Fwd: [git/cctbx] master: Making cif behavior exactly the same as pdb (throwing away SCALE records if crystal_symmetry is supplied to obtain xray_structure). With test. (bdf35cf1c)
by Graeme.Winter＠diamond.ac.uk 16 Jun '17

16 Jun '17

Please could people refrain from making cctbx tests dependent on phenix tools? E.g. not easy_run.call("phenix.cif_as_pdb %s.cif > junk.out"%prefix) Obliged in anticipation Cheers Graeme Begin forwarded message: From: CCTBX Commit via DLS Jenkins <graeme.winter(a)GMAIL.COM<mailto:[email protected]>> Subject: [git/cctbx] master: Making cif behavior exactly the same as pdb (throwing away SCALE records if crystal_symmetry is supplied to obtain xray_structure). With test. (bdf35cf1c) Date: 15 June 2017 at 19:43:57 BST To: <CCTBX-COMMIT(a)JISCMAIL.AC.UK<mailto:[email protected]>> Reply-To: <cctbxbb(a)PHENIX-ONLINE.ORG<mailto:[email protected]>> Repository : ssh://g18-sc-serv-04.diamond.ac.uk/cctbx On branch : master ________________________________ commit bdf35cf1c69d235d2578d915836aad2a739c1c15 Author: Oleg Sobolev <osobolev(a)lbl.gov<mailto:[email protected]>> Date: Thu Jun 15 11:43:52 2017 -0700 Making cif behavior exactly the same as pdb (throwing away SCALE records if crystal_symmetry is supplied to obtain xray_structure). With test. ________________________________ bdf35cf1c69d235d2578d915836aad2a739c1c15 iotbx/pdb/mmcif.py | 24 +++++----- iotbx/regression/tst_xray_scale.py | 93 ++++++++++++++++++++++++++++++++++++++ iotbx/run_tests.py | 1 + 3 files changed, 107 insertions(+), 11 deletions(-) diff --git a/iotbx/pdb/mmcif.py b/iotbx/pdb/mmcif.py index 6a5221a6a..2b79df99a 100644 --- a/iotbx/pdb/mmcif.py +++ b/iotbx/pdb/mmcif.py @@ -508,17 +508,19 @@ class cif_input(iotbx.pdb.pdb_input_mixin): return extract_tls_from_cif_block(self.cif_block, hierarchy) def scale_matrix(self): - fractionalization_matrix = [ - self.cif_block.get('_atom_sites.fract_transf_matrix[%s][%s]' %(i, j)) - for i,j in ('11', '12', '13', '21', '22', '23', '31','32', '33')] - if fractionalization_matrix.count(None) == 9: - return None - assert fractionalization_matrix.count(None) == 0 - fractionalization_vector = [ - self.cif_block.get('_atom_sites.fract_transf_vector[%s]' %i) for i in '123'] - assert fractionalization_matrix.count(None) == 0 - return [[float(i) for i in fractionalization_matrix], - [float(i) for i in fractionalization_vector]] + if (not hasattr(self, "_scale_matrix")): + fractionalization_matrix = [ + self.cif_block.get('_atom_sites.fract_transf_matrix[%s][%s]' %(i, j)) + for i,j in ('11', '12', '13', '21', '22', '23', '31','32', '33')] + if fractionalization_matrix.count(None) == 9: + return None + assert fractionalization_matrix.count(None) == 0 + fractionalization_vector = [ + self.cif_block.get('_atom_sites.fract_transf_vector[%s]' %i) for i in '123'] + assert fractionalization_matrix.count(None) == 0 + self._scale_matrix = [[float(i) for i in fractionalization_matrix], + [float(i) for i in fractionalization_vector]] + return self._scale_matrix def model_ids(self): return flex.std_string([model.id for model in self.hierarchy.models()]) diff --git a/iotbx/regression/tst_xray_scale.py b/iotbx/regression/tst_xray_scale.py new file mode 100644 index 000000000..e1a4ec9f3 --- /dev/null +++ b/iotbx/regression/tst_xray_scale.py @@ -0,0 +1,93 @@ +from __future__ import division +from libtbx import easy_run +import iotbx.pdb +from cctbx import crystal +from libtbx.test_utils import approx_equal + +cif_str=""" +data_5JUP +# +_atom_sites.entry_id 5JUP +_atom_sites.fract_transf_matrix[1][1] 1.000000 +_atom_sites.fract_transf_matrix[1][2] 0.000000 +_atom_sites.fract_transf_matrix[1][3] 0.000000 +_atom_sites.fract_transf_matrix[2][1] 0.000000 +_atom_sites.fract_transf_matrix[2][2] 1.000000 +_atom_sites.fract_transf_matrix[2][3] 0.000000 +_atom_sites.fract_transf_matrix[3][1] 0.000000 +_atom_sites.fract_transf_matrix[3][2] 0.000000 +_atom_sites.fract_transf_matrix[3][3] 1.000000 +_atom_sites.fract_transf_vector[1] 0.00000 +_atom_sites.fract_transf_vector[2] 0.00000 +_atom_sites.fract_transf_vector[3] 0.00000 +# +loop_ +_atom_type.symbol +C +MG +N +O +P +S +# +loop_ +_atom_site.group_PDB +_atom_site.id +_atom_site.type_symbol +_atom_site.label_atom_id +_atom_site.label_alt_id +_atom_site.label_comp_id +_atom_site.label_asym_id +_atom_site.label_entity_id +_atom_site.label_seq_id +_atom_site.pdbx_PDB_ins_code +_atom_site.Cartn_x +_atom_site.Cartn_y +_atom_site.Cartn_z +_atom_site.occupancy +_atom_site.B_iso_or_equiv +_atom_site.auth_seq_id +_atom_site.auth_comp_id +_atom_site.auth_asym_id +_atom_site.auth_atom_id +_atom_site.pdbx_PDB_model_num +ATOM 9 C "C3'" . U A 1 1 ? 175.229 258.091 229.127 1.00 113.67 1 U A "C3'" 1 +ATOM 7 C "C4'" . U A 1 1 ? 175.388 257.913 227.616 1.00 113.21 1 U A "C4'" 1 +ATOM 6 C "C5'" . U A 1 1 ? 174.124 257.916 226.769 1.00 114.11 1 U A "C5'" 1 +ATOM 10 O "O3'" . U A 1 1 ? 176.253 257.344 229.761 1.00 112.89 1 U A "O3'" 1 +ATOM 8 O "O4'" . U A 1 1 ? 176.291 258.974 227.185 1.00 112.70 1 U A "O4'" 1 +ATOM 5 O "O5'" . U A 1 1 ? 173.735 259.270 226.430 1.00 115.09 1 U A "O5'" 1 +ATOM 3 O OP1 . U A 1 1 ? 174.056 259.302 223.918 1.00 117.16 1 U A OP1 1 +ATOM 4 O OP2 . U A 1 1 ? 172.661 261.090 225.058 1.00 104.87 1 U A OP2 1 +ATOM 1 O OP3 . U A 1 1 ? 171.796 258.709 224.899 1.00 116.23 1 U A OP3 1 +ATOM 2 P P . U A 1 1 ? 173.022 259.610 224.994 1.00 116.34 1 U A P 1 +# +""" + +def run(prefix="iotbx_tst_xray_scale"): + """ + Exercise obtaining exactly the same xray structure from pdb and cif. + Caveate is SCALE records (fract_transf_matrix). In pdb they explicitly + suppressed when crystal_symmetry is supplied, see + iotbx/pdb/__init__.py:939 (def xray_structure_simple): + if(crystal_symmetry is not None): self._scale_matrix = None + """ + fo = open("%s.cif" % prefix,"w") + print >> fo, cif_str + fo.close() + # crystal symmetry from map + cs = crystal.symmetry((419., 419., 419., 90.0, 90.0, 90.0), 1) + # xrs from mmCIF + cif_inp = iotbx.pdb.input(file_name="%s.cif" % prefix) + # print "======== doing xrs from cif ==============" + xrs_cif = cif_inp.xray_structure_simple(crystal_symmetry = cs) + # xrs from PDB + easy_run.call("phenix.cif_as_pdb %s.cif > junk.out"%prefix) + pdb_inp = iotbx.pdb.input(file_name="%s.pdb" % prefix) + # print "======== doing xrs from pdb ==============" + xrs_pdb = pdb_inp.xray_structure_simple(crystal_symmetry = cs) + # + assert approx_equal(xrs_cif.sites_cart()[0], xrs_pdb.sites_cart()[0]) + +if (__name__ == "__main__"): + run() diff --git a/iotbx/run_tests.py b/iotbx/run_tests.py index 52eb3d2e4..9f3e50a98 100644 --- a/iotbx/run_tests.py +++ b/iotbx/run_tests.py @@ -93,6 +93,7 @@ tst_list = ( "$D/regression/tst_show_systematic_absences.py", "$D/regression/tst_miller_sort_asu.py", "$D/regression/tst_reflection_file_reader.py", + "$D/regression/tst_xray_scale.py", ) def run(): -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 1

Backwards compatibility; Python 2.3?
by Nicholas Devenish 12 Jun '17

12 Jun '17

Hi All, I'm starting to go through and learn some of the structure of these packages, starting with libtbx. What's the supported targets supposed to be? There seems to be code specifically marked as backwards compatibility for Python 2.3 (2003), and packages explicitly included that were merged into the standard library around the same time (e.g. Optik). Is this really as far back as we need to support? Thanks, Nick Devenish

7 12

bz2 support on Ubuntu
by David Waterman 08 Jun '17

08 Jun '17

Hi folks, My bootstrap build of cctbx/DIALS on Ubuntu 16.04.2 LTS produces a base Python that does not have bz2 support. This is despite the fact that the package "libbz2-dev is already the newest version (1.0.6-8)." Looking in Python_install_log I see Python build finished, but the necessary bits to build these modules were not found: _bsddb _curses _curses_panel _sqlite3 _tkinter bsddb185 bz2 dbm dl gdbm imageop readline sunaudiodev Has anyone got any ideas? Anyone done a boostrap build on Ubuntu where bz2 can be imported in Python? Also, what's the procedure to rebuild from base, recovering all the currently configured modules? My guess is that something like this would do it, but is there a better way? export MYMODULES=$(libtbx.list_modules | tr '\n' ' ') rm -rf base_tmp/ base/ build/ python bootstrap.py --builder=dials base python bootstrap.py --builder=dials build cd build source setpaths.sh libtbx.configure $MYMODULES make Cheers -- David

2 3

simtbx
by Graeme.Winter＠diamond.ac.uk 05 Jun '17

05 Jun '17

Hi Folks We have noticed that simtbx, part of the core cctbx now, depends on dials This means if you do not have a dials build your cctbx will be broken This is probably not the way we want things? I thought simtbx was a separate repo? Cheerio Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

7 17

cctbx has acquired a hidden dependency on iotbx
by Luc Bourhis 27 May '17

27 May '17

Hi, C:\Users\luc\Documents\Developer\cctbx\modules\cctbx_project\cctbx\maptbx\SConscript, line 5-8: env = env_base.Clone( SHLINKFLAGS=env_etc.shlinkflags, LIBS=["iotbx_xplor", "cctbx_sgtbx_asu", "cctbx", "omptbx", env_etc.libm] ) This is not supposed to be: the cctbx shall be usable with only what’s defined in its libtbx_config. Best wishes, Luc

3 2

shall iotbx really depends on dxtbx?
by Luc Bourhis 26 May '17

26 May '17

Hi, iotbx/libtbx_config: { "modules_required_for_use": ["cctbx", "fable", "ucif", "smtbx", "dxtbx"], "optional_modules": ["ccp4io", "cbflib"], } A search for dxtbx in the source code of iotbx does not return anything. Did I miss something? Moreover dxtbx/libtbx_config: { "modules_required_for_use" : [ "iotbx" ], } The problem is that dxtbx pulls in HDF5 and that it fails to compiles on Windows 64 with a 64-bit Python, a configuration we need for Olex 2. If the solution was as simple as dropping the dependence of iotbx on dxtbx, that would make my day! Thoughts? Best wishes, Luc Bourhis

3 4

probe/reduce repositories
by Hattne, Johan 04 May '17

04 May '17

Dear all; I appear to be missing probe in my source build. On running phenix.model_vs_data I get the traceback posted below. The error message appears to be a tad misleading, because having probe in PATH is not enough: the actual test in the code is if (not libtbx.env.has_module(name="probe")): which leads me to believe that I’m supposed to have something probe-like in my source tree, but the duke-server I used to get these kinds of things from (quidditch?) seems unresponsive from here. This is the first time in weeks that I’m doing this, so it is very much likely that whatever happened to break my setup happened quite some time ago. Cluebat, anyone? // Cheers; Johan Traceback (most recent call last): File "/mnt/gonen/home/hattnej/projects/phenix-build/../phenix-src/phenix/phenix/command_line/model_vs_data.py", line 6, in <module> show_geometry_statistics = True) File "/mnt/gonen/home/hattnej/projects/phenix-src/cctbx_project/mmtbx/model_vs_data.py", line 710, in run atom_selections = atom_selections) File "/mnt/gonen/home/hattnej/projects/phenix-src/cctbx_project/mmtbx/model_vs_data.py", line 346, in show_geometry restraints_manager = restraints_manager) File "/mnt/gonen/home/hattnej/projects/phenix-src/cctbx_project/mmtbx/model_statistics.py", line 138, in __init__ molprobity_scores=molprobity_scores) File "/mnt/gonen/home/hattnej/projects/phenix-src/cctbx_project/mmtbx/model_statistics.py", line 51, in __init__ self.clashscore = clashscore(pdb_hierarchy=pdb_hierarchy).get_clashscore() File "/mnt/gonen/home/hattnej/projects/phenix-src/cctbx_project/mmtbx/validation/clashscore.py", line 90, in __init__ "http://kinemage.biochem.duke.edu/") RuntimeError: Probe could not be detected on your system. Please make sure Probe is in your path. Probe is available at http://kinemage.biochem.duke.edu/ Research Specialist @ Gonen Lab ____________________________________________________ Janelia Research Campus * 19700 Helix Drive Ashburn, VA 20147 * +1 (571) 209-4000 extension 3376

2 5

normalize() is not safe
by Oleg Sobolev 02 May '17

02 May '17

Dear colleagues, I recently discovered that normalize() function of scitbx::vec3<double> is not safe, i.e. this case scitbx::vec3<double> a(0,0,0); a.normalize(); it will produce division by zero and fail. If function contains this being called from python the traceback would be rather vague: <...> Floating-point error (Python and libc call stacks above) This crash may be due to a problem in any imported Python module, including modules which are not part of the cctbx project. To disable the traps leading to this message, define these environment variables (e.g. assign the value 1): BOOST_ADAPTBX_FPE_DEFAULT BOOST_ADAPTBX_SIGNALS_DEFAULT This will NOT solve the problem, just mask it, but may allow you to proceed in case it is not critical. The code that generates this error sits in scitbx/vec3.h, lines 143-153 with a comment that this was done intentionally and a pointer to an appropriate function to do safe normalization. This function is not wrapped to be available in python directly. Similar functionality (but coded in a different place) accessible via python by: >>> from scitbx import matrix >>> b = matrix.col([0,0,0]) >>> b.normalize() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/net/anaconda/raid1/olegs/phenix_queue/modules/cctbx_project/scitbx/matrix/__init__.py", line 270, in normalize return self / abs(self) File "/net/anaconda/raid1/olegs/phenix_queue/modules/cctbx_project/scitbx/matrix/__init__.py", line 158, in __truediv__ return rec([e/other for e in self.elems], self.n) ZeroDivisionError: float division by zero Fast grepping shows that normalize() is widely used in xfel, smtbx, rstbx, dxtbx, mmtbx/hydrogens, scitbx/rigid_body, and in some other places. My questions to CCTBX community are: 1. Were you aware of this issue? 2. Should we do something about it: - make the function more safe and slow; - leave it like it is and let everybody know about the issue and let developers deal with it individually; 3. Any other input on the matter? Best regards, Oleg Sobolev.

5 5

Boost Python 1.56
by Luc Bourhis 14 Apr '17

14 Apr '17

Hi, I have improvements to the smtbx on their way to be committed which require Boost version 1.56. This is related to Boost.Threads, whose support I re-activated a few months ago on Nick’s request. I need the function boost::thread::physical_concurrency which returns the number of physical cores on the machine, as opposed to virtual cores when hyperthreading is enabled (which it is by default on any Intel machine). That function is not available in Boost 1.55 which is the version currently used in the nightly tests: it appeared in 1.56. So, would it be possible to move to Boost 1.56? Otherwise, I will need to backport that function. Not too difficult but not thrilling. Best wishes, Luc

14 36

Re: [cctbxbb] [git/cctbx] master: Replacing illegal C++ code with proper dynamic array allocation. (a31eb2bf0)
by Nicholas Sauter 12 Apr '17

12 Apr '17

Rob, Just as a general thing, "new" statements in C++ need to be paired with "delete" statements (delete pk, delete a_sum, and delete akml). Unless this is done there will be a memory leak. Nick Nicholas K. Sauter, Ph. D. Senior Scientist, Molecular Biophysics & Integrated Bioimaging Division Lawrence Berkeley National Laboratory 1 Cyclotron Rd., Bldg. 33R0345 Berkeley, CA 94720 (510) 486-5713 On Tue, Apr 11, 2017 at 3:31 AM, CCTBX Commit via DLS Jenkins < graeme.winter(a)gmail.com> wrote: > Repository : ssh://g18-sc-serv-04.diamond.ac.uk/cctbx > On branch : master > > ------------------------------ > > > commit a31eb2bf0d5bb129425bb759466f45622c773fd4 > Author: Robert Oeffner <rdo20(a)cam.ac.uk> > Date: Tue Apr 11 11:31:24 2017 +0100 > > Replacing illegal C++ code with proper dynamic array allocation. > > > ------------------------------ > > > a31eb2bf0d5bb129425bb759466f45622c773fd4 > scitbx/lbfgs.h | 6 +++--- > 1 file changed, 3 insertions(+), 3 deletions(-) > > diff --git a/scitbx/lbfgs.h b/scitbx/lbfgs.h > index 41be8e8cc..f0000143d 100755 > --- a/scitbx/lbfgs.h > +++ b/scitbx/lbfgs.h > @@ -917,9 +917,9 @@ namespace lbfgs { > int info = 5; > brackt = false; > FloatType sxnorm = std::sqrt(ddot(SizeType(n), sx, sx)); > - FloatType pk[n]; > - FloatType a_sum[n]; > - FloatType akm1[n]; > + FloatType* pk = new FloatType[n]; > + FloatType* a_sum = new FloatType[n]; > + FloatType* akm1 = new FloatType[n]; > for (SizeType i=0; i < n; i++) { > pk[i] = sx[i]/sxnorm; > akm1[i] = -gx[i]/sxnorm; >

3 2

libtiff dependency on macOS
by Billy Poon 11 Apr '17

11 Apr '17

Hi everyone, I just updated install_base_packages.py to build libtiff generally instead of just for Linux. So if you update your sources and get build errors relating to TIFF, please rebuild your base directory. Thanks! -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

1 0

Re: [cctbxbb] bug fix in cctbx_project/scitbx/lbfgs.h
by R.D. Oeffner 11 Apr '17

11 Apr '17

There is now a fatal warning in libtbx/SConscript if gcc encounters variable length arrays as this is not portable code and we've had broken builds on more than one occasion due to this recently. Rob On 11/04/2017 12:01, Pavel Afonine wrote: > Hi Rob, > > thanks for the fix. This is was Min's code (remember her from the past workshop?). I simply applied her patch without looking into it too much, and made sure all t96 pass. > > Again, thanks for fixing it. > > Pavel > > On 4/11/17 03:48, R.D. Oeffner wrote: > >> Hi Pavel, I have made a change to that file as it didn't compile on Windows. In fact I can't understand how it would compile with other compilers before as the syntax to me seemed illegal given that n is not known at compile time. I'm not sure how a dynamically created FloatType should be destroyed as I couldn't find any examples of that being done throughout this file. Rob

2 1

bootstrap.py on windows
by Horst Puschmann 07 Apr '17

07 Apr '17

Hello I am trying to install the cctbx on a Windows 7 64 bit machine using the following bootstrap.py file: https://raw.githubusercontent.com/cctbx/cctbx_project/master /libtbx/auto_build/bootstrap.py The script will fail, unless I comment out lines 1101 and 1102: #if self.isPlatformWindows(): #tarurl, arxname, dirpath = MODULES.get_module(module)(). get_tarauthenticated(auth=self.get_auth()) If it tries to execute the commented lines, the error will be "KeyError: 'cciuser' in line 590" After commenting this out, it will fail in line 271, with "AttributeError: 'module' object has no attribute '_create_unverified_context'" I can get round that with disabling line 247: if sys.platform == "win32": (i.e. change this to if sys.platform == "xxx":) After that, things start downloading. All appears well until the same thing happens again in another bootstrap.py file in \modules\cctbx_project\ libtbx\auto_build\bootstrap.py If I disable *that*, it goes further but fails finally with ===== Running in build: run configure.py Traceback (most recent call last): File "bootstrap.py", line 2113, in <module> run() File "bootstrap.py", line 2108, in run enable_shared=options.enable_shared, File "bootstrap.py", line 1036, in run i.run() File "bootstrap.py", line 189, in run raise RuntimeError("Could not run %s: File not found" % executable) RuntimeError: Could not run base\bin\python: File not found I guess the automatic tests haven't picked this up, because no authentication is needed, maybe? Greetings Horst

6 16

Fw: cctbx developer guidelines
by R. D. Oeffner 06 Apr '17

06 Apr '17

Hi Pavel and others, Attached is the original word document you sent out if that's useful for anyone. Rob -----Original Message----- From: Pavel Afonine Sent: Saturday, July 11, 2015 7:12 PM To: Paul Adams ; developers(a)phenix-online.org Subject: Re: cctbx developer guidelines Hi Paul, looks great! Attached is slightly edited copy. Pavel On 7/10/15 11:14, Paul Adams wrote: > Hi Everyone, > > we have developed a cctbx developer guidance document (much of which > also applies to Phenix development). I’m attaching it for your review and > comments. If you have any suggestions let me know. The goal is to > distribute this to all developers who are contributing to the cctbx and to > put it somewhere prominent on the web. > > Cheers, > Paul >

1 0

bootstrap.py not overwriting modified sources
by R.D. Oeffner 01 Apr '17

01 Apr '17

Hi all, It appears that when bootstrap.py is invoked to update sources then unsurprisingly it doesn't overwrite existing sources if these are more recent, as in someone has modified them and not committed the changes. Occasionally this leads to broken builds. To prevent this from happening would it not be a good idea to change bootstrap.py to default to overwrite any uncommitted changes unless some flag is set on the command line? Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234 mobile: +44(0)7712 887162

4 5

Berkeley Server Downtime (3/23 - 3/24)
by Billy Poon 24 Mar '17

24 Mar '17

Dear all, On Thursday, March 23, our servers will be shut down starting at 5:00 pm Pacific for some emergency electrical work near our building. We expect that the servers will be available the next day (3/24) by 1:00 pm Pacific. There is a possibility that the outage will last longer. This will affect, - The Phenix web site (http://phenix-online.org/) - The DIALS West web site (http://dials.lbl.gov/) - The CCI web site (http://cci.lbl.gov/) - The Structural Biology Technology Portal (http://technology.sbkb.org/) Bug reports and emails requesting help should be queued and we will receive them once the servers are back online. We apologize for any inconvenience and the short notice. -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

1 1

Re: [cctbxbb] cctbxbb Digest, Vol 96, Issue 3
by Allan Lyckegaard 21 Mar '17

21 Mar '17

Hi Billy, Thanks, it works! It now compiles and when adding it to the path, “source build/setpaths.sh”, it can be imported in python. A few questions, though: - is it necessary to create a new conda environment to make it work? My first try where I got the error was in an existing environment. - can I update any packages in my new environment without breaking cctbx? I would i.e. be wonderful to have access to ipython Cheers, Allan ******************************************************** Allan Lyckegaard, PhD, Scientist Xnovo Technology ApS Galoche Alle 15 4600 Køge CVR: 3489 4221 Mail: alyckegaard(a)xnovotech.com <mailto:[email protected]> Mobile: +45 5373 8581 ******************************************************* The information in this email, including any attachments, is confidential and intended only for the recipient(s) listed. Any use of this email for any other purpose is prohibited. If you have received this email in error, please notify me immediately by reply email, delete this email, and do not disclose its contents to anyone. > On 21 Mar 2017, at 20.00, cctbxbb-request(a)phenix-online.org wrote: > > Send cctbxbb mailing list submissions to > cctbxbb(a)phenix-online.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://phenix-online.org/mailman/listinfo/cctbxbb > or, via email, send a message with subject or body 'help' to > cctbxbb-request(a)phenix-online.org > > You can reach the person managing the list at > cctbxbb-owner(a)phenix-online.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of cctbxbb digest..." > > > Today's Topics: > > 1. Re: Build error, bootstrap.py (Billy Poon) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 20 Mar 2017 12:38:31 -0700 > From: Billy Poon <bkpoon(a)lbl.gov> > To: cctbx mailing list <cctbxbb(a)phenix-online.org> > Subject: Re: [cctbxbb] Build error, bootstrap.py > Message-ID: > <CAHmYUcY2oJp=D9w_de=S++fn4D9rzanB+zFtqU8ME9OgWmSu-Q(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi Allan, > > Your command should have installed the hdf5 dependency among the other > dependencies. The hdf5 dependency is required because there are functions > for reading image data. Was there an error in compiling HDF5? There should > be a log file in <cctbx installation directory>/base_tmp named > HDF5_install_log. > > Are you using a recent version of bootstrap.py? I just tested this on macOS > 10.12.3 with conda 4.3.11 and the command-line tools for Xcode 8.2.1. The > steps I followed (in a clean directory with just bootstrap.py) are, > > 1) conda create -n test python=2 > 2) source activate test > 3) python bootstrap.py --with-python=`which python` --builder=cctbx > --nproc=8 > 4) cd build; make > > The `which python` command in step 3 just returns the path for the python > in the "test" environment. Step 4 is just to make sure everything is > compiled. > > Let us know if you it still does not work. Thanks! > > -- > Billy K. Poon > Research Scientist, Molecular Biophysics and Integrated Bioimaging > Lawrence Berkeley National Laboratory > 1 Cyclotron Road, M/S 33R0345 > Berkeley, CA 94720 > Tel: (510) 486-5709 > Fax: (510) 486-5909 > Web: https://phenix-online.org > > On Fri, Mar 17, 2017 at 7:27 AM, Allan Lyckegaard <alyckegaard(a)xnovotech.com >> wrote: > >> Hi all, >> >> I am trying to install cctbx on my Mac (Sierra) using bootstrap.py with my >> Anaconda python. >> >> When executing 'python bootstrap.py --with-python=/Users/myuser/ >> miniconda2/envs/python2/bin/python --builder=cctbx? I get the following >> error: >> >> /usr/bin/c++ -o dxtbx/format/boost_python/nexus_ext.o -c >> -DBOOST_HAS_INT128=1 -DBOOST_ALL_NO_LIB -DBOOST_PYTHON_MAX_BASES=2 -Wmost >> -Wno-unknown-pragmas -Wno-logical-op-parentheses -Wno-unused-local-typedefs >> -I/Users/myuser/Development/cctbx/modules/boost -fPIC >> -fno-strict-aliasing -Wno-c++0x-extensions -Wno-array-bounds -DNDEBUG -O3 >> -ffast-math -I/Users/myuser/miniconda2/envs/python2/include >> -I/Users/myuser/Development/cctbx/base/include >> -I/Users/myuser/Development/cctbx/build/include >> -I/Users/myuser/Development/cctbx/modules/cctbx_project >> -I/Users/myuser/miniconda2/envs/python2/include/python2.7 >> /Users/myuser/Development/cctbx/modules/cctbx_project/ >> dxtbx/format/boost_python/nexus_ext.cc >> warning: unknown warning option '-Wno-unused-local-typedefs' >> [-Wunknown-warning-option] >> 1 warning generated. >> /usr/bin/c++ -o lib/dxtbx_format_nexus_ext.so -w -bundle -undefined >> dynamic_lookup dxtbx/format/boost_python/nexus_ext.o >> -L/Users/myuser/Development/cctbx/base/lib -lm -lhdf5 >> ld: library not found for -lhdf5 >> clang: error: linker command failed with exit code 1 (use -v to see >> invocation) >> scons: *** [lib/dxtbx_format_nexus_ext.so] Error 1 >> scons: building terminated because of errors. >> usr+sys time: 8.69 seconds >> wall clock time: 21.70 seconds >> Process failed with return code 2 >> >> Any suggestions to how I solve this? I have not seen anywhere in the >> documentation that an hdf5 installation is a prerequisite. >> >> Cheers, >> /Allan >> >> >> _______________________________________________ >> cctbxbb mailing list >> cctbxbb(a)phenix-online.org >> http://phenix-online.org/mailman/listinfo/cctbxbb >> >

2 1

Build error, bootstrap.py
by Allan Lyckegaard 20 Mar '17

20 Mar '17

Hi all, I am trying to install cctbx on my Mac (Sierra) using bootstrap.py with my Anaconda python. When executing 'python bootstrap.py --with-python=/Users/myuser/miniconda2/envs/python2/bin/python --builder=cctbx’ I get the following error: /usr/bin/c++ -o dxtbx/format/boost_python/nexus_ext.o -c -DBOOST_HAS_INT128=1 -DBOOST_ALL_NO_LIB -DBOOST_PYTHON_MAX_BASES=2 -Wmost -Wno-unknown-pragmas -Wno-logical-op-parentheses -Wno-unused-local-typedefs -I/Users/myuser/Development/cctbx/modules/boost -fPIC -fno-strict-aliasing -Wno-c++0x-extensions -Wno-array-bounds -DNDEBUG -O3 -ffast-math -I/Users/myuser/miniconda2/envs/python2/include -I/Users/myuser/Development/cctbx/base/include -I/Users/myuser/Development/cctbx/build/include -I/Users/myuser/Development/cctbx/modules/cctbx_project -I/Users/myuser/miniconda2/envs/python2/include/python2.7 /Users/myuser/Development/cctbx/modules/cctbx_project/dxtbx/format/boost_python/nexus_ext.cc warning: unknown warning option '-Wno-unused-local-typedefs' [-Wunknown-warning-option] 1 warning generated. /usr/bin/c++ -o lib/dxtbx_format_nexus_ext.so -w -bundle -undefined dynamic_lookup dxtbx/format/boost_python/nexus_ext.o -L/Users/myuser/Development/cctbx/base/lib -lm -lhdf5 ld: library not found for -lhdf5 clang: error: linker command failed with exit code 1 (use -v to see invocation) scons: *** [lib/dxtbx_format_nexus_ext.so] Error 1 scons: building terminated because of errors. usr+sys time: 8.69 seconds wall clock time: 21.70 seconds Process failed with return code 2 Any suggestions to how I solve this? I have not seen anywhere in the documentation that an hdf5 installation is a prerequisite. Cheers, /Allan

2 1

hkl and F
by James Holton 04 Mar '17

04 Mar '17

Hello all, Is there any standard way to get the "C++ signature" of a python object? I'm trying to get my _ext.cpp working, and its' maddening trying to figure out how to declare objects that need to be exposed via boost. -James

2 1

Quick Boost python question
by Nicholas Sauter 21 Feb '17

21 Feb '17

<http://phenix-online.org/mailman/listinfo/cctbxbb>Quick question from James Holton and myself. We want to use a boost python *.add_property() to add a make_getter and make_setter to our wrapper, so we can expose a class variable to the Python layer. However, we want to execute a change of measurement units, i.e., the Python value will always be given in millimetres, and the C++ value always in metres. How can this be done within a simple add_property() paradigm? Nick

1 1

Berkeley Server Downtime (1/13 - 1/14)
by Billy Poon 15 Jan '17

15 Jan '17

Dear all, On Friday, January 13, our servers will be shut down starting at 5:00 pm Pacific for scheduled maintenance in our building. We expect that the servers will be available the next day (1/14) by 6:00 pm Pacific at the latest. This will affect, - The Phenix web site (http://phenix-online.org/) - The DIALS West web site (http://dials.lbl.gov/) - The CCI web site (http://cci.lbl.gov/) - The Structural Biology Technology Portal (http://technology.sbkb.org/) Bug reports and emails requesting help should be queued and we will receive them once the servers are back online. We apologize for any inconvenience. -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

1 3

Re: [cctbxbb] iotbx.merging_statistics test failing: data file removed from phenix_regression
by richard.gildea＠diamond.ac.uk 06 Jan '17

06 Jan '17

Hi Tom, Thanks for the quick response! To run this specific test: $ libtbx.python $(libtbx.find_in_repositories iotbx/regression/tst_merging_statistics.py) The test also uses the file phenix_regression/wizards/unmerged.mtz which I guess was also removed with the same commit? $ grep phenix_regression $(libtbx.find_in_repositories iotbx/regression/tst_merging_statistics.py) if (not libtbx.env.has_module("phenix_regression")) : print "phenix_regression not configured, skipping." relative_path="phenix_regression/wizards/p9_se_w2.sca", relative_path="phenix_regression/harvesting/unmerged.sca", relative_path="phenix_regression/harvesting/refine.mtz", relative_path="phenix_regression/reflection_files/i_anomalous.mtz", relative_path="phenix_regression/wizards/unmerged.mtz", relative_path="phenix_regression/reflection_files/AUTOMATIC_DEFAULT_scaled_unmerged_WAVE1.mtz", $ libtbx.find_in_repositories phenix_regression/wizards/unmerged.mtz Sorry: libtbx.find_in_repositories: cannot locate "phenix_regression/wizards/unmerged.mtz" The other files still seem to be present in phenix_regression. Cheers, Richard Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE ________________________________________ From: Terwilliger, Thomas Charles [terwilliger(a)lanl.gov] Sent: 06 January 2017 15:57 To: Gildea, Richard (DLSLtd,RAL,LSCI); cctbxbb(a)phenix-online.org Cc: Terwilliger, Thomas Charles Subject: Re: iotbx.merging_statistics test failing: data file removed from phenix_regression Hi Richard, Thanks for pointing that out! I'll fix it. Yes, I deleted that file when I moved all the small files in phenix_regression/wizards/ into sub-directories. I made the mistake of assuming that only wizards/ regression tests used files in the wizards/ directory (I should have run all the cctbx regression tests to make sure). All the best, Tom T ________________________________________ From: richard.gildea(a)diamond.ac.uk <richard.gildea(a)diamond.ac.uk> Sent: Friday, January 6, 2017 8:51 AM To: cctbxbb(a)phenix-online.org; Terwilliger, Thomas Charles Subject: iotbx.merging_statistics test failing: data file removed from phenix_regression Hi all, The test for iotbx.merging_statistics has recently started failing as a result of one of the files it uses for testing being deleted, phenix_regression/wizards/p9_se_w2.sca: https://github.com/cctbx/cctbx_project/blob/master/iotbx/regression/tst_mer… Would it be possible to restore this file to phenix_regression so that the test will pass? $ svn log -r r3879 -v ------------------------------------------------------------------------ r3879 | terwill | 2016-12-29 02:48:15 +0000 (Thu, 29 Dec 2016) | 1 line Changed paths: D /trunk/wizards/000README D /trunk/wizards/4c5q.pdb D /trunk/wizards/ATP.pdb D /trunk/wizards/MM-all.tgz D /trunk/wizards/NAD.pdb D /trunk/wizards/PARAMS_1.eff D /trunk/wizards/a2u-data.mtz D /trunk/wizards/a2u-sigmaa.mtz D /trunk/wizards/a2u.sca D /trunk/wizards/a2u_short.pdb D /trunk/wizards/anb.pdb D /trunk/wizards/autosol_unmerged_False.std D /trunk/wizards/autosol_unmerged_True.std D /trunk/wizards/autosol_unmerged_p9.std D /trunk/wizards/bad_model.pdb D /trunk/wizards/check.xyz D /trunk/wizards/check_multiple_input_files_042908.tgz D /trunk/wizards/check_multiple_input_files_062608.tgz D /trunk/wizards/check_multiple_input_files_070508.tgz D /trunk/wizards/check_multiple_input_files_2011-03-03.tgz D /trunk/wizards/check_multiple_input_files_2011-06-29.tgz D /trunk/wizards/check_multiple_input_files_2011-11-04.tgz D /trunk/wizards/clean.com D /trunk/wizards/cocn.cif D /trunk/wizards/coords.pdb D /trunk/wizards/coords1.ali D /trunk/wizards/coords1.mtz D /trunk/wizards/coords1.pdb D /trunk/wizards/coords_A001.pdb D /trunk/wizards/coords_rna.pdb D /trunk/wizards/coords_rna_gap.pdb D /trunk/wizards/data_for_commands_in_doc.tgz D /trunk/wizards/find_all_ligands.eff D /trunk/wizards/find_all_ligands.standard D /trunk/wizards/find_ncs_ncs.standard D /trunk/wizards/find_ncs_resolve.standard D /trunk/wizards/fobs.mtz D /trunk/wizards/fobs_std.mtz D /trunk/wizards/fobs_std_edit.mtz D /trunk/wizards/gene-5.pdb D /trunk/wizards/generate.com D /trunk/wizards/get_cc_mtz_mtz.standard D /trunk/wizards/get_cc_mtz_pdb.standard D /trunk/wizards/ha.pdb D /trunk/wizards/ha_lig.pdb D /trunk/wizards/ha_perfect.pdb D /trunk/wizards/ha_perfect_2.pdb D /trunk/wizards/hipip.pdb D /trunk/wizards/hipip_2.sca D /trunk/wizards/hipip_shortseq.dat D /trunk/wizards/ignore.dat D /trunk/wizards/inf.sca D /trunk/wizards/ki.sca D /trunk/wizards/ligand.pdb D /trunk/wizards/mad.eff D /trunk/wizards/mad.mtz D /trunk/wizards/map_coeffs.mtz D /trunk/wizards/map_coeffs_1.mtz D /trunk/wizards/map_coeffs_2.mtz D /trunk/wizards/map_to_object.standard D /trunk/wizards/maps_only.standard D /trunk/wizards/mir.eff D /trunk/wizards/mlt.mtz D /trunk/wizards/mlt_gap.pdb D /trunk/wizards/mlt_regression.tgz D /trunk/wizards/mol1.pdb D /trunk/wizards/mol2.pdb D /trunk/wizards/mr_rosetta.eff D /trunk/wizards/mult_regr.tgz D /trunk/wizards/multi.pdb D /trunk/wizards/multi.pdb.mtz D /trunk/wizards/my_autobuild.eff D /trunk/wizards/my_automr.eff D /trunk/wizards/my_autosol.eff D /trunk/wizards/my_ligand.pdb D /trunk/wizards/my_ligandfit.eff D /trunk/wizards/my_target.pdb D /trunk/wizards/native.sca D /trunk/wizards/noligand.mtz D /trunk/wizards/omit_lig_results.list D /trunk/wizards/omit_regr.tgz D /trunk/wizards/p.pdb D /trunk/wizards/p21.sym D /trunk/wizards/p9-seq.dat D /trunk/wizards/p9-sites.xyz D /trunk/wizards/p9_ha.pdb D /trunk/wizards/p9_se_w2.sca D /trunk/wizards/part_mlt.pdb D /trunk/wizards/partial.pdb D /trunk/wizards/partial_refine_001.pdb D /trunk/wizards/partial_refine_001_map_coeffs.mtz D /trunk/wizards/peak.sca D /trunk/wizards/perfect.mtz D /trunk/wizards/perfect_rna.mtz D /trunk/wizards/phaser_1_offset.mtz D /trunk/wizards/pt.sca D /trunk/wizards/quick_clashlist.standard D /trunk/wizards/random.pdb D /trunk/wizards/random_0.7_mlt.mtz D /trunk/wizards/resolve_1_offset.mtz D /trunk/wizards/rna.seq D /trunk/wizards/run_multi.eff D /trunk/wizards/run_multi_new.eff D /trunk/wizards/seq.dat D /trunk/wizards/seq1.dat D /trunk/wizards/seq2.dat D /trunk/wizards/seq_mlt.dat D /trunk/wizards/short_seq.standard D /trunk/wizards/side.pdb D /trunk/wizards/sir_sad.eff D /trunk/wizards/sites_mlt.pdb D /trunk/wizards/smiles.dat D /trunk/wizards/solve.setup D /trunk/wizards/start.pdb D /trunk/wizards/starting_map.mtz D /trunk/wizards/test3.gz D /trunk/wizards/test9.gz D /trunk/wizards/test_ampl_int_autobuild.tgz D /trunk/wizards/test_labels_1.std D /trunk/wizards/test_labels_2.std D /trunk/wizards/test_labels_3.std D /trunk/wizards/test_missing.std D /trunk/wizards/test_multi.tgz D /trunk/wizards/test_ncs.standard D /trunk/wizards/test_ncs_find_ncs.standard D /trunk/wizards/test_ncs_lc.pdb D /trunk/wizards/test_resno.std D /trunk/wizards/test_resno.tgz D /trunk/wizards/unmerged.mtz D /trunk/wizards/w1.sca D /trunk/wizards/w1_rna.sca D /trunk/wizards/w2.sca remove unneeded files ------------------------------------------------------------------------ Cheers, Richard Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

iotbx.merging_statistics test failing: data file removed from phenix_regression
by richard.gildea＠diamond.ac.uk 06 Jan '17

06 Jan '17

Hi all, The test for iotbx.merging_statistics has recently started failing as a result of one of the files it uses for testing being deleted, phenix_regression/wizards/p9_se_w2.sca: https://github.com/cctbx/cctbx_project/blob/master/iotbx/regression/tst_mer… Would it be possible to restore this file to phenix_regression so that the test will pass? $ svn log -r r3879 -v ------------------------------------------------------------------------ r3879 | terwill | 2016-12-29 02:48:15 +0000 (Thu, 29 Dec 2016) | 1 line Changed paths: D /trunk/wizards/000README D /trunk/wizards/4c5q.pdb D /trunk/wizards/ATP.pdb D /trunk/wizards/MM-all.tgz D /trunk/wizards/NAD.pdb D /trunk/wizards/PARAMS_1.eff D /trunk/wizards/a2u-data.mtz D /trunk/wizards/a2u-sigmaa.mtz D /trunk/wizards/a2u.sca D /trunk/wizards/a2u_short.pdb D /trunk/wizards/anb.pdb D /trunk/wizards/autosol_unmerged_False.std D /trunk/wizards/autosol_unmerged_True.std D /trunk/wizards/autosol_unmerged_p9.std D /trunk/wizards/bad_model.pdb D /trunk/wizards/check.xyz D /trunk/wizards/check_multiple_input_files_042908.tgz D /trunk/wizards/check_multiple_input_files_062608.tgz D /trunk/wizards/check_multiple_input_files_070508.tgz D /trunk/wizards/check_multiple_input_files_2011-03-03.tgz D /trunk/wizards/check_multiple_input_files_2011-06-29.tgz D /trunk/wizards/check_multiple_input_files_2011-11-04.tgz D /trunk/wizards/clean.com D /trunk/wizards/cocn.cif D /trunk/wizards/coords.pdb D /trunk/wizards/coords1.ali D /trunk/wizards/coords1.mtz D /trunk/wizards/coords1.pdb D /trunk/wizards/coords_A001.pdb D /trunk/wizards/coords_rna.pdb D /trunk/wizards/coords_rna_gap.pdb D /trunk/wizards/data_for_commands_in_doc.tgz D /trunk/wizards/find_all_ligands.eff D /trunk/wizards/find_all_ligands.standard D /trunk/wizards/find_ncs_ncs.standard D /trunk/wizards/find_ncs_resolve.standard D /trunk/wizards/fobs.mtz D /trunk/wizards/fobs_std.mtz D /trunk/wizards/fobs_std_edit.mtz D /trunk/wizards/gene-5.pdb D /trunk/wizards/generate.com D /trunk/wizards/get_cc_mtz_mtz.standard D /trunk/wizards/get_cc_mtz_pdb.standard D /trunk/wizards/ha.pdb D /trunk/wizards/ha_lig.pdb D /trunk/wizards/ha_perfect.pdb D /trunk/wizards/ha_perfect_2.pdb D /trunk/wizards/hipip.pdb D /trunk/wizards/hipip_2.sca D /trunk/wizards/hipip_shortseq.dat D /trunk/wizards/ignore.dat D /trunk/wizards/inf.sca D /trunk/wizards/ki.sca D /trunk/wizards/ligand.pdb D /trunk/wizards/mad.eff D /trunk/wizards/mad.mtz D /trunk/wizards/map_coeffs.mtz D /trunk/wizards/map_coeffs_1.mtz D /trunk/wizards/map_coeffs_2.mtz D /trunk/wizards/map_to_object.standard D /trunk/wizards/maps_only.standard D /trunk/wizards/mir.eff D /trunk/wizards/mlt.mtz D /trunk/wizards/mlt_gap.pdb D /trunk/wizards/mlt_regression.tgz D /trunk/wizards/mol1.pdb D /trunk/wizards/mol2.pdb D /trunk/wizards/mr_rosetta.eff D /trunk/wizards/mult_regr.tgz D /trunk/wizards/multi.pdb D /trunk/wizards/multi.pdb.mtz D /trunk/wizards/my_autobuild.eff D /trunk/wizards/my_automr.eff D /trunk/wizards/my_autosol.eff D /trunk/wizards/my_ligand.pdb D /trunk/wizards/my_ligandfit.eff D /trunk/wizards/my_target.pdb D /trunk/wizards/native.sca D /trunk/wizards/noligand.mtz D /trunk/wizards/omit_lig_results.list D /trunk/wizards/omit_regr.tgz D /trunk/wizards/p.pdb D /trunk/wizards/p21.sym D /trunk/wizards/p9-seq.dat D /trunk/wizards/p9-sites.xyz D /trunk/wizards/p9_ha.pdb D /trunk/wizards/p9_se_w2.sca D /trunk/wizards/part_mlt.pdb D /trunk/wizards/partial.pdb D /trunk/wizards/partial_refine_001.pdb D /trunk/wizards/partial_refine_001_map_coeffs.mtz D /trunk/wizards/peak.sca D /trunk/wizards/perfect.mtz D /trunk/wizards/perfect_rna.mtz D /trunk/wizards/phaser_1_offset.mtz D /trunk/wizards/pt.sca D /trunk/wizards/quick_clashlist.standard D /trunk/wizards/random.pdb D /trunk/wizards/random_0.7_mlt.mtz D /trunk/wizards/resolve_1_offset.mtz D /trunk/wizards/rna.seq D /trunk/wizards/run_multi.eff D /trunk/wizards/run_multi_new.eff D /trunk/wizards/seq.dat D /trunk/wizards/seq1.dat D /trunk/wizards/seq2.dat D /trunk/wizards/seq_mlt.dat D /trunk/wizards/short_seq.standard D /trunk/wizards/side.pdb D /trunk/wizards/sir_sad.eff D /trunk/wizards/sites_mlt.pdb D /trunk/wizards/smiles.dat D /trunk/wizards/solve.setup D /trunk/wizards/start.pdb D /trunk/wizards/starting_map.mtz D /trunk/wizards/test3.gz D /trunk/wizards/test9.gz D /trunk/wizards/test_ampl_int_autobuild.tgz D /trunk/wizards/test_labels_1.std D /trunk/wizards/test_labels_2.std D /trunk/wizards/test_labels_3.std D /trunk/wizards/test_missing.std D /trunk/wizards/test_multi.tgz D /trunk/wizards/test_ncs.standard D /trunk/wizards/test_ncs_find_ncs.standard D /trunk/wizards/test_ncs_lc.pdb D /trunk/wizards/test_resno.std D /trunk/wizards/test_resno.tgz D /trunk/wizards/unmerged.mtz D /trunk/wizards/w1.sca D /trunk/wizards/w1_rna.sca D /trunk/wizards/w2.sca remove unneeded files ------------------------------------------------------------------------ Cheers, Richard Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

easy_mp.multi_core_run function
by R.D. Oeffner 19 Dec '16

19 Dec '16

Hi, I needed a way to do multiprocessing job runs in a simple fashion on a multicore PC. Gabor provided much of the underlying machinery in libtbx for that. But I have often found I'm wasting too much time setting it all up. So I crafted a small function, easy_mp.multi_core_run(), out of his example in Computational Crystallography Newsletter (2013). 4, p21 which does exactly that. Unlike easy_mp.parallel_map() this function continues if one of the jobs crashes catastrophically. It takes as arguments name of function to be parallelised, a list of argument tuples and number of processors to run on. As the jobs completes it returns an iterator of tuples of arguments with corresponding results. Example: def RunMyJob(args1, args2): pass def GatherResults(parmres): pass from libtbx import easy_mp argstuples = [(args1a, args2a), (args1b, args2b), (args1c, args2c) ] for i, parmres in enumerate(easy_mp.multi_core_run( RunMyJob, argstuples, nproc)): GatherResults( parmres ) I'm interested in comments or if this kind of simple function is already available elsewhere. Regards, Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234 mobile: +44(0)7712 887162

1 0

Making branches by accident
by Graeme.Winter＠diamond.ac.uk 19 Dec '16

19 Dec '16

Morning all I am seeing a certain amount of "Merge branch 'master' of github.com:cctbx/cctbx_project" coming through on the commits - this usually means you did not do a git pull -rebase before the git push. This can be set to the default by using the spell Markus sent out git config --global pull.rebase true This will need to be done on each machine you push from, else getting the habit of doing a git pull -rebase before push is a good one. We have had this on and off with DIALS but it tends to pass easily enough. What bad happens? Nothing really but the history becomes confusing... So: may be worth checking that you have the pull.rebase thing set? Cheerio Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

6 11

Re: [cctbxbb] issue tracker on https://github.com/cctbx/cctbx_project
by markus.gerstel＠diamond.ac.uk 10 Dec '16

10 Dec '16

Hi, Yes, we can do that with the tools we have. (Jenkins + Python + Github API) -Markus On 9 Dec 2016 16:13, Graeme.Winter(a)diamond.ac.uk wrote: Hi Nigel, all That could be harder but not beyond our capabilities I expect However the response seems to be a resounding go for it, so I’m sure the issue tracker will be activated RSN Best wishes Graeme From: cctbxbb-bounces(a)phenix-online.org [mailto:[email protected]] On Behalf Of Nigel Moriarty Sent: 09 December 2016 15:43 To: cctbx mailing list Subject: Re: [cctbxbb] issue tracker on https://github.com/cctbx/cctbx_project One feature that would be cool is to set a (say) weekly email to remind developers of the issues assigned to them. In my other collabs, there is a complaints that the issues are not being noticed... Cheers Nigel --- Nigel W. Moriarty Building 33R0349, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709 Email : NWMoriarty(a)LBL.gov<mailto:[email protected]> Fax : 510-486-5909 Web : CCI.LBL.gov<http://CCI.LBL.gov> On Fri, Dec 9, 2016 at 7:40 AM, Aaron Brewster <asbrewster(a)lbl.gov<mailto:[email protected]>> wrote: Also +1. I like how it's possible to reference issues in commits by using #, which then show up as links in the commit emails and website history. One can cite issue numbers, or even interact with them. For example, if your commit includes the phrase "closes #42" or "fixes #42", then issue 42 will be closed when you push your commit. It's Nifty. -Aaron On Fri, Dec 9, 2016 at 7:29 AM, Nigel Moriarty <nwmoriarty(a)lbl.gov<mailto:[email protected]>> wrote: We at Phenix would like to see how the issue/bug tracker works as it's an important feature we'd like to use when we migrate our other repos to GitHub. Cheers Nigel --- Nigel W. Moriarty Building 33R0349, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709<tel:%28510%29%20486-5709> Email : NWMoriarty(a)LBL.gov<mailto:[email protected]> Fax : 510-486-5909<tel:%28510%29%20486-5909> Web : CCI.LBL.gov<http://CCI.LBL.gov> On Fri, Dec 9, 2016 at 6:23 AM, Luc Bourhis <luc_j_bourhis(a)mac.com<mailto:[email protected]>> wrote: Hi Graeme, Does anyone have an objection to opening up the issue tracker on […] Your thoughts? +1 Best wishes, Luc _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org<mailto:[email protected]> http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org<mailto:[email protected]> http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org<mailto:[email protected]> http://phenix-online.org/mailman/listinfo/cctbxbb -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

issue tracker on https://github.com/cctbx/cctbx_project
by Graeme.Winter＠diamond.ac.uk 10 Dec '16

10 Dec '16

Good afternoon all Does anyone have an objection to opening up the issue tracker on https://github.com/cctbx/cctbx_project at the moment we make a lot of use of issue trackers for dials and xia2, and for cctbx this would be helpful - so we could for example host issues in a more appropriate place like https://github.com/dials/dials/issues/265 which would be more "at home" in cctbx Your thoughts? Thanks & best wishes Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

4 6

Moving cctbx to git
by markus.gerstel＠diamond.ac.uk 23 Nov '16

23 Nov '16

Hello everyone, As Aaron requested - here is an update regarding the move to GitHub. At this time if you want to commit code: *keep committing to Sourceforge*. Do not worry about leaving uncommitted changes in your SVN tree. Those can be migrated later. The Sourceforge cctbx repository history has now been migrated to GitHub. The repository lives at https://github.com/cctbx/cctbx. Please do not commit to this repository yet! Over the next few days the bootstrap script will be modified alongside the build servers to point to the GitHub repository. This means there will only be limited testing, and the produced builds will lag behind the development effort. To be listed as a developer on https://github.com/cctbx you need to set your visibility to public at https://github.com/orgs/cctbx/people<%20https:/github.com/orgs/cctbx/people>. You can also view some fancy graphs about the cctbx project history at https://github.com/cctbx/cctbx/graphs/contributors<%20https:/github.com/cctbx/cctbx/graphs/contributors> If you haven't done it before, now is a good time to set up your local installation of git to use your name and mail address with git config --global user.name "John Doe" git config --global user.email your_email(a)example.com<mailto:[email protected]> as well as this setting to avoid merge commits: git config --global pull.rebase true Note that you must set these on every computer you use! You may want to keep a copy of those commands around in case you ever get a new computer. You should use mail addresses registered with your github account, or - if you enable it in your github account - the anonymous address offered by github. You may want to use the time to sort out your ssh access keys with your GitHub account, and play around with git a bit. You can test that everything works using the temporary sandbox repository of cctbx. Here are some commands to get you started: git clone [email protected]:cctbx/cctbx-playground.git cd cctbx-playground echo some text >> some-file git add some-file git commit -m "some commit" git push git log Then go to https://github.com/cctbx/cctbx-playground/commits/master to see how other people will see your commits. The next steps: *Tuesday 8th November 2016, 11:59pm Pacific Time* *= Wednesday 9th November 2016, 07:59am GMT:* Sourceforge: Committing allowed. GitHub: Checkout/Cloning allowed, *Read-only*. We will update our build servers and the bootstrap script to refer to the git repositories. All commits still run into Sourceforge, and will be copied over to GitHub automatically every couple of hours. Since the build servers will only see code on GitHub, testing and deployment will be limited during this period. *Tuesday 22nd November 2016, 11:59pm Pacific Time* *= Wednesday 23rd November 2016, 07:59am GMT:* Sourceforge: *Read-only*. GitHub: Committing allowed We should have achieved a stable build on both Jenkins and Buildbot by this time. Change any website pointers from SF to GitHub. Instructions to migrate local developer repositories will be sent around. All future commits should go to GitHub only. Remaining commits on Sourceforge will be migrated to GitHub. -Markus Dr Markus Gerstel MBCS Postdoctoral Research Associate Tel: +44 1235 778698 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

5 8

Re: [cctbxbb] cctbxbb Digest, Vol 92, Issue 17
by Tijskens Engelbert 22 Nov '16

22 Nov '16

> Message: 2 > Date: Mon, 21 Nov 2016 13:22:20 +0000 > From: Gabor Bunkoczi <gabor.bunkoczi(a)googlemail.com> > To: cctbx mailing list <cctbxbb(a)phenix-online.org> > Subject: Re: [cctbxbb] convert an arbitrary point to one inside the > asymmetric unit > Message-ID: > <CABYxhoWJgnRWL0Tun6M9MzcQAWNfy77YMNRSqEw+L4_b5yFkvA(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Dear Engelbert, > > you can use the asu_mappings objects to compute this (there may be > something more suitable for regular grids). Here is a link to a tutorial > written by Ralf Grosse-Kunstleve, where this is explained: > > http://cci.lbl.gov/publications/download/iucrcompcomm_aug2004.pdf > > BW, Gabor I have looked into this, but if i understand it well, it does exactly the opposite of what i want: it extends the asymmetric unit to a spherical region and makes symmetric copies of some specified points in the (scatterers in the tutorial mentioned). i have points outside the asymmetric unit, and i want to find the corresponding points inside the asymmetric unit from which i can interpolate the electron density. kindest regards and still hoping … bert

2 1

convert an arbitrary point to one inside the asymmetric unit
by Tijskens Engelbert 21 Nov '16

21 Nov '16

Hi, I am an experienced C++ and Python programmer, but new to cctbx. Can any tell me if there is an algorithm in cctbx available to convert an arbritrary point in direct space to the corresponding point in the asymmetric unit of a given space group? i can figure out how to convert it to the corresponding point in the unit cell at the origin, but from there to the asymmetric unit is unclear to me. I found the brick class which gives me the assymmetric unit from cctbx import crystal,sgtbx if __name__=='__main__': crystal_symmetry = crystal.symmetry( unit_cell=(3.5093,3.5093,3.5093,90,90,90) , space_group_symbol="229" # = "I m -3 m" ) sg = crystal_symmetry.space_group() brick = sgtbx.brick( sg.type() ) print brick I also found cctbx::sgtbx::asu and the classes therein, but they are largely undocumented. For the interested, this is the problem i want to solve: I have a (scalar) physical property P(r), depending on spatial position, presented as a 3D array on a regular grid over the unit cell (space group is known). I need to integrate P(r)*f(r), f(r) being an arbitrary function over a region of direct space, which typically comprises several unit cells. In order to reduce the memory footprint of the P array, i would like to restrict it to the space group’s asymmetric unit. All help is greatly appreciated. kindest regards, Dr [Engel]bert Tijskens HPC Analyst/Consultant Flemish Supercomputer Center vscentrum.be<http://vscentrum.be> HPC core facility CalcUA uantwerp.be/calcua<http://uantwerp.be/calcua> Computational mathematics www.uantwerpen.be/en/rg/cma<http://www.uantwerpen.be/en/rg/cma> University of Antwerp Middelheimlaan1 G.309, B-2020 Antwerp, Belgium engelbert.tijskens(a)uantwerpen.be<mailto:[email protected]> ++32 3 265 3879

3 3

Re: [cctbxbb] cctbxbb Digest, Vol 92, Issue 17
by Tijskens Engelbert 21 Nov '16

21 Nov '16

> Message: 3 > Date: Mon, 21 Nov 2016 13:38:21 +0000 > From: Gabor Bunkoczi <gabor.bunkoczi(a)googlemail.com> > To: cctbx mailing list <cctbxbb(a)phenix-online.org> > Subject: Re: [cctbxbb] convert an arbitrary point to one inside the > asymmetric unit > Message-ID: > <CABYxhoUxqFsc0f_TjWwn4oC4hKFk8AEbzwx5=9=_+gwW3xmkpg(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > On second thought, the simplest solution is to use the direct_space_asu > object. This is available for all space groups: > >>>> from cctbx import sgtbx >>>> sgi = sgtbx.space_group_info( "P2" ) >>>> asu = sgi.direct_space_asu() >>>> asu.is_inside( ( 0, 0, 0 ) ) > True >>>> asu.is_inside( (0, 0, 0.51 ) ) > False > > BW, Gabor > Thanks for your quick reply, Gabor. Unfortunately, it does not solve my question. For the example you give above, the second point is not inside the asu, and my question is how i can the corresponding point inside the asu, For a simple space group as P2 the answer is easily figured out by hand. For a space group such as I m -3 m, it becomes complicated. I would like to have a (computationally efficient) method for finding the point inside the asu given an arbitrary point which works for every space group. Any “third” thoughts? kindest regards, bert

2 1

Re: [cctbxbb] Unstable platform-dependent sort order for flex.sort_permutation
by richard.gildea＠diamond.ac.uk 20 Nov '16

20 Nov '16

Hi Oleg, The permutations would lead to the same sequence of miller indices, but in the context of an unmerged reflection array (i.e. miller.array.sort_permutation), then the data associated with those miller indices would not necessarily be in the same order. As the dataset is then split into two half-datasets, this difference in sort order leads to a different value of the calculate correlation coefficient between those two half datasets: https://github.com/cctbx/cctbx_project/blob/master/cctbx/miller/__init__.py… Cheers, Richard Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE ________________________________ From: cctbxbb-bounces(a)phenix-online.org [cctbxbb-bounces(a)phenix-online.org] on behalf of oleg(a)olexsys.org [oleg(a)olexsys.org] Sent: 18 November 2016 15:59 To: cctbx mailing list Subject: Re: [cctbxbb] Unstable platform-dependent sort order for flex.sort_permutation Hi Richard, it is a bug only if the permutations do not lead to the same sequence. Otherwise you cannot expect to get the same sorting permutations for collections with redundant data on different platforms or even between different versions of compilers. Cheers, Oleg. ________________________________ From: cctbxbb-bounces(a)phenix-online.org <cctbxbb-bounces(a)phenix-online.org> on behalf of richard.gildea(a)diamond.ac.uk <richard.gildea(a)diamond.ac.uk> Sent: 18 November 2016 14:12:29 To: cctbxbb(a)phenix-online.org Subject: [cctbxbb] Unstable platform-dependent sort order for flex.sort_permutation Hi, I've been trying to diagnose why the miller array cc_anom calculations seem to be platform dependent. I've narrowed this down to a variation in the sort order returned by flex.sort_permutation which is called from within miller.array.sort("packed_indices"). The following code demonstrates the problem and the different output I get on mac and Linux: Mac: >>> from scitbx.array_family import flex >>> a = flex.size_t([7, 1, 1, 5, 1, 3, 3, 7, 1, 7, 3, 3, 5, 7, 5, 5, 5, 1, 1, 1]) >>> print list(flex.sort_permutation(a)) [1, 2, 4, 8, 17, 18, 19, 5, 6, 10, 11, 3, 12, 14, 15, 16, 0, 7, 9, 13] Linux: >>> from scitbx.array_family import flex >>> a = flex.size_t([7, 1, 1, 5, 1, 3, 3, 7, 1, 7, 3, 3, 5, 7, 5, 5, 5, 1, 1, 1]) >>> print list(flex.sort_permutation(a)) [19, 1, 2, 18, 4, 17, 8, 11, 10, 6, 5, 12, 14, 15, 16, 3, 9, 7, 13, 0] Is it known/expected for the sort order to be platform dependent, or is this a bug? Here is the relevant code for flex.sort_permutation(): https://github.com/cctbx/cctbx_project/blob/master/scitbx/array_family/sort… Cheers, Richard Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org http://phenix-online.org/mailman/listinfo/cctbxbb

5 9

Unstable platform-dependent sort order for flex.sort_permutation
by richard.gildea＠diamond.ac.uk 18 Nov '16

18 Nov '16

Hi, I've been trying to diagnose why the miller array cc_anom calculations seem to be platform dependent. I've narrowed this down to a variation in the sort order returned by flex.sort_permutation which is called from within miller.array.sort("packed_indices"). The following code demonstrates the problem and the different output I get on mac and Linux: Mac: >>> from scitbx.array_family import flex >>> a = flex.size_t([7, 1, 1, 5, 1, 3, 3, 7, 1, 7, 3, 3, 5, 7, 5, 5, 5, 1, 1, 1]) >>> print list(flex.sort_permutation(a)) [1, 2, 4, 8, 17, 18, 19, 5, 6, 10, 11, 3, 12, 14, 15, 16, 0, 7, 9, 13] Linux: >>> from scitbx.array_family import flex >>> a = flex.size_t([7, 1, 1, 5, 1, 3, 3, 7, 1, 7, 3, 3, 5, 7, 5, 5, 5, 1, 1, 1]) >>> print list(flex.sort_permutation(a)) [19, 1, 2, 18, 4, 17, 8, 11, 10, 6, 5, 12, 14, 15, 16, 3, 9, 7, 13, 0] Is it known/expected for the sort order to be platform dependent, or is this a bug? Here is the relevant code for flex.sort_permutation(): https://github.com/cctbx/cctbx_project/blob/master/scitbx/array_family/sort… Cheers, Richard Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

convert an arbitrary point to one inside the asymmetric unit
by Tijskens Engelbert 18 Nov '16

18 Nov '16

Hi, I am an experienced C++ and Python programmer, but new to cctbx. Can any tell me if there is an algorithm in cctbx available to convert an arbritrary point in direct space to the corresponding point in the asymmetric unit of a given space group? i can figure out how to convert it to the corresponding point in the unit cell at the origin, but from there to the asymmetric unit is unclear to me. I found the brick class which gives me the assymmetric unit from cctbx import crystal,sgtbx if __name__=='__main__': crystal_symmetry = crystal.symmetry( unit_cell=(3.5093,3.5093,3.5093,90,90,90) , space_group_symbol="229" # = "I m -3 m" ) sg = crystal_symmetry.space_group() brick = sgtbx.brick( sg.type() ) print brick I also found cctbx::sgtbx::asu and the classes therein, but they are largely undocumented. For the interested, this is the problem i want to solve: I have a (scalar) physical property P(r), depending on spatial position, presented as a 3D array on a regular grid over the unit cell (space group is known). I need to integrate P(r)*f(r), f(r) being an arbitrary function over a region of direct space, which typically comprises several unit cells. In order to reduce the memory footprint of the P array, i would like to restrict it to the space group’s asymmetric unit. All help is greatly appreciated. kindest regards, Dr [Engel]bert Tijskens HPC Analyst/Consultant Flemish Supercomputer Center vscentrum.be<http://vscentrum.be> HPC core facility CalcUA uantwerp.be/calcua<http://uantwerp.be/calcua> Computational mathematics www.uantwerpen.be/en/rg/cma<http://www.uantwerpen.be/en/rg/cma> University of Antwerp Middelheimlaan1 G.309, B-2020 Antwerp, Belgium engelbert.tijskens(a)uantwerpen.be<mailto:[email protected]> ++32 3 265 3879

1 0

who else has troubles to commit to cctbx now?
by Pavel Afonine 16 Nov '16

16 Nov '16

Can't commit anything since yesterday: pva_mbp:/Users/pafonine/phenix_dev/modules/cctbx_project/mmtbx % svn commit -m "test" maps/correlation.py Sending maps/correlation.py Transmitting file data .svn: E000013: Commit failed (details follow): svn: E000013: Can't create directory '/svn/p/cctbx/code/db/transactions/25717-1.txn': Permission denied pva_mbp:/Users/pafonine/phenix_dev/modules/cctbx_project/mmtbx % Nigel has the same issue. I don't see any warnings on sourceforge. Pavel

3 3

command line syntax for multiple scope PHIL
by R.D. Oeffner 09 Nov '16

09 Nov '16

Say a program, myprog, has a master PHIL definition like: minimization.input .multiple = True { file_name = None .type = str nval = None .type = int } and a PHIL input file like: minimization.input { file_name = experiment1.dat nval = 42 } minimization.input { file_name = experiment2.dat nval = 13 } so that the user on the command line can type "myprog phil_input.txt" What would be the equivalent command line syntax of the PHIL input above? Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234 mobile: +44(0)7712 887162

2 2

Failing shelxl test
by Graeme.Winter＠diamond.ac.uk 09 Nov '16

09 Nov '16

Folks, I keep getting this error: libtbx.python "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/development/run_shelx.py" P31 [FAIL] Time: 1.38 Return code: 1 OKs: 0 Standard error: Traceback (most recent call last): File "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/development/run_shelx.py", line 253, in <module> run(args=sys.argv[1:]) File "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/development/run_shelx.py", line 250, in run debug_utils.parse_options_loop_space_groups(args, run_call_back) File "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/development/debug_utils.py", line 120, in parse_options_loop_space_groups show_cpu_times, **kwds) File "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/development/debug_utils.py", line 83, in loop_space_groups call_back_result = call_back(flags, space_group_info, **kwds) File "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/development/run_shelx.py", line 244, in run_call_back verbose=flags.Verbose) File "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/development/run_shelx.py", line 234, in exercise run_shelx(shelx_titl, structure_factors, verbose=verbose) File "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/development/run_shelx.py", line 210, in run_shelx check_anisou(shelx_titl, xray_structure, shelx_pdb, verbose) File "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/development/run_shelx.py", line 163, in check_anisou assert TotalMismatches == 0 AssertionError Please verify these tests manually. it is annoying as I understand it is to do with a change in shelxl - would it be OK to comment the test out to allow builds to pass? I am not sure it is useful at the moment Thank you Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

Re: [cctbxbb] Git status update
by markus.gerstel＠diamond.ac.uk 04 Nov '16

04 Nov '16

Hello everyone, since I have received no complaints and heard nothing to the contrary, so I thought I would let you know on the current state of the git migration: Until *Thursday, 1 September 2016*, 07:59 BST/08:59 CEST/11:59 pm PDT Cctbx development happens on Sourceforge. Note that cctbx development is "frozen" in preparation for the phenix release. If you are actively developing cctbx you should *now* sign up for a GitHub account. If you are not among the 27 people that have already joined the cctbx project at https://github.com/cctbx then please send me a mail with your github account name so I can invite you. In preparation for the move to github you should also install git on your machines. Set up your local installation of git on each machine you are using by running > git config --global pull.rebase true > git config --global user.name "John Doe" > git config --global user.email "your_email(a)example.com" Use an email address you have set up in your GitHub account at https://github.com/settings/emails. Note that this address will be forever publically readable! If you do not want your mail address to be public, you need to verify any (private) mail address with GitHub and then check the /Keep my email address private/ box at the link above. You can then use the dummy address <yourgithubname>@users.noreply.github.com instead. Other useful settings may be > git config --global alias.up "pull --rebase" To enable a *git up* command working much like svn up does. > git config --global alias.unstage "reset HEAD" *git unstage* to easily undo *git add*/*git rm*. > git config --global alias.mkremotebranch "push origin -u" *git mkremotebranch* will copy your current local branch to GitHub. > git config --global alias.rmremotebranch "push origin --delete" *git rmremotebranch <name>* will delete said branch from GitHub. > git config --global color.ui auto This adds color to the git output, which some may find useful for eg. *git status*. On the 1st of September the transitional period starts, and we will begin changing scripts to point to the new code locations. The transition may be delayed somewhat if the phenix release encounters any delays. I will send you further updates closer to the day. You can find and comment on the transition plan at https://docs.google.com/document/d/1bTiGbnvw8lNUrBLbRmhEdnd22McDqBbiKxcpyga… Best wishes -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

6 16

use_internal_variance in iotbx.merging_statistics
by Keitaro Yamashita 01 Nov '16

01 Nov '16

Dear Phenix/CCTBX developers, iotbx/merging_statistics.py is used by phenix.merging_statistics, phenix.table_one, and so on. By upgrading phenix from 1.10.1 to 1.11, merging statistics-related codes were significantly changed. Previously, miller.array.merge_equivalents() was always called with argument use_internal_variance=False, which is consistent with XDS, Aimless and so on. Currently, use_internal_variance=True is default, and cannot be changed by users (see below). These changes were made by @afonine and @rjgildea in rev. 22973 (Sep 26, 2015) and 23961 (Mar 8, 2016). Could anyone explain why these changes were introduced? https://sourceforge.net/p/cctbx/code/22973 https://sourceforge.net/p/cctbx/code/23961 My points are: - We actually cannot control use_internal_variance= parameter because it is not passed to merge_equivalents() in class filter_intensities_by_sigma. - In previous versions, if I gave XDS output to phenix.merging_statistics, <I/sigma> values calculated in the same way (as XDS does) were shown; but not in the current version. - For (for example) phenix.table_one users who expect this behavior, it can give inconsistency. The statistics would not be consistent with the data used in refinement. cf. the related discussion in cctbxbb: http://phenix-online.org/pipermail/cctbxbb/2012-October/000611.html Best regards, Keitaro

4 6

Monthly "how far are we from moving to git" message
by Graeme.Winter＠diamond.ac.uk 21 Oct '16

21 Oct '16

Good afternoon / morning / evening (depending on your locale) I'm just running a bootstrap build and having to check out the whole of cctbx one ... file ... at ... a ... time from svn & it reminded me we were going to move over to git at some point - ISTR was determined by the PHENIX release. Just wondering how this is going & how long it is likely to be until we can get this ball rolling :) Thanks & best wishes Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 6

sginfo and sgtbx
by Pascal 21 Oct '16

21 Oct '16

Hi, Just out of curiosity, sginfo website mentions that it has been superseeded by sgtbx. Is that a brand new implementation? I don't recognize sginfo code at all. Also, anyone can confirm that sgtbx is tightly embedded is cctbx and cannot be used standalone? Regards, Pascal

4 6

flex.int.as_double() performance
by richard.gildea＠diamond.ac.uk 11 Oct '16

11 Oct '16

Hi, Whilst looking into performance of various DIALS routines recently, I noticed that the flex.int.as_double() conversion takes roughly 3x as long as the equivalent functionality in numpy: (the choice of a 2527x2463 array is simply because this is the size of the raw data array for a Pilatus 6M detector) import time from dials.array_family import flex n = 100 a = flex.int(flex.grid(2527, 2463)) t0 = time.time() for i in range(n): b = a.as_double() t1 = time.time() t = t1-t0 print "flex.int.as_double(): %.2fs (%.3fs/call)" %(t, t/n) import numpy as np a = np.random.randint(100, size=a.all()) t0 = time.time() for i in range(n): b = a.astype(np.float64) t1 = time.time() t = t1-t0 print "numpy.astype(np.float64): %.2fs (%.3fs/call)" %(t, t/n) $ libtbx.python time_as_double.py flex.int.as_double(): 5.76s (0.058s/call) numpy.astype(np.float64): 1.86s (0.019s/call) This is the current version of the as_double function: static flex_double as_double(f_t const& a) { shared_plain<double> result(a.size(), init_functor_null<double>()); for(std::size_t i=0;i<a.size();i++) result[i] = a[i]; return flex_double(result, a.accessor()); } Replacing the above code with the following, I see a significant runtime improvement, now flex.int.as_double() is marginally quicker than the equivalent numpy routine: static flex_double as_double(f_t const& a) { shared_plain<double> result(a.begin(), a.end()); return flex_double(result, a.accessor()); } $ libtbx.python time_as_double.py flex.int.as_double(): 1.64s (0.016s/call) numpy.astype(np.float64): 1.82s (0.018s/call) Is there any obvious reason why making this change would be A Bad Thing? In the context of dials spot-finding, this change results in a 22s saving (out of a total 172s with the original code) when running dials.find_spots on a 540 image Pilatus 6M dataset using a single processor, so it is definitely a noticeable speed-up. All timings done on a macbook pro. Cheers, Richard Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 3

"import pickle"
by markus.gerstel＠diamond.ac.uk 23 Sep '16

23 Sep '16

Hi everyone, I note that we are using "import pickle" on some 57 occasions. Is there a particular reason we don't import cPickle? For example: --- libtbx/env_config.py (revision 25451) +++ libtbx/env_config.py (working copy) @@ -7,7 +7,10 @@ from libtbx import adopt_init_args import platform import shutil -import pickle +try: + import cPickle as pickle +except: + import pickle from cStringIO import StringIO import re import sys, os This should still work on all systems without issue. According to python docs the only difference is that cPickle can't be subclassed. If it's alright with you I would go through those imports and update them to use cPickle where possible. -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 2

bootstrap / svn method / say reason not anonymous && http
by Graeme.Winter＠diamond.ac.uk 20 Sep '16

20 Sep '16

Hello folks, In bootstrap.py is there any logical reason why we can’t use %sfmethod for anonymous as well as authenticated access to the cctbx_project sources i.e. in # Core CCTBX repositories # These must all provide anonymous access. class cctbx_module(SourceModule): module = 'cctbx_project' anonymous = ['svn','svn://svn.code.sf.net/p/cctbx/code/trunk'] authenticated = ['svn', '%(sfmethod)s://%(sfuser)[email protected]/p/cctbx/code/trunk'] also respect %(sfmethod) in place of svn as transport layer. Will help people behind firewalls. Appreciate that this will also need changing when we move to git anyhow… but meantimes there are people out there this would help. Not keen to make this change myself without checking first…! Thanks & best wishes Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

rpath problem on Mac system
by Haiguang Liu 20 Sep '16

20 Sep '16

Dear CCTBX developing team, Is there already a solution for the run path problem in the new Mac OS? The System Integrity Protection <https://support.apple.com/en-us/HT204899> does not allow relative run path. I temporarily disabled this feature to run the programs that requires boost (for example). If there is already a solution, please let me know. Thanks, Haiguang

3 4

Python 2.4 compatibility
by richard.gildea＠diamond.ac.uk 16 Sep '16

16 Sep '16

This commit uses the "with" statement, which was introduced in Python 2.5, to libtbx/__init__.py, preventing installation of DIALS from sources on systems with Python 2.4: https://sourceforge.net/p/cctbx/code/25101/ Could this code be changed to remove the use of the "with" statement? Cheers, Richard Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 5

failing test with CCP4 7.0.18
by markus.gerstel＠diamond.ac.uk 16 Sep '16

16 Sep '16

Hi, The test libtbx.python $(libtbx.find_in_repositories cctbx/development/run_shelx.py) P31 works with CCP4 7.0.17, but fails when run with SHELXL from CCP4 7.0.18. The test apparently involves creating a small random crystal structure, calculating the structure factors and solving the structure using SHELXL. The updated SHELXL version creates deterministic, but slightly different anisotropic temperature factors. I am not sure if this is a problem with the test or with the SHELXL update. Could anybody have a look into that? -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

4 4

Re: [cctbxbb] [Cctbx-cvs] SF.net SVN: cctbx:[25333] trunk/libtbx/env_config.py
by markus.gerstel＠diamond.ac.uk 09 Sep '16

09 Sep '16

Hi, I just spent some time tracking software crashes to this change. Is setting the default to en_US really appropriate and what we want? In particular it affects the output of downstream, external software we run from within python. What is the unicode issue you hint at in the commit message? -Markus Dr Markus Gerstel MBCS Postdoctoral Research Associate Tel: +44 1235 778698 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE -----Original Message----- From: bkpoon(a)users.sourceforge.net [mailto:[email protected]] Sent: 07 September 2016 00:54 To: cctbx-cvs(a)lists.sourceforge.net Subject: [Cctbx-cvs] SF.net SVN: cctbx:[25333] trunk/libtbx/env_config.py Revision: 25333 http://sourceforge.net/p/cctbx/code/25333 Author: bkpoon Date: 2016-09-06 23:54:29 +0000 (Tue, 06 Sep 2016) Log Message: ----------- Unicode support: set LC_ALL in dispatchers to the one in the user's environment (if available, and supports UTF-8), otherwise use the default setting of en_US.UTF-8; fixes unicode issue with python in Linux (e.g. os.path functions do not work correctly with unicode if LC_ALL=C Modified Paths: -------------- trunk/libtbx/env_config.py Modified: trunk/libtbx/env_config.py =================================================================== --- trunk/libtbx/env_config.py 2016-09-06 21:15:34 UTC (rev 25332) +++ trunk/libtbx/env_config.py 2016-09-06 23:54:29 UTC (rev 25333) @@ -945,6 +945,15 @@ def write_bin_sh_dispatcher(self, source_file, target_file, source_is_python_exe=False): + + # determine LC_ALL from environment (Python UTF-8 compatibility in Linux) + LC_ALL = os.environ.get('LC_ALL') # user setting + if (LC_ALL is not None): + if ( ('UTF-8' not in LC_ALL) and ('utf8' not in LC_ALL) ): + LC_ALL = None + if (LC_ALL is None): + LC_ALL = 'en_US.UTF-8' # default + f = target_file.open("w") if (source_file is not None): print >> f, '#! /bin/sh' @@ -975,7 +984,7 @@ print >> f, '#' print >> f, _SHELLREALPATH_CODE print >> f, 'unset PYTHONHOME' - print >> f, 'LC_ALL=C' + print >> f, 'LC_ALL=' + LC_ALL print >> f, 'export LC_ALL' print >> f, 'LIBTBX_BUILD="$(shellrealpath "$0" && cd "$(dirname "$RESULT")/.." && pwd)"' print >> f, 'export LIBTBX_BUILD' This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. ------------------------------------------------------------------------------ _______________________________________________ Cctbx-cvs mailing list Cctbx-cvs(a)lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/cctbx-cvs -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 3

Inserting F000 into reflection (mtg) file
by Nicholas Pearce 22 Aug '16

22 Aug '16

Hi Everyone, I’ve been looking through the cctbx/phenix scripts and although there’s a script for estimating F000 (phenix.f000), I can’t find any way of inserting it into an mtz file to make it accessible for easy map generation. Is this implemented anywhere/what do people use as the best way of doing this? If the insertion of F000 into an mtz file has been purposefully avoided, I presume it's done because we don't want the possibility of someone refining a model where F000 is used in refinement. Unless there’s another reason? Thanks, Nick

2 1

Git
by Graeme.Winter＠diamond.ac.uk 01 Aug '16

01 Aug '16

Folks, Can we revisit the idea of moving to git for cctbx? Thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

7 18

Structure Factor for 000
by Gareth.Nisbet＠diamond.ac.uk 28 Jul '16

28 Jul '16

Is it possible to get the structure_factors function to give the structure factor for the 000 reflection? I appreciate this might be a slightly odd question. Gareth -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 3

Berkeley Server Downtime (24 hours, 7/22 - 7/23)
by Billy Poon 20 Jul '16

20 Jul '16

Dear all, On Friday, July 22, our servers will be shut down starting at 5:00 pm Pacific for scheduled maintenance in our building. We expect that the servers will be available the next day (7/23) by 5:00 pm Pacific at the latest. This will affect, - The Phenix web site (http://phenix-online.org/) - The DIALS West web site (http://dials.lbl.gov/) - The CCI web site (http://cci.lbl.gov/) - The Structural Biology Technology Portal (http://technology.sbkb.org/) Bug reports and emails requesting help should be queued and we will receive them once the servers are back online. We apologize for any inconvenience. -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

1 0

Re: [cctbxbb] Git
by R. D. Oeffner 02 Jul '16

02 Jul '16

How often is this git repo of cctbx copied from the svn repo of cctbx? Rob -----Original Message----- From: cctbxbb-request(a)phenix-online.org Sent: Friday, July 1, 2016 8:00 PM To: cctbxbb(a)phenix-online.org Subject: cctbxbb Digest, Vol 88, Issue 1 Send cctbxbb mailing list submissions to cctbxbb(a)phenix-online.org To subscribe or unsubscribe via the World Wide Web, visit http://phenix-online.org/mailman/listinfo/cctbxbb or, via email, send a message with subject or body 'help' to cctbxbb-request(a)phenix-online.org You can reach the person managing the list at cctbxbb-owner(a)phenix-online.org When replying, please edit your Subject line so it is more specific than "Re: Contents of cctbxbb digest..." Today's Topics: 1. Re: Git (markus.gerstel(a)diamond.ac.uk) ---------------------------------------------------------------------- Message: 1 Date: Fri, 1 Jul 2016 12:34:56 +0000 From: <markus.gerstel(a)diamond.ac.uk> To: <cctbxbb(a)phenix-online.org> Subject: Re: [cctbxbb] Git Message-ID: <415F7F573406BB498256C68158D45674666211BB@exchmbx01> Content-Type: text/plain; charset="utf-8" Hello everyone, Following up the earlier discussion, do have we reached a consensus? Also, I think timeframe has not been discussed, but I assume this will boil down to ?after phenix release? anyway. In the meantime I have contacted (nearly) all cctbx developers and obtained their github user-IDs - alongside some enthusiastic* feedback. I will mass-invite you all soonishly to a cctbx organisation with a preliminary git copy of the svn history to play with. Best wishes -Markus Markus Gerstel MBCS Postdoctoral Research Associate Tel: +44 1235 778698 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE * The best comment I received was ?What?s CCTBX?? ? and that was talking to the correct person :) (name withheld) -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

Installation failure after package updates
by markus.gerstel＠diamond.ac.uk 27 Jun '16

27 Jun '16

Following the latest package updates we are now observing installer failures. In particular the configure step appears to fail with a missing libffi.so.6 Any ideas how to rectify this, or if this error can be ignored safely? -Markus Running with --verbose: ========================================================================== DIALS Installation version: dev20160610 machine type: intel-linux-2.6-x86_64 OS version: 2.6.32-573.18.1.el6.x86_64 destination: /dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610 # of processors: 24 ========================================================================= Configuring DIALS components... /dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/build ['/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/base/bin/python', '/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/libtbx/configure.py', '--current_working_directory', '/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/build', 'dials', 'xia2', 'iota', 'prime'] : Python: 2.7.8 "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/base/bin/python" : Compiler: default : Build mode: release : Warning level: 0 : Precompiled Headers: False : Static libraries: False : Static exe: False : Scan Boost headers: False : Write full flex_fwd.h files: False : Build Boost.Python extensions: True : Define BOOST_PYTHON_NO_PY_SIGNATURES: False : Define BOOST_PYTHON_BOOL_INT_STRICT: True : Enable OpenMP if possible: False : Boost threads enabled: False : Enable CUDA: False : Use opt_resources if available: False : Use environment flags: False : Enable C++11: False : Force true division: True : command_version_suffix: None : Relocatable paths anchored at: /dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/build : Top-down list of all modules involved: : prime "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/prime" : iota "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/iota" : xia2 "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/xia2" : mmtbx "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/mmtbx" : dials "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/dials" : xfel "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/xfel" : cma_es "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/cma_es" : gltbx "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/gltbx" : crys3d "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/crys3d" : rstbx "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/rstbx" : spotfinder "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/spotfinder" : annlib+adaptbx "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/annlib" : "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/annlib_adaptbx" : dxtbx "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/dxtbx" : iotbx "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/iotbx" : cbflib+adaptbx "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cbflib" : "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/cbflib_adaptbx" : ccp4io+adaptbx "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/ccp4io" : "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/ccp4io_adaptbx" : smtbx "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/smtbx" : ucif "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/ucif" : cctbx "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/cctbx" : chiltbx "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/chiltbx" : scitbx "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/scitbx" : fable "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/fable" : omptbx "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/omptbx" : boost+adaptbx "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/boost" : "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/boost_adaptbx" : tbxx "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/tbxx" : libtbx "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/libtbx" : Creating files in build directory: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/build" : dispatcher_include_template.sh : dispatcher_head.sh : setpaths.sh : setpaths.csh : setpaths_all.sh : setpaths_all.csh : setpaths_debug.sh : unsetpaths.sh : setpaths_debug.csh : unsetpaths.csh : SConstruct : Makefile : run_tests.csh : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/libtbx/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/boost_adaptbx/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/fable/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/scitbx/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/cctbx/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/smtbx/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/cbflib_adaptbx/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/iotbx/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/dxtbx/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/spotfinder/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/rstbx/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/crys3d/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/gltbx/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/xfel/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/xfel/ui/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/dials/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/mmtbx/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/xia2/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/iota/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/prime/command_line" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/boost_adaptbx/libtbx_refresh.py" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/scitbx/libtbx_refresh.py" : Generating C++ header files in: : "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/build/include/scitbx/array_family" : Generating C++ files in: : "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/build/include/scitbx/array_family/boost_python" : Using fable to convert scitbx/lbfgs.f : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/cctbx/libtbx_refresh.py" : Generating C++ files in: : "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/build/cctbx/eltbx" : Generating C++ files in: : "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/build/include/cctbx/boost_python" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/smtbx/libtbx_refresh.py" : Generating C++ files in: : "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/build/include/smtbx/boost_python" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/cbflib_adaptbx/libtbx_refresh.py" : Copying to lib: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cbflib/pycbf/pycbf.py" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/iotbx/libtbx_refresh.py" : Using fable to convert iotbx/pdb/hybrid_36_f.f : Writing: iotbx/pdb/hybrid_36_fem.cpp : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/annlib_adaptbx/libtbx_refresh.py" : Generating C++ files for self-inclusive approximate nearest neighbor : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/crys3d/libtbx_refresh.py" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/gltbx/libtbx_refresh.py" : Generating C++ files in: : "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/build/gltbx" : Info: Re-using existing font cpp files. : Hint: Remove gltbx/FONTS_UCS_DONE_FLAG_FILE : to force generation of new font files. : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/xfel/libtbx_refresh.py" : Creating cxi_user directory : Creating cxi_user/__init__.py : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/dials/libtbx_refresh.py" : DIALS 1.dev.360-ge6510c8 : Identifying autocompletable commands: dials.analyse_background dials.combine_found_spots dials.find_spots dials.index dials.integrate dials.reciprocal_lattice_viewer dials.refine dials.reindex dials.rl_png : Installing autocompletion script into: setpaths_all.sh setpaths_debug.sh setpaths.sh : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/mmtbx/libtbx_refresh.py" : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/xia2/libtbx_refresh.py" : XIA2 0.4.0.381-g9b3a992 : Processing: "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/build/exe_dev" : Regenerating module files in /dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/base : /dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/base/etc/pango not present, could not regenerate pango files : generating gtk.immodules file : /dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/base/bin/gtk-query-immodules-2.0: error while loading shared libraries: libffi.so.6: cannot open shared object file: No such file or directory : Traceback (most recent call last): : File "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/libtbx/configure.py", line 22, in <module> : if not run(): : File "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/libtbx/configure.py", line 17, in run : libtbx.env_config.cold_start(sys.argv) : File "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/libtbx/env_config.py", line 2264, in cold_start : env.refresh() : File "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/libtbx/env_config.py", line 1537, in refresh : regenerate_module_files.run(libtbx.env.under_base('.'), only_if_needed=True) : File "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/libtbx/auto_build/regenerate_module_files.py", line 154, in run : fix_gtk(base_dir, out) : File "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/libtbx/auto_build/regenerate_module_files.py", line 72, in fix_gtk : log=out) : File "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/libtbx/auto_build/installer_utils.py", line 81, in call : raise RuntimeError("Call to '%s' failed with exit code %d" % (args, rc)) : RuntimeError: Call to '/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/base/bin/gtk-query-immodules-2.0 /dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/base/lib/gtk-2.0/2.10.0/immodules/*.so> /dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/base/etc/gtk-2.0/gtk.immodules' failed with exit code 127 Traceback (most recent call last): File "./bin/install.py", line 97, in <module> installer(sys.argv[1:]).install() File "/scratch/jenkins_slave/workspace/dials_install_dls_module/dials-installer-dev/lib/libtbx/auto_build/install_distribution.py", line 197, in install self.install_from_binary() File "/scratch/jenkins_slave/workspace/dials_install_dls_module/dials-installer-dev/lib/libtbx/auto_build/install_distribution.py", line 318, in install_from_binary self.reconfigure(log=log) File "/scratch/jenkins_slave/workspace/dials_install_dls_module/dials-installer-dev/lib/libtbx/auto_build/install_distribution.py", line 484, in reconfigure call(args=args, log=log, verbose=self.options.verbose) File "/scratch/jenkins_slave/workspace/dials_install_dls_module/dials-installer-dev/lib/libtbx/auto_build/installer_utils.py", line 81, in call raise RuntimeError("Call to '%s' failed with exit code %d" % (args, rc)) RuntimeError: Call to '/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/base/bin/python /dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/modules/cctbx_project/libtbx/configure.py --current_working_directory /dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20160610/build dials xia2 iota prime' failed with exit code 1 Markus Gerstel MBCS Postdoctoral Research Associate Tel: +44 1235 778698 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 6

Re: [cctbxbb] error
by markus.gerstel＠diamond.ac.uk 23 Jun '16

23 Jun '16

Hi James, CCing cctbxbb for further support I can reproduce at least similar errors with both DIALS 1.2.5 and current nightly builds. I think this may be a matplotlib 1.3.1 problem. A minimal reproducer is: $ dials.python > import matplotlib.pyplot as plt > plt.imshow([[1]]) Gtk-Message... > (press ctrl+D) Segmentation fault When compiled on the same machine I do not get the Gtk-Messages, but I do get the segmentation fault. Markus Gerstel MBCS Postdoctoral Research Associate Tel: +44 1235 778698 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE From: Parkhurst, James (DLSLtd,RAL,LSCI) Sent: 21 June 2016 16:58 To: Gerstel, Markus (DLSLtd,RAL,LSCI) Subject: FW: error Hi Markus, I've been getting my summer student to do the cctbx_introduction tutorials in dials_scratch. He's using a version of dials downloaded from the website and when he runs the introduction script he gets errors about gdk-pixbuf giving paths to /scratch/jenkins_slave/workspace/. This seems like there is a jenkins path being set in the distributed installers. Would you have any idea how I can fix this? I'm downloading the linux installer of DIALS now so I can test this myself. Thanks, James ________________________________ [sse87232@ws188 cctbx_introduction_1]$ dials.python cctbx_introduction_1_test.py ximg2701_00001.cbf (cctbx_introduction_1_test.py:6243): GdkPixbuf-WARNING **: Cannot open pixbuf loader module file '/scratch/jenkins_slave/workspace/dials_release_build/compilationtarget/native/label/dials-ws133/build_dials/base/etc/gtk-2.0/gdk-pixbuf.loaders': No such file or directory (cctbx_introduction_1_test.py:6243): GdkPixbuf-WARNING **: Cannot open pixbuf loader module file '/scratch/jenkins_slave/workspace/dials_release_build/compilationtarget/native/label/dials-ws133/build_dials/base/etc/gtk-2.0/gdk-pixbuf.loaders': No such file or directory (cctbx_introduction_1_test.py:6243): GdkPixbuf-WARNING **: Cannot open pixbuf loader module file '/scratch/jenkins_slave/workspace/dials_release_build/compilationtarget/native/label/dials-ws133/build_dials/base/etc/gtk-2.0/gdk-pixbuf.loaders': No such file or directory (cctbx_introduction_1_test.py:6243): GdkPixbuf-WARNING **: Cannot open pixbuf loader module file '/scratch/jenkins_slave/workspace/dials_release_build/compilationtarget/native/label/dials-ws133/build_dials/base/etc/gtk-2.0/gdk-pixbuf.loaders': No such file or directory (cctbx_introduction_1_test.py:6243): GdkPixbuf-WARNING **: Cannot open pixbuf loader module file '/scratch/jenkins_slave/workspace/dials_release_build/compilationtarget/native/label/dials-ws133/build_dials/base/etc/gtk-2.0/gdk-pixbuf.loaders': No such file or directory (cctbx_introduction_1_test.py:6243): GdkPixbuf-WARNING **: Cannot open pixbuf loader module file '/scratch/jenkins_slave/workspace/dials_release_build/compilationtarget/native/label/dials-ws133/build_dials/base/etc/gtk-2.0/gdk-pixbuf.loaders': No such file or directory (cctbx_introduction_1_test.py:6243): GdkPixbuf-WARNING **: Cannot open pixbuf loader module file '/scratch/jenkins_slave/workspace/dials_release_build/compilationtarget/native/label/dials-ws133/build_dials/base/etc/gtk-2.0/gdk-pixbuf.loaders': No such file or directory Segmentation fault (core dumped) [sse87232@ws188 cctbx_introduction_1]$ which dials.python /dls/science/users/sse87232/dials/dials-v1-2-4/build/bin/dials.python -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

4 6

past contributors
by markus.gerstel＠diamond.ac.uk 16 Jun '16

16 Jun '16

I have now contacted everyone who ever contributed to cctbx in the past to get name/email details in preparation for an svn->git conversion. Everyone except these three: Erik McKee (emckee) Vishal Verma (vermav) ydr (y0udar) Anyone know how to get into contact with them? -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

Not wishing to raise this again but...
by Graeme.Winter＠diamond.ac.uk 16 Jun '16

16 Jun '16

... is there any enthusiasm out there for moving cctbx to git? The more I work with git the more I wish this existed properly for cctbx. I've raised this a bunch of times and always it is "not right now" however "a good idea" - this'll be the last time I ask... Could we move to git, please? I note the bootstrap script for Windows now has a git dependency even! Am happy to help if I can with anything though I think it should be pretty simple... was for DIALS & xia2 :) Thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

8 23

Not wishing to raise this again but...
by R.D. Oeffner 15 Jun '16

15 Jun '16

Personally I think moving the cctbx to git is a great idea as it would make it easier and safer when developing and testing new features. Given so many teams are using the cctbx this seems like a compelling reason for it to be moved. We had a chat a while ago with Berkeley about this but I've forgotten what the consensus was. Rob > > Message: 1 > Date: Tue, 14 Jun 2016 12:10:03 +0000 > From: <Graeme.Winter(a)diamond.ac.uk> > To: <cctbxbb(a)phenix-online.org> > Subject: Re: [cctbxbb] Not wishing to raise this again but... > Message-ID: <13999B6FC038EB44B067771E3CFBD091BC327D8E@exchmbx03> > Content-Type: text/plain; charset="us-ascii" > > Hi All, > > This sounds like it is heading in a pragmatic direction - I would find > it very helpful to gauge feeling for or against this. To make this > simple I have attempted (starting from a completely ignorant position) > to put together one of those online survey things: > > https://www.surveymonkey.co.uk/r/GXPY5JW > > If you could pick an option that would be great, explaining why would > be even better, or hitting reply-all to this thread would also work > (as has previously done so) I'll collate answers over the next few > days & feed back when we have (say) 10-20 or a few days have passed > >> From experience with moving xia2 & DIALS it was surprisingly >> straightforward (an opinion) > > Thanks & cheerio Graeme > > -----Original Message----- > From: cctbxbb-bounces(a)phenix-online.org > [mailto:[email protected]] On Behalf Of > markus.gerstel(a)diamond.ac.uk > Sent: 14 June 2016 12:18 > To: cctbxbb(a)phenix-online.org > Subject: Re: [cctbxbb] Not wishing to raise this again but... > > Hi Pavel, > > I volunteered to do this some time ago, just like for DIALS, and would > still be happy to do this. > We can do the move slowly, so allow users (developers) with a git > repository while development still happens in SVN for say a couple of > weeks, and I would be happy to run the git support for that time and > for the aftermath of the move. > > Best wishes > -Markus > > -----Original Message----- > From: cctbxbb-bounces(a)phenix-online.org > [mailto:[email protected]] On Behalf Of Pavel Afonine > Sent: 14 June 2016 12:11 > To: cctbx mailing list > Subject: Re: [cctbxbb] Not wishing to raise this again but... > > Hey there! > > Well I guess I say something even though I don't have much to say > really. I've been contributing to CCTBX for the past 13 years, so I > feel like this conversation is not too irrelevant to me to say the > least. > > I don't have strong opinion about staying with svn vs moving to git. I > routinely contribute to CCTBX (which is SVN-based) and I routinely > contribute to my other activity - Q|R (Quantum Refinement, which is > hosted by GitHub). I'm comfortable with both environments as I'm using > only the minimal functionality of both: check in, check out, diff.. > period. I'm not interested in branching/etc sophistication; this is > just because the way I structure my workflow.. > > My current understanding is that moving CCTBX to use git boils down to: > - actually someone doing it; AND > - taking good proper responsible care of the aftermath (if any). > > So far no-one volunteered to embrace this challenge, as far as I know. > > In summary I think we should move to git, at least to follow the > vector of progress. > > All the best, > Pavel >

3 2

bootstrap.py build on Ubuntu
by David Waterman 14 Jun '16

14 Jun '16

Hi folks I recently tried building cctbx+dials on Ubuntu 15.04 following the instructions here: http://dials.github.io/documentation/installation_developer.html This failed during installation of pango-1.16.1. Looking at pango_install_log, I see the command that failed was as follows: gcc -DHAVE_CONFIG_H -I. -I. -I../.. -DSYSCONFDIR=\"/home/fcx32934/sw/dials_bootstrap_test/base/etc\" -DLIBDIR=\"/home/fcx32934/sw/dials_bootstrap_test/base/lib\" -DG_DISABLE_CAST_CHECKS -I../.. -DG_DISABLE_DEPRECATED -I/home/fcx32934/sw/dials_bootstrap_test/base/include -I/home/fcx32934/sw/dials_bootstrap_test/base/include/freetype2 -g -O2 -Wall -MT fribidi.lo -MD -MP -MF .deps/fribidi.Tpo -c fribidi.c -fPIC -DPIC -o .libs/fribidi.o In file included from fribidi.h:31:0, from fribidi.c:28: fribidi_config.h:1:18: fatal error: glib.h: No such file or directory The file glib.h appears to be in base/include/glib-2.0/, however this directory was not explicitly included in the command above, only its parent. This suggests a configuration failure in pango to me. Taking a look at base_tmp/pango-1.16.1/config.log, I see what look like the relevant lines: configure:22227: checking for GLIB configure:22235: $PKG_CONFIG --exists --print-errors "$GLIB_MODULES" configure:22238: $? = 0 configure:22253: $PKG_CONFIG --exists --print-errors "$GLIB_MODULES" configure:22256: $? = 0 configure:22304: result: yes but this doesn't tell me very much. Does anyone have any suggestions as to how I might proceed? Many thanks -- David

3 10

Nightly test server down?
by Luc Bourhis 14 Jun '16

14 Jun '16

Hi, http://cci-vm-6.lbl.gov:8010/waterfall does not answer anymore from where I stand (Paris). Best wishes, Luc

3 2

flex.histogram: add weights?
by Graeme.Winter＠diamond.ac.uk 14 Jun '16

14 Jun '16

Hi Folks I would like to be able to compute a weighted flex histogram i.e. instead of adding 1 for every element in array add weight[i] for array[i] instead (ie. generalized) Has anyone done this already? Don't want to break anything, will require the histogram itself being non-integral... Thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 3

Project Scientist positions for computational methods
by Nicholas Sauter 13 Jun '16

13 Jun '16

Two Project Scientist positions are available in my group in the *Molecular Biophysics and Integrated Bioimaging* <http://biosciences.lbl.gov/divisions/mbib/> (MBIB) Division at Lawrence Berkeley National Lab. *Bioimaging Project Scientist* Position Summary: We have an exciting opportunity for a Bioimaging Project Scientist who will apply neuromorphic computing techniques to structural biology problems in X-ray diffraction, CryoEM, and CryoET. This will be a cross-disciplinary effort between MBIB and the Computational Research (CRD) Division at Lawrence Berkeley National Laboratory. The research group will include investigators Nick Sauter (X-ray Crystallography), Karen Davies (CryoEM/ET), and Chao Yang (Scalable Solvers). Specific Responsibilities: - Investigate how to apply deep learning algorithms to specific data processing and data interpretation problems in structural biology. - Implement convolutional neural network (CNN) code on neuromorphic computing hardware such as the IBM TrueNorth chip. Example problems include the identification of positive diffraction events in X-ray free-electron laser diffraction, conformational classification in CryoEM single particle reconstruction, and the identification of 3D sections for CryoET subtomogram averaging. - Use tools such as MatConvNet, CAFFE, THEANO or TensorFlow to construct and train CNNs that can be used to optimally classify experimental data. Required Qualifications: - Research experience in Structural Biology, with expertise in allied fields such as Bioinformatics, Mathematics, Computer Science, or Engineering (Ph.D. + 3 years or equivalent) - Strong background in image analysis and machine learning, including neural networks. - Working knowledge of image classification and/or other image processing techniques. - Proficient programming skills in MATLAB and/or Python. - Working knowledge of performance optimization for scientific codes. *Algorithm Developer Project Scientist* Position Summary: We are seeking an algorithm developer for the increasingly complex analysis of large diffraction datasets in structural biology. Current projects utilize XFEL crystallography and spectroscopy to investigate the photosynthetic mechanism of water splitting and to probe other metalloenzyme reactions. We also wish to test whether diffuse scattering can reveal correlated atomic motions in crystals. Many problems remain to be solved, including the details of how to optimally merge datasets from thousands of crystals. Our software development projects (including packages such as DIALS <http://dials.lbl.gov/> and cctbx.xfel <http://cci.lbl.gov/xfel>) have been highlighted in several high-impact publications listed HERE <http://biosciences.lbl.gov/profiles/nicholas-sauter>. Specific Responsibilities: - Continue work on the DIALS data integration suite. - Test new algorithms for optimally characterizing and merging data from serial crystallography experiments. - Analyze anomalous scattering from metalloenzymes using XFEL diffraction. - Investigate whether diffuse scattering can reveal correlated atomic motions in crystals. Required Qualifications: - Expertise in one or more computational techniques including, but not limited to, ray tracing to test underlying physical models of the diffraction, Bayesian approaches for refining model parameters, macromolecular modeling and refinement, neural networks for interpreting image features, as well as signal processing and denoising methods. - Extensive data analysis experience in crystallography or from more general bioimaging backgrounds are welcome. - We particularly encourage strong mathematical intuition and a track record of bringing new ideas and tools to fruition, as evidenced through written publication and clear presentation. Candidates should send an expression of interest, CV, and list of three references to Nick Sauter, nksauter(a)lbl.gov*. *Further details are posted at https://lbl.taleo.net/careersection/2/jobdetail.ftl?lang=en&job=82554. Nicholas K. Sauter, Ph. D. Computer Staff Scientist, Molecular Biophysics and Integrated Bioimaging Division Lawrence Berkeley National Laboratory 1 Cyclotron Rd., Bldg. 33R0345 Berkeley, CA 94720

1 0

Not wishing to raise this again but...
by R. D. Oeffner 13 Jun '16

13 Jun '16

Switching to git for the cctbx would be very liberating for developing and testing code I believe. If bisecting git hash values seems cumbersome one can always tag each commit with the number of previous commits made to the branch used for the build. These numbers are sequential and obviously straightforward to use for bisecting. We do this currently for the phaser git repo: https://git.csx.cam.ac.uk/x/cimr-phaser/phaser.git Rob Date: Mon, 13 Jun 2016 21:24:58 +0000 From: <richard.gildea(a)diamond.ac.uk> To: <cctbxbb(a)phenix-online.org> Subject: Re: [cctbxbb] Not wishing to raise this again but... Message-ID: <1BB183A24316FD4DA62A5F8B952D4291CC715321@exchmbx03> Content-Type: text/plain; charset="windows-1252" At least according to Google Trends, svn has been on a downward trend since about 2009/10, whereas git is still firmly on an upward trend (and overtook svn in popularity long ago): <https://www.google.co.uk/trends/explore#q=git%20commit%2C%20svn%20commit&cm…> https://www.google.co.uk/trends/explore#q=git%20commit%2C%20svn%20commit&cm… <https://www.google.co.uk/trends/explore#q=git%20commit%2C%20svn%20commit&cm…><https://www.google.co.uk/trends/explore#q=git%20commit%2C%20svn%20commit&cm…><https://www.google.co.uk/trends/explore#q=git%20commit%2C%20svn%20commit&cm…> [cid:162fa876-cc25-6335-73af-714b0957d71e] I think that avoiding a move to git is simply delaying the inevitable (and if the cctbx didn't ever move with the times then we'd still be stuck with CVS). In my opinion, the potential benefits to using git far outweigh the (slightly) added complexity. <https://www.google.co.uk/trends/explore#q=git%20commit%2C%20svn%20commit&cm…><https://www.google.co.uk/trends/explore#q=git%20commit%2C%20svn%20commit&cm…>Cheers, Richard ________________________________ From: cctbxbb-bounces(a)phenix-online.org [cctbxbb-bounces(a)phenix-online.org] on behalf of Nicholas Sauter [nksauter(a)lbl.gov] Sent: 13 June 2016 22:05 To: cctbx mailing list Subject: Re: [cctbxbb] Not wishing to raise this again but... I'd vote against the switch to git. Simple things, like svn update [remote directory] seem too complicated. As for slightly involved tasks like code bisection to locate the past commit that broke the test, well....I've never been able to figure this one out so I am pretty lost trying to debug DIALS. I do not wish to make things difficult for the maintenance of cctbx.xfel. Nick Nicholas K. Sauter, Ph. D. Computer Staff Scientist, Molecular Biophysics and Integrated Bioimaging Division Lawrence Berkeley National Laboratory 1 Cyclotron Rd., Bldg. 33R0345 Berkeley, CA 94720 (510) 486-5713 On Mon, Jun 13, 2016 at 8:52 AM, Luc Bourhis <luc_j_bourhis(a)mac.com<mailto:[email protected]>> wrote: On 13 Jun 2016, at 17:09, Graeme.Winter(a)Diamond.ac.uk<mailto:[email protected]> wrote: ? is there any enthusiasm out there for moving cctbx to git? The more I work with git the more I wish this existed properly for cctbx. I?ve raised this a bunch of times and always it is ?not right now? however ?a good idea? ? this?ll be the last time I ask? Could we move to git, please? I note the bootstrap script for Windows now has a git dependency even! +1 ? and time to share some thoughts! It would be great if we could get the cctbx up and running by just cloning, configuring and building without needing to download anything else. That could be done by adding SCons, Boost, and ccp4io as submodules in the cctbx repository. And I reckon ccp4io_adaptbx should be added as a subtree since it is developed by the same people as the cctbx. Just an idea to think about as I haven?t not worked out the details at all. Best wishes, Luc _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org<mailto:[email protected]> http://phenix-online.org/mailman/listinfo/cctbxbb -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

XQuartz 2.7.9 issue for OS X
by Billy Poon 10 Jun '16

10 Jun '16

Hi all, There seems to be an issue with XQuartz 2.7.9 on OS X when trying to render an OpenGL window (e.g. Coot, PyMOL, Phenix GLViewer) over SSH using X forwarding (i.e. ssh -Y). You will get an error that looks like X Error of failed request: BadValue (integer parameter out of range for operation) Major opcode of failed request: 150 (GLX) Minor opcode of failed request: 3 (X_GLXCreateContext) It seems like indirect rendering of a GLX context does not work properly on 2.7.9. Downgrading to XQuartz 2.7.8 works for me. -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

1 0

Who is using bootstrap.py on Windows?
by Robert Oeffner 10 Jun '16

10 Jun '16

Hi all, I would like to know if there are teams other than Phenix and Dials-west in Berkeley doing regular builds on Windows using cctbx_project/libtbx/auto_build/bootstrap.py. I plan to make some simplifications of this script using the ssh client that is part of Git for Windows rather than the one found in the PuTTY suite which we use currently. Many thanks, Robert

2 1

Berkeley Server Downtime (6/3/2016)
by Billy Poon 01 Jun '16

01 Jun '16

Dear all, On Friday, June 3, our servers will be shutdown starting at 6:30 am Pacific for scheduled maintenance in our building (7 - 8 am). This will affect, - The Phenix web site (http://phenix-online.org/) - The DIALS West web site (http://dials.lbl.gov/) - The CCI web site (http://cci.lbl.gov/) - The Structural Biology Technology Portal (http://technology.sbkb.org/) We expect that the servers will be available by 8:30 am Pacific at the latest. Bug reports and emails requesting help should be queued and we will receive them once the servers are back online. We apologize for any inconvenience. -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

1 0

The curious incident of the test at install time
by R. D. Oeffner 13 May '16

13 May '16

Hi Markus, Those changes were made by me and Oleg. I just tested a fresh installation of phenix-dev-2407-intel-linux-2.6-centos5 which is built with cctbx_project revision 24442. The test of cctbx_project/cctbx/regression/tst_grm_pickling.py passed without problems. I'm not aware of why this test is failing on dials after installation. Since the test is to do with the phenix restraints manager perhaps this test is not critical to dials. Perhaps the phenix restraints manager code together with this test should be moved into the phenix repository but that is not my call. Rob -----Original Message----- From: cctbxbb-request(a)phenix-online.org Sent: Friday, May 13, 2016 20:00 To: cctbxbb(a)phenix-online.org Subject: cctbxbb Digest, Vol 86, Issue 7 Message: 2 Date: Fri, 13 May 2016 09:47:18 +0000 From: <markus.gerstel(a)diamond.ac.uk> To: <cctbxbb(a)phenix-online.org> Subject: [cctbxbb] The curious incident of the test at install time Message-ID: <415F7F573406BB498256C68158D45674612998A2(a)EXCHMBX01.fed.cclrc.ac.uk> Content-Type: text/plain; charset=iso-8859-1 Hi everyone, we have observed the test cctbx_project/cctbx/regression/tst_grm_pickling.py to be failing, however only on an installation installed from the installer. It appears to run successfully on our bootstrapped developer build. I presume that the test requires some external data that we do not distribute? The relevant change occurred between r24437 and r24441, and is almost certainly r24439, which enabled the failing bit of the test. Stacktrace below. -Markus $ libtbx.python /dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py Time pickling/unpickling: 0.0187, 0.0097 Traceback (most recent call last): File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py", line 198, in <module> exercise_all(sys.argv[1:]) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py", line 195, in exercise_all test_nucleic_acid(mon_lib_srv, ener_lib) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py", line 177, in test_nucleic_acid log=null_out()) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 5455, in run log=log) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 5005, in __init__ restraints_loading_flags=restraints_loading_flags, File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2956, in __init__ log=log, File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2343, in __init__ m_j=mm) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 1568, in get_lib_link return mon_lib_srv.link_link_id_dict["rna3p"] KeyError: 'rna3p' -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom ------------------------------ Message: 3 Date: Fri, 13 May 2016 09:50:03 +0000 From: <Graeme.Winter(a)diamond.ac.uk> To: <cctbxbb(a)phenix-online.org> Subject: Re: [cctbxbb] The curious incident of the test at install time Message-ID: <13999B6FC038EB44B067771E3CFBD091BC2E3E7A(a)EXCHMBX03.fed.cclrc.ac.uk> Content-Type: text/plain; charset="us-ascii" Markus I got this error this morning in the regular update / compile shuffle and it's on my to-nag list, also feel like I have seen this before... Cheerio G -----Original Message----- From: cctbxbb-bounces(a)phenix-online.org [mailto:[email protected]] On Behalf Of markus.gerstel(a)diamond.ac.uk Sent: 13 May 2016 10:47 To: cctbxbb(a)phenix-online.org Subject: [cctbxbb] The curious incident of the test at install time Hi everyone, we have observed the test cctbx_project/cctbx/regression/tst_grm_pickling.py to be failing, however only on an installation installed from the installer. It appears to run successfully on our bootstrapped developer build. I presume that the test requires some external data that we do not distribute? The relevant change occurred between r24437 and r24441, and is almost certainly r24439, which enabled the failing bit of the test. Stacktrace below. -Markus $ libtbx.python /dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py Time pickling/unpickling: 0.0187, 0.0097 Traceback (most recent call last): File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py", line 198, in <module> exercise_all(sys.argv[1:]) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py", line 195, in exercise_all test_nucleic_acid(mon_lib_srv, ener_lib) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py", line 177, in test_nucleic_acid log=null_out()) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 5455, in run log=log) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 5005, in __init__ restraints_loading_flags=restraints_loading_flags, File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2956, in __init__ log=log, File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2343, in __init__ m_j=mm) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 1568, in get_lib_link return mon_lib_srv.link_link_id_dict["rna3p"] KeyError: 'rna3p' -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org http://phenix-online.org/mailman/listinfo/cctbxbb ------------------------------ _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org http://phenix-online.org/mailman/listinfo/cctbxbb End of cctbxbb Digest, Vol 86, Issue 7 **************************************

1 0

The curious incident of the test at install time
by markus.gerstel＠diamond.ac.uk 13 May '16

13 May '16

Hi everyone, we have observed the test cctbx_project/cctbx/regression/tst_grm_pickling.py to be failing, however only on an installation installed from the installer. It appears to run successfully on our bootstrapped developer build. I presume that the test requires some external data that we do not distribute? The relevant change occurred between r24437 and r24441, and is almost certainly r24439, which enabled the failing bit of the test. Stacktrace below. -Markus $ libtbx.python /dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py Time pickling/unpickling: 0.0187, 0.0097 Traceback (most recent call last): File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py", line 198, in <module> exercise_all(sys.argv[1:]) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py", line 195, in exercise_all test_nucleic_acid(mon_lib_srv, ener_lib) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/cctbx/regression/tst_grm_pickling.py", line 177, in test_nucleic_acid log=null_out()) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 5455, in run log=log) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 5005, in __init__ restraints_loading_flags=restraints_loading_flags, File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2956, in __init__ log=log, File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2343, in __init__ m_j=mm) File "/dls/science/groups/scisoft/DIALS/CD/nightly/dials-dev20160513/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 1568, in get_lib_link return mon_lib_srv.link_link_id_dict["rna3p"] KeyError: 'rna3p' -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

easy_mp stack traces
by Graeme.Winter＠diamond.ac.uk 13 May '16

13 May '16

Dear Gabor, some time back Markus added the capability to retain stack traces through easy_mp execution. These were reverted which now means that by default the stack traces are lost :( I don't have a clear understanding what the exact design of the newly built scheduling/stacktrace system is, perhaps you could explain this since we’re in the dark here. We are however pretty certain that the corner cases that lead to you modifying the code do not apply to us with DIALS or xia2. Could you please advise how we can now enable the stacktrace to be preserved globally, in all cases, and for all dials.*/xia2.* programs? Just today this has come up twice, and is a support issue for us, see e.g. https://github.com/dials/dials/issues/139 where we are simply flying blind. Thanks in advance and best wishes, Graeme & xia2 / DIALS team -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

Installation with distutils?
by Luc Bourhis 09 May '16

09 May '16

Hi fellow cctbx developers, unless I am missing something, nobody has written a “setup.py” to install the cctbx with distutils in Python standard site-packages? If so, I am planning to contribute one as this would make it easier for us at Bruker. I am just asking in case I would duplicate existing work. Best wishes, Luc

3 4

problem when using Python to convert from numpy array to flex.int array
by Jun Aishima 03 May '16

03 May '16

I have used the bootstrap method to build cctbx with my own Python (in a virtual environment). I setup my virtual environment, then source $CCTBX/build/setpaths.sh (bash). I am currently trying to turn a detector image (uint16) read in with fabio, which results in a numpy ndarray, and then turn that into a scitbx flex array. Since I've had problems doing this in my system, I've written a little script that reproduces the problem: ********** import fabio file=fabio.open('testcrystal_1_001.img') # adsc detector image file.data[0] from scitbx.array_family import flex flex.int(file.data[3071]) # this is a 3072x3072 image. For some reason, row 0 causes no problems. ********** At the end of this email is the stack trace I get. Has anybody had similar issues? I first saw this when I was trying to convert the entire image into a flex array, but now I've seen it with only one row [3071], while still working for [0:1500], so it doesn't seem like a simple memory issue. Any help would be appreciated. Jun Aishima ********** show_stack(1): <stdin>(1) <module> libc backtrace (23 frames, most recent call last): /home/aishimaj/spotter/bin/python() [0x414199] /lib64/libc.so.6(__libc_start_main+0xfd) [0x366d21ed5d] /home/aishimaj/spotter/bin/python(Py_Main+0xb22) [0x4150b2] /home/aishimaj/spotter/bin/python(PyRun_AnyFileExFlags+0x6b) [0x4ce82b] /home/aishimaj/spotter/bin/python(PyRun_InteractiveLoopFlags+0x4e) [0x4ce01e] /home/aishimaj/spotter/bin/python(PyRun_InteractiveOneFlags+0x1ff) [0x4cdd9f] /home/aishimaj/spotter/bin/python(PyEval_EvalCode+0x32) [0x4ad362] /home/aishimaj/spotter/bin/python(PyEval_EvalCodeEx+0x877) [0x4ad257] /home/aishimaj/spotter/bin/python(PyEval_EvalFrameEx+0x6a0b) [0x4aa8bb] /home/aishimaj/spotter/bin/python(PyObject_Call+0x47) [0x419ba7] /home/aishimaj/spotter/bin/python() [0x46df9f] /home/aishimaj/spotter/bin/python() [0x47623f] /home/aishimaj/spotter/bin/python(PyObject_Call+0x47) [0x419ba7] /home/aishimaj/spotter/bin/python() [0x42296f] /home/aishimaj/spotter/bin/python(PyObject_Call+0x47) [0x419ba7] /beamline/phenix/bootstrap/build/lib/libboost_python.so(+0x2d088) [0x7f5561101088] /beamline/phenix/bootstrap/build/lib/libboost_python.so(boost::python::handle_exception_impl(boost::function0<void>)+0x43) [0x7f5561112013] /beamline/phenix/bootstrap/build/lib/libboost_python.so(+0x2fe08) [0x7f5561103e08] /beamline/phenix/bootstrap/build/lib/libboost_python.so(boost::python::objects::function::call(_object*, _object*) const+0xef) [0x7f55611031ff] /beamline/phenix/bootstrap/build/lib/scitbx_array_family_flex_ext.so(boost::python::objects::signature_py_function_impl<boost::python::detail::caller<scitbx::af::versa<int, scitbx::af::flex_grid<scitbx::af::small<long, 10ul> > >* (*)(boost::python::numeric::array const&), boost::python::detail::constructor_policy<boost::python::default_call_policies>, boost::mpl::vector2<scitbx::af::versa<int, scitbx::af::flex_grid<scitbx::af::small<long, 10ul> > >*, boost::python::numeric::array const&> >, boost::mpl::v_item<void, boost::mpl::v_item<boost::python::api::object, boost::mpl::v_mask<boost::mpl::vector2<scitbx::af::versa<int, scitbx::af::flex_grid<scitbx::af::small<long, 10ul> > >*, boost::python::numeric::array const&>, 1>, 1>, 1> >::operator()(_object*, _object*)+0x52) [0x7f555ed68452] /beamline/phenix/bootstrap/build/lib/scitbx_array_family_flex_ext.so(scitbx::af::boost_python::flex_int_from_numpy_array(boost::python::numeric::array const&)+0x29) [0x7f555f039f79] /beamline/phenix/bootstrap/build/lib/scitbx_array_family_flex_ext.so(scitbx::af::versa<int, scitbx::af::flex_grid<scitbx::af::small<long, 10ul> > > scitbx::af::boost_python::versa_flex_from_numpy_array<int>(boost::python::numeric::array const&)+0x1c87) [0x7f555f0415c7] /lib64/libc.so.6() [0x366d2326a0] Segmentation fault (Python and libc call stacks above) This crash may be due to a problem in any imported Python module, including modules which are not part of the cctbx project. To disable the traps leading to this message, define these environment variables (e.g. assign the value 1): BOOST_ADAPTBX_FPE_DEFAULT BOOST_ADAPTBX_SIGNALS_DEFAULT This will NOT solve the problem, just mask it, but may allow you to proceed in case it is not critical. Jun Aishima | Post Doc - MX | Australian Synchrotron p: (03) 8540 4114 | f: (03) 8540 4200 jun.aishima(a)synchrotron.org.au<mailto:[email protected]> | www.synchrotron.org.au 800 Blackburn Road, Clayton, Victoria 3168

3 6

Algorithm developer postdoc opening at Berkeley National Lab
by Nicholas Sauter 28 Apr '16

28 Apr '16

Berkeley Lab's Molecular Biophysics and Integrated Bioimaging Division <http://biosciences.lbl.gov/divisions/mbib/> is looking for a Computational Biologist Postdoctoral Fellow to work in the Sauter group. We are seeking an algorithm developer for the increasingly complex analysis of large diffraction datasets in structural biology. Current projects utilize XFEL crystallography and spectroscopy to investigate the photosynthetic mechanism of water splitting and to probe other metalloenzyme reactions. We also wish to test whether diffuse scattering can reveal correlated atomic motions in crystals. Many problems remain to be solved, including the details of how to optimally merge datasets from thousands of crystals. Our software development projects (including packages such as DIALS <http://dials.lbl.gov> and cctbx.xfel <http://cci.lbl.gov/xfel>) have been highlighted in several high-impact publications listed HERE <http://biosciences.lbl.gov/profiles/nicholas-sauter>. Candidates should have expertise in one or more computational techniques including, but not limited to, ray tracing to test underlying physical models of the diffraction, Bayesian approaches for refining model parameters, macromolecular modeling and refinement, neural networks for interpreting image features, as well as signal processing and denoising methods. Extensive data analysis experience in crystallography or from more general bioimaging backgrounds are welcome. We particularly encourage strong mathematical intuition and a track record of bringing new ideas and tools to fruition, as evidenced through written publication and clear presentation. Candidates should send an expression of interest, CV, and list of three references to *Nick Sauter, nksauter(a)lbl.gov <nksauter(a)lbl.gov>.* Further details are posted at https://lbl.taleo.net/careersection/2/jobdetail.ftl?lang=en&job=82354. Nicholas K. Sauter, Ph. D. Computer Staff Scientist, Molecular Biophysics and Integrated Bioimaging Division Lawrence Berkeley National Laboratory 1 Cyclotron Rd., Bldg. 33R0345 Berkeley, CA 94720

1 0

Re: [cctbxbb] new fail on windows
by R. D. Oeffner 25 Apr '16

25 Apr '16

Markus, I'd say that this is a wrong take on the current state of affairs regarding supporting Windows. I believe line 22 in easy_mp.py you refer to is supposed to be one of the last warnings one would get if one were to use some of the functions there with Windows in an unsupported manner, say if objects can't be pickled. I don't think it means this file was written with non-windows support intentionally. I appreciate people code on different platforms but the way of working shouldn't be aloof denial of the existence of Windows. In the short term until you get a Windows PC perhaps Billy Poon might grant you access to one of the virtual Windows systems he has set up for testing. Otherwise I'm happy to run tests for you every now and then on my PC. Regards, Rob -----Original Message----- From: cctbxbb-request(a)phenix-online.org Sent: Saturday, April 23, 2016 20:00 To: cctbxbb(a)phenix-online.org Subject: cctbxbb Digest, Vol 85, Issue 23 Send cctbxbb mailing list submissions to cctbxbb(a)phenix-online.org To subscribe or unsubscribe via the World Wide Web, visit http://phenix-online.org/mailman/listinfo/cctbxbb or, via email, send a message with subject or body 'help' to cctbxbb-request(a)phenix-online.org You can reach the person managing the list at cctbxbb-owner(a)phenix-online.org When replying, please edit your Subject line so it is more specific than "Re: Contents of cctbxbb digest..." Today's Topics: 1. Re: new fail on windows (markus.gerstel(a)diamond.ac.uk) ---------------------------------------------------------------------- Message: 1 Date: Fri, 22 Apr 2016 21:26:38 +0000 From: <markus.gerstel(a)diamond.ac.uk> To: <cctbxbb(a)phenix-online.org> Subject: Re: [cctbxbb] new fail on windows Message-ID: <415F7F573406BB498256C68158D456746125CE10(a)EXCHMBX01.fed.cclrc.ac.uk> Content-Type: text/plain; charset=iso-8859-1 Hi Nigel, Should be resolved now. It is true that we do not currently test on windows. In this particular case it turns out neither does anyone else, cf. https://sourceforge.net/p/cctbx/code/HEAD/tree/trunk/libtbx/easy_mp.py line 22 which causes the other easy_mp tests to be skipped on Windows. This will make it kind of difficult to support windows software building on top of it, so at some point this needs to be addressed properly. Best wishes & good weekend -Markus ________________________________ From: cctbxbb-bounces(a)phenix-online.org [cctbxbb-bounces(a)phenix-online.org] on behalf of Nigel Moriarty [nwmoriarty(a)lbl.gov] Sent: Friday, April 22, 2016 17:39 To: cctbx mailing list Subject: Re: [cctbxbb] new fail on windows On Fri, Apr 22, 2016 at 1:35 AM, <markus.gerstel(a)diamond.ac.uk<mailto:[email protected]>> wrote: Hopefully this is fixed with r24353. Afraid not. Is it true that you are not testing this windows feature on a windows machine? Regarding junit, I'll talk with Aaron when he returns. Cheers Nigel --- Nigel W. Moriarty Building 33R0349, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709 Email : NWMoriarty(a)LBL.gov Fax : 510-486-5909 Web : CCI.LBL.gov<http://CCI.LBL.gov> -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom ------------------------------ _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org http://phenix-online.org/mailman/listinfo/cctbxbb End of cctbxbb Digest, Vol 85, Issue 23 ***************************************

2 1

new fail on windows
by Nigel Moriarty 23 Apr '16

23 Apr '16

******************************************************************************** ERROR: 1 TEST FAILURES. PLEASE FIX BEFORE COMMITTING CODE. ******************************************************************************** Standard error: Traceback (most recent call last): File "c:\users\builder\slave\cctbx-nightly-intel-windows\modules\cctbx_project\libtbx\tst_easy_mp.py", line 210, in <module> run(args=sys.argv[1:]) File "c:\users\builder\slave\cctbx-nightly-intel-windows\modules\cctbx_project\libtbx\tst_easy_mp.py", line 205, in run check_if_stacktrace_is_propagated_properly(method='multiprocessing') File "c:\users\builder\slave\cctbx-nightly-intel-windows\modules\cctbx_project\libtbx\tst_easy_mp.py", line 187, in check_if_stacktrace_is_propagated_properly assert isinstance(e, exceptions.ZeroDivisionError) or "ZeroDivisionError" in str(e), "Exception type mismatch: %s" % str(e) AssertionError: Exception type mismatch: Can't pickle <function may_divide_by_zero at 0x01C06470>: it's not found as __main__.may_divide_by_zero Traceback (most recent call last): File "<string>", line 1, in <module> File "C:\Users\builder\slave\cctbx-nightly-intel-windows\base\bin\python\lib\multiprocessing\forking.py", line 381, in main self = load(from_parent) File "C:\Users\builder\slave\cctbx-nightly-intel-windows\base\bin\python\lib\pickle.py", line 1378, in load return Unpickler(file).load() File "C:\Users\builder\slave\cctbx-nightly-intel-windows\base\bin\python\lib\pickle.py", line 858, in load dispatch[key](self) File "C:\Users\builder\slave\cctbx-nightly-intel-windows\base\bin\python\lib\pickle.py", line 880, in load_eof raise EOFError EOFError Cheers Nigel --- Nigel W. Moriarty Building 33R0349, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709 Email : NWMoriarty(a)LBL.gov Fax : 510-486-5909 Web : CCI.LBL.gov

3 4

Re: [cctbxbb] cctbxbb Digest, Vol 85, Issue 21
by R.D. Oeffner 22 Apr '16

22 Apr '16

Hi Markus, I was gonna ask why don't you guys set up a windows machine and test for the cctbx crash on that? The VC++9.0 compiler is free and if you use the python that comes in a phenix for windows installation the necessary modules are all present. Rob On 22/04/2016 09:35, cctbxbb-request(a)phenix-online.org wrote: > Send cctbxbb mailing list submissions to > cctbxbb(a)phenix-online.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://phenix-online.org/mailman/listinfo/cctbxbb > or, via email, send a message with subject or body 'help' to > cctbxbb-request(a)phenix-online.org > > You can reach the person managing the list at > cctbxbb-owner(a)phenix-online.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of cctbxbb digest..." > > > Today's Topics: > > 1. Re: new fail on windows (markus.gerstel(a)diamond.ac.uk) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 22 Apr 2016 08:35:42 +0000 > From: <markus.gerstel(a)diamond.ac.uk> > To: <cctbxbb(a)phenix-online.org> > Subject: Re: [cctbxbb] new fail on windows > Message-ID: > <415F7F573406BB498256C68158D456746125C23B(a)EXCHMBX01.fed.cclrc.ac.uk> > Content-Type: text/plain; charset="utf-8" > > Hi Nigel, > > Hopefully this is fixed with r24353. > > One question: Do you think it would be possible to install > https://pypi.python.org/pypi/junit-xml on your build machines, run > libtbx.run_tests_parallel with the output_junit_xml=True option, and > collect the output.xml files somewhere where we can download them? > > At the moment we do monitor the buildbot status - but only in a very > coarse way. We only see test failures on a high level, see attached > screenshot. As you can see there are a number of things broken, some > of them for quite some time, so because on that level there is always > something guaranteed to fail the overall status we see is always set > to ?warning?. I don?t think any of us bothers anymore looking into the > details, a number of errors are about things we can?t do anything > about anyway. > > However, if you could give us a collection of output.xml?s we could, > instead of taking the overall build status, use the test results and > set up a DIALS-East internal mail on the buildbot test status. Then we > would pick up these things earlier. > > Best wishes, > -Markus > > From: cctbxbb-bounces(a)phenix-online.org > [mailto:[email protected]] On Behalf Of Nigel Moriarty > Sent: 21 April 2016 17:41 > To: cctbx mailing list > Subject: [cctbxbb] new fail on windows > > > ******************************************************************************** > > ERROR: 1 TEST FAILURES. PLEASE FIX BEFORE COMMITTING CODE. > > ******************************************************************************** > > > > Standard error: > > Traceback (most recent call last): > > File > "c:\users\builder\slave\cctbx-nightly-intel-windows\modules\cctbx_project\libtbx\tst_easy_mp.py", > line 210, in <module> > > run(args=sys.argv[1:]) > > File > "c:\users\builder\slave\cctbx-nightly-intel-windows\modules\cctbx_project\libtbx\tst_easy_mp.py", > line 205, in run > > > check_if_stacktrace_is_propagated_properly(method='multiprocessing') > > File > "c:\users\builder\slave\cctbx-nightly-intel-windows\modules\cctbx_project\libtbx\tst_easy_mp.py", > line 187, in check_if_stacktrace_is_propagated_properly > > assert isinstance(e, exceptions.ZeroDivisionError) or > "ZeroDivisionError" in str(e), "Exception type mismatch: %s" % str(e) > > AssertionError: Exception type mismatch: Can't pickle <function > may_divide_by_zero at 0x01C06470>: it's not found as > __main__.may_divide_by_zero > > Traceback (most recent call last): > > File "<string>", line 1, in <module> > > File > "C:\Users\builder\slave\cctbx-nightly-intel-windows\base\bin\python\lib\multiprocessing\forking.py", > line 381, in main > > self = load(from_parent) > > File > "C:\Users\builder\slave\cctbx-nightly-intel-windows\base\bin\python\lib\pickle.py", > line 1378, in load > > return Unpickler(file).load() > > File > "C:\Users\builder\slave\cctbx-nightly-intel-windows\base\bin\python\lib\pickle.py", > line 858, in load > > dispatch[key](self) > > File > "C:\Users\builder\slave\cctbx-nightly-intel-windows\base\bin\python\lib\pickle.py", > line 880, in load_eof > > raise EOFError > > EOFError > Cheers > > Nigel > > --- > Nigel W. Moriarty > Building 33R0349, Molecular Biophysics and Integrated Bioimaging > Lawrence Berkeley National Laboratory > Berkeley, CA 94720-8235 > Phone : 510-486-5709 Email : > NWMoriarty(a)LBL.gov<mailto:[email protected]> > Fax : 510-486-5909 Web : CCI.LBL.gov<http://CCI.LBL.gov> > > -- > This e-mail and any attachments may contain confidential, copyright > and or privileged material, and are for the use of the intended > addressee only. If you are not the intended addressee or an authorised > recipient of the addressee please notify us of receipt by returning > the e-mail and do not use, copy, retain, distribute or disclose the > information in or attached to the e-mail. > Any opinions expressed within this e-mail are those of the individual > and not necessarily of Diamond Light Source Ltd. > Diamond Light Source Ltd. cannot guarantee that this e-mail or any > attachments are free from viruses and we cannot accept liability for > any damage which you may sustain as a result of software viruses which > may be transmitted in or with the message. > Diamond Light Source Limited (company no. 4375679). Registered in > England and Wales with its registered office at Diamond House, Harwell > Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United > Kingdom >

2 1

Trapping NaN or not trapping NaN
by Luc Bourhis 13 Apr '16

13 Apr '16

Richard and I have been hammering at our build system and code to try to solve several crashes on OS X with Xcode 7.3. Several of them came from the fact that we trap NaN whereas Xcode 7.3 aggressive optimisations produce code that may create a NaN but still deliver the correct final result. A trivial example came from cctbx/xray/conversions.h. We have if (f_sq > 0) f = std::sqrt(f_sq); else f = 0; With the version of clang shipping with Xcode 7.3, both branches are evaluated in parallel before selecting the result based on the test f_sq > 0. This is done so as not to stall the pipeline and this can result in much faster code indeed. But that means that every time f_sq < 0, a NaN will be produced, and in the context of our tests, this means we crash. Richard and I avoided the elephant in the room, stop to trap NaN, and as a result came with uglier and uglier fixes. Moreover, we are now at the mercy of the compiler applying such an optimisation to another file at the next update of Xcode, which would then force us to fix the compilation of said file in the relevant SConscript. It’s a loosing game. Thus I reckon that at least on OS X, we should stop trapping NaN. That will leave us with the true clang regression for which I will try to submit reduced cases to the clang bugzilla. Best wishes, Luc

5 7

Boost broken on mac?
by markus.gerstel＠diamond.ac.uk 12 Apr '16

12 Apr '16

Dials west is currently observing test failures on Mac related to Boost. To exclude any errors on our side I have run things twice clean from bootstrap: One of a number of affected tests: libtbx.python cctbx/xray/boost_python/tst_xray.py show_stack(1): /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/cctbx_project/cctbx/xray/boost_python/tst_xray.py(381) exercise_conversions show_stack(2): /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/cctbx_project/cctbx/xray/boost_python/tst_xray.py(1993) run show_stack(3): /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/cctbx_project/cctbx/xray/boost_python/tst_xray.py(2009) <module> libc backtrace (29 frames, most recent call last): 30 Python 0x0000000100000d71 Python + 3441 29 Python 0x0000000100000e58 Python + 3672 28 Python 0x00000001000ef06f Py_Main + 3135 27 Python 0x00000001000d78be PyRun_SimpleFileExFlags + 702 26 Python 0x00000001000d7d84 PyRun_FileExFlags + 164 25 Python 0x00000001000aea06 PyEval_EvalCode + 54 24 Python 0x00000001000af244 PyEval_EvalCodeEx + 2100 23 Python 0x00000001000b25c8 PyEval_EvalFrameEx + 12280 22 Python 0x00000001000b6ab6 fast_function + 118 21 Python 0x00000001000af244 PyEval_EvalCodeEx + 2100 20 Python 0x00000001000b25c8 PyEval_EvalFrameEx + 12280 19 Python 0x00000001000b6ab6 fast_function + 118 18 Python 0x00000001000af244 PyEval_EvalCodeEx + 2100 17 Python 0x00000001000b4198 PyEval_EvalFrameEx + 19400 16 Python 0x0000000100011023 PyObject_Call + 99 15 Python 0x000000010006ad8b type_call + 347 14 Python 0x00000001000708af slot_tp_init + 175 13 Python 0x0000000100011023 PyObject_Call + 99 12 Python 0x000000010001f0d8 instancemethod_call + 232 11 Python 0x0000000100011023 PyObject_Call + 99 10 libboost_python.dylib 0x0000000101888663 boost::python::objects::function_call(_object*, _object*, _object*) + 83 9 libboost_python.dylib 0x000000010188ce71 boost::python::handle_exception_impl(boost::function0<void>) + 81 8 libboost_python.dylib 0x00000001018889ca boost::detail::function::void_function_ref_invoker0<boost::python::objects::(anonymous namespace)::bind_return, void>::invoke(boost::detail::function::function_buffer&) + 26 7 libboost_python.dylib 0x000000010188693a boost::python::objects::function::call(_object*, _object*) const + 874 6 cctbx_xray_ext.so 0x0000000107c8e1c5 boost::python::detail::caller_arity<3u>::impl<void (*)(_object*, scitbx::af::const_ref<double, scitbx::af::trivial_accessor> const&, scitbx::af::const_ref<double, scitbx::af::trivial_accessor> const&), boost::python::default_call_policies, boost::mpl::vector4<void, _object*, scitbx::af::const_ref<double, scitbx::af::trivial_accessor> const&, scitbx::af::const_ref<double, scitbx::af::trivial_accessor> const&> >::operator()(_object*, _object*) + 229 5 cctbx_xray_ext.so 0x0000000107c91f44 boost::python::objects::make_holder<2>::apply<boost::python::objects::value_holder<cctbx::xray::array_f_sq_as_f<cctbx::xray::f_sq_as_f_xtal_3_7, double> >, boost::mpl::joint_view<boost::python::detail::drop1<boost::python::detail::type_list<scitbx::af::const_ref<double, scitbx::af::trivial_accessor> const&, scitbx::af::const_ref<double, scitbx::af::trivial_accessor> const&, boost::python::optional<double const&, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_>, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_> >, boost::python::optional<double const&, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_> > >::execute(_object*, scitbx::af::const_ref<double, scitbx::af::trivial_accessor> const&, scitbx::af::const_ref<double, scitbx::af::trivial_accessor> const&) + 100 4 ??? 0x0000000000000002 0x0 + 2 3 libsystem_platform.dylib 0x00007fff8caabeaa _sigtramp + 26 2 boost_python_meta_ext.so 0x0000000101841370 initboost_python_meta_ext + 0 Floating-point error (Python and libc call stacks above) This crash may be due to a problem in any imported Python module, including modules which are not part of the cctbx project. To disable the traps leading to this message, define these environment variables (e.g. assign the value 1): BOOST_ADAPTBX_FPE_DEFAULT BOOST_ADAPTBX_SIGNALS_DEFAULT This will NOT solve the problem, just mask it, but may allow you to proceed in case it is not critical. Any ideas? Best wishes, -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

6 35

cctbx can't build without iotbx
by Luc Bourhis 06 Apr '16

06 Apr '16

Hi all, am I getting mad or % libtbx.configure —only cctbx % libtbx.scons will result in a linking error because of the line marked with <<<<< in cctbx/maptbx/SConscript ? if (not env_etc.no_boost_python): Import("env_cctbx_boost_python_ext") env_bpl = env_cctbx_boost_python_ext.Clone() env_etc.include_registry.append( env=env_bpl, paths=env_etc.cctbx_common_includes) env_bpl.Prepend(LIBS=["asymmetric_map", "iotbx_xplor", "cctbx_sgtbx_asu”, <<<<<<<<<<<<<<<<<<<<<<<<<<<<< "cctbx", "omptbx"]) env_bpl.SharedLibrary( target="#lib/cctbx_asymmetric_map_ext", source=["asymmetric_map_ext.cpp"]) This code was introduced in Dec 2013!!! The only workaround % libtbx.configure cctbx iotbx So much for the modularisation with the libtbx_config’s… Best wishes, Luc

2 1

Re: [cctbxbb] sequential revision number from git repo
by markus.gerstel＠diamond.ac.uk 06 Apr '16

06 Apr '16

Hi Johan, No, not really. .gitversion is not in the repository and only kept (and updated) locally. Basically the logic is: Can I determine the current version number (ie do I have git installed and the repository present) - if so, write .gitversion. Otherwise, attempt to read .gitversion and use the information therein (this covers the route via the installer). Hence, if you download the zip file frm github you will not have a .gitversion file. But really in this case we can assume that the user knows what they're doing. -Markus On 5 Apr 2016 16:34, "Hattne, Johan" <hattnej(a)janelia.hhmi.org> wrote: Would this not mean that .gitversion is updated on every commit, at least if as in our case, it not only records the release version but also the current commit? @Rob: Apologies for my liberal use of “hook”. What our build system does is that it generates what essentially amounts to Markus’s .gitversion whenever we run “make dist” (using “git describe” or some such, it need not involve the network). There is little need for this file when compiling from a sources under git control, because it can be done on the fly. // Cheers; Johan > On Apr 5, 2016, at 09:47, markus.gerstel(a)diamond.ac.uk wrote: > > Hi Johan, > > We generate a file named '.gitversion' that lives in the main directory. That way the version number survives being packed up in an installer. > If you download the code via the "Download ZIP" button on Github there is no way (as far as I know) to recover any version information, so we fall back to a default of "DIALS 1.dev" ("DIALS 1.1" for the release branch) > > -Markus > > -----Original Message----- > From: cctbxbb-bounces(a)phenix-online.org [mailto:[email protected]] On Behalf Of Hattne, Johan > Sent: 05 April 2016 14:21 > To: cctbx mailing list > Subject: Re: [cctbxbb] sequential revision number from git repo > > Hi Markus; > > I didn’t actually read the code, but it looks like we’re doing something similar over here, and I find the the tricky bit is how to deal with any kind of releases that don’t have all the little git bits. We sort that out by having hooks in the “make dist” equivalent, which basically generate a version file that is included in the release tarball. > > However, it was recently pointed out to us that that all falls apart if someone uses the "Download ZIP” button from the code tab, because then the hooks are not run. I haven’t even attempted to RTFM around this, and if you or anybody else here has a solution I won’t have to. > > Do you by any chance? > > // Cheers; Johan > > >> On Apr 5, 2016, at 08:55, markus.gerstel(a)diamond.ac.uk wrote: >> >> Hi Rob, >> >> This is close to what we do with DIALS. >> If you run dials.version you may get e.g. >> DIALS 1.dev.132-g7042805 >> So this is the 132nd commit after the commit we tagged "1.dev", which >> happened when we branched for the 1.1 stable release so we could keep >> the version numbers sufficiently different >> >> If you run DIALS 1.1, for example the version from CCP4, you might get >> DIALS 1.1.3-gb5528a5-release The 3rd commit on the 1.1 release branch. >> >> Same with xia2: >> XIA2 0.4.0.301-g7f16ab6 >> >> In xia2 we also add the branch name if it is not 'master', so the version distributed with DIALS 1.1 is: >> XIA2 0.4.0.289-g2a1e557-dials-1.1 >> >> We kept the commit ID in - and I would recommend to keep it with cctbx, too. >> The reason for this is: >> - The number on its own does not uniquely identify a commit, as you >> might be on a different branch (although we do put in the branch >> names, too) >> - Because then I can go to >> https://github.com/dials/dials/commit/b5528a5 and see everything about >> that commit >> - Because then I can do git checkout b55528a5 and instantly have that particular code version on my machine to fiddle with things. >> >> To see the magic happen, check >> https://github.com/dials/dials/blob/master/util/version.py or >> https://github.com/xia2/xia2/blob/master/XIA2Version.py >> >> -Markus >> >> >> -----Original Message----- >> From: cctbxbb-bounces(a)phenix-online.org >> [mailto:[email protected]] On Behalf Of R.D. Oeffner >> Sent: 05 April 2016 13:38 >> To: cctbxbb(a)phenix-online.org >> Subject: [cctbxbb] sequential revision number from git repo >> >> Hi, >> >> We are slowly moving phaser svn to git. One issue that we would like to address is if there is a practical way of conveying a linear commit history from the git repo. >> >> One approach is to just count the number of revisions that was made to the master branch in the git repo with "git rev-list --count HEAD" >> and then pretend that number is a revision number. This enables us to tell users "This bug was fixed after revision 6456 of phaser" rather than "This bug was fixed after commit 29b0a1c... of phaser". Is there any pitfalls with this approach? >> >> What workarounds do other people using git come up with? >> >> >> >> Rob >> >> >> -- >> Robert Oeffner, Ph.D. >> Research Associate, The Read Group >> Department of Haematology, >> Cambridge Institute for Medical Research University of Cambridge >> Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road >> Cambridge CB2 0XY >> >> www.cimr.cam.ac.uk/investigators/read/index.html<http://www.cimr.cam.ac.uk/investigators/read/index.html> >> tel: +44(0)1223 763234 >> mobile: +44(0)7712 887162 >> _______________________________________________ >> cctbxbb mailing list >> cctbxbb(a)phenix-online.org >> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> -- >> This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. >> Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. >> Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. >> Diamond Light Source Limited (company no. 4375679). Registered in >> England and Wales with its registered office at Diamond House, Harwell >> Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United >> Kingdom >> >> >> _______________________________________________ >> cctbxbb mailing list >> cctbxbb(a)phenix-online.org >> http://phenix-online.org/mailman/listinfo/cctbxbb > > Research Specialist @ Gonen Lab ____________________________________________________ > Janelia Research Campus * 19700 Helix Drive Ashburn, VA 20147 * +1 (571) 209-4000 extension 3376 > > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org > http://phenix-online.org/mailman/listinfo/cctbxbb > > -- > This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. > Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. > Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. > Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org > http://phenix-online.org/mailman/listinfo/cctbxbb Research Specialist @ Gonen Lab ____________________________________________________ Janelia Research Campus * 19700 Helix Drive Ashburn, VA 20147 * +1 (571) 209-4000 extension 3376 _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org http://phenix-online.org/mailman/listinfo/cctbxbb

3 3

CCTBX.XFEL problems with synchrotron stills
by Schulz, Eike-Christian 06 Apr '16

06 Apr '16

Dear all, I am new to serial crystallography and trying to process still images from a synchrotron source (Pilatus 6M detector) using CCTBX. Obviously the process is not glitch-free and I was hoping to get some help here. The indexing routine starts, indexes the images correctly and and produces a log file but no .pickle files are created in the results/000/ directory as described in the wiki entry. I assume that there is a software module that is not properly loaded during the process since I get following error message during indexing: ---------BEGIN Integrate one frame 75 warsaw0048_00075.pickle EFFECTIVE TILING 0 0 2282 2282 Indexing error: No module named cxi_user Is there a way to find out if all dependencies are matched ? I suppose CCTBX.XFEL depends on some (specific) python libraries … ? Or am I completely wrong and the error is something different ? With best regards Eike

3 4

sequential revision number from git repo
by R.D. Oeffner 05 Apr '16

05 Apr '16

Hi, We are slowly moving phaser svn to git. One issue that we would like to address is if there is a practical way of conveying a linear commit history from the git repo. One approach is to just count the number of revisions that was made to the master branch in the git repo with "git rev-list --count HEAD" and then pretend that number is a revision number. This enables us to tell users "This bug was fixed after revision 6456 of phaser" rather than "This bug was fixed after commit 29b0a1c... of phaser". Is there any pitfalls with this approach? What workarounds do other people using git come up with? Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234 mobile: +44(0)7712 887162

3 6

changes to cctbx we have in ccp4
by Marcin Wojdyr 04 Apr '16

04 Apr '16

Hi All, we have a bunch of changes to cctbx in ccp4. If any of them are of general interest I can push them to svn. They'd need some cleanup before applying, so I'm not attaching any patches for now. - option to use external Boost. It's the same what debian and gentoo distros are doing when packaging cctbx. We change a few lines in the scons scripts and rm cctbx_project/boost/* (but the boost directory itself must be present). We could add an option for it (--use-system-boost ?). - removing -funroll-loops - this option must be a historical leftover. Generally it's better to let compiler decide because unrolling loops may make the program slower as well as faster. I tested it on Phaser a couple years ago. The only noticeable effect of this flag was much bigger binary. - extend --use-environment-flags to handle also CC and CXX. With most of makefiles / build scripts, if the user has multiple compilers installed he selects one with CC=gcc-mp-5 etc. Then if the build script really needs to differentiate between compilers it tries to determine what kind of compiler it is. In cctbx it's the other way around - the --compiler flag tells what kind of compiler it is and the build script guesses executable names. For ccp4 I added in two places in libtbx/SConscript: + if libtbx.env.build_options.use_environment_flags: + if 'CC' in os.environ: cc = os.environ['CC'] + if 'CXX' in os.environ: cxx = os.environ['CXX'] so for _some_ platform/compiler combinations it's possible to use compilers with non-default names. Useful for systems with multiple GCC versions. - probe and reduce are called in mmtbx through phenix wrappers (phenix.probe and phenix.reduce). We prefer to call them directly, but I guess it's not to be changed in cctbx. But after changing the name I also had to remove error checking in mmtbx/validation/clashscore.py that checks return code from phenix.reduce. Reduce itself quite often returns non-zero, it's just that the phenix.reduce wrapper ignores it and always returns 0 (probably not intentionally). - cctbx_project/spotfinder/SConscript - on Windows we have to link dxtbxspotfinder with spotfinder. That's the only such problem when compiling cctbx. OTOH it apparently works for everyone else. I don't know why. It must have something to do with us using MinGW. - A couple of #ifdefs for compiling with MinGW (annlib and gltbx) Marcin

2 2

Berkeley Server Downtime (3/30/2016)
by Billy Poon 30 Mar '16

30 Mar '16

Dear all, On Wednesday, March 30, our servers will be shutdown starting at 7:00 am Pacific for scheduled maintenance in our server room. This will affect, - The Phenix web site (http://phenix-online.org/) - The DIALS West web site (http://dials.lbl.gov/) - The CCI web site (http://cci.lbl.gov/) - The Structural Biology Technology Portal (http://technology.sbkb.org/) We expect that the servers will be available by 5 pm Pacific at the latest. Bug reports and emails requesting help should be queued and we will receive them once the servers are back online. We apologize for any inconvenience. -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

1 1

new parallel_map implementation
by Gabor Bunkoczi 29 Mar '16

29 Mar '16

Hi, I have switched over libtbx.easy_mp.parallel_map to use new code that is not susceptible to deadlock when large amounts of data are returned from a worker process. This also means that parameters added to parallel_map to prevent exactly this are no longer necessary (but will be respected), although unlikely to influence performance so much to warrant changing existing code. I have run all tests, but unsure how well this gets tested by regression test, so please let me know if you come across something that is amiss. BW, Gabor

1 0

Troublesome Hall Symbols in CIF
by Luc Bourhis 21 Mar '16

21 Mar '16

Hi, Consider the following two Hall symbols: (1) -C 2y (x,y,-x+z) (2) -I 2y They represent the same setting of the same spacegroup, so in a perfect world the fact that the sgtbx comes with (1) from some list of operators should not matter. Unfortunately, (1) is rejected by CheckCIF right now. Reminder: every single small molecule publication in Acta Cryst. must pass CheckCIF. On the contrary, CheckCIF accepts (2). Yes, I know, this is a bug in CheckCIF. But the IUCR is very slow at fixing bug in that kerberos of a program and we do therefore put our Olex2 users in trouble when they use the smtbx to refine their structure, as opposed to ShelXL, which does output (2). Besides, (2) is clearly more pleasing to the human mind than (1). Thus I reckon there is definitively a case to make a change in the cctbx here. The question is at which level? I don’t think we want to touch the sgtbx. I am of the opinion of patching iotbx.cif. Would that cause any trouble in Phenix and consort? Any remark or suggestion? Best wishes, Luc

3 2

New top-level module: fast_linalg
by Luc Bourhis 07 Mar '16

07 Mar '16

Hi, this is an advanced warning for the introduction of a new top-level module named fast_linalg. In a couple of days, I will push it in one go as 25 commits but don’t be scared, nothing is changed outside the top directory fast_linalg, which also means it is not configured by default. Thus there cannot be any disruption but I thought it would be polite to warn all cctbx developers. The module features the automatic installation and integration of OpenBLAS which I have used to speed up smtbx refinement. I will commit that work later. Best wishes, Luc J. Bourhis

2 2

Dead link for old builds on cctbx_build page
by Luc Bourhis 06 Mar '16

06 Mar '16

Hi, on http://cci.lbl.gov/cctbx_build/, in the phrase “ Old code bundles prior to 2015 may be accessed here.”, the word “here" is a link toward http://cci.lbl.gov/cctbx_legacy_build but this is a dead link. Is this a typo or are those builds truly gone? Best wishes, Luc

2 1

libtbx.easy_mp semantics
by markus.gerstel＠diamond.ac.uk 23 Feb '16

23 Feb '16

Hello everyone, Could someone please explain the libtbx.easy_mp semantics to me? Currently, if you run easy_mp with multiple processes, objects are 'reset' back to their original state, ie. the state of the object in the main thread is retained, changes in the processing threads are dropped. However if you run easy_mp with the number of processes set to 1, object states are modified, because the main thread is the executing thread. How it this supposed to work? For a more detailed test please run $ libtbx.python libtbx/tst_easy_mp_state.py The source code explains the test in more detail. This behaviour currently causes dials.integrate to fail on bzipped input files. Given that this code is used in many places and this particular feature is not obvious, it may explain other anecdotal emergent behaviour, too. -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 3

FW: Problems with cctbx - Explore symmetry
by robert.mcmeeking＠stfc.ac.uk 19 Feb '16

19 Feb '16

Dear Colleagues I may not have sent my previous message to the most appropriate email address. I now forward including the "phenix-online.org" one. Is there anyone out there who can point me in the right direction? Best Regards Bob From: McMeeking, Robert (STFC,DL,SC) Sent: 13 February 2016 12:18 To: cctbx(a)cci.lbl.gov Cc: McMeeking, Robert (STFC,DL,SC) Subject: Problems with cctbx - Explore symmetry Hello from the UK I am currently experiencing problems with the Explore symmetry web service - http://cci.lbl.gov/cctbx/explore_symmetry.html. When dealing with a particular Hall Symbol there is a cctbx runtime error (Malformed matrix symbol). The symbol in question is: -R 3 2"c It represents a setting for Space Group R -3 c :H (167) I am practically certain this worked fine in the past. I suspect the problem may be related to parsing the double quote symbol. If I replace with a single quote it manages to run OK, but the output information is a non-standard setting for Space Group 165. By way of background: I call the web service from my own CGI script and extract the symmetry operators that I need to display the crystal structure in question. The Hall symbol used is from the Crystallographic Open Database CIF (COD 2100992). I noticed the problem very recently, but because I cache information it is not clear when the problem first appeared. I only need to go through this process when explicit symmetry operators are not present in the CIF in question. I have not, as yet, installed the cctbx system on our server. Inoticed the problem when using our CrystalWorks system to attempt to display entries from the Crystallography365 Project http://cds.dl.ac.uk/cgi-bin/rfm/crystalworks_365?Feb17 Best Regards Bob Dr R.F. McMeeking Honorary Scientist STFC Chemical Database Service at Daresbury

4 8

Updated base dependencies (not Boost)
by Billy Poon 19 Feb '16

19 Feb '16

Hi all, I have a set of new dependencies for base running on a CentOS 6.7 machine. The tests for PHENIX pass, but I'm still testing the GUI. It looks like most of the DIALS tests pass, but there are some that fail due to an updated Python Imaging Library. Testing will take a while longer, but I attached the modified install_base_packages.py and package_defs.py scripts for anyone who wants to try it out. Just use these in place of the ones in cctbx_project/libtbx/auto_build. I'll also clean up the scripts (e.g. remove cruft from the migration to wxPython 3.0.2, refactor some functions) once testing is complete. Aaron and I will be setting up "clean" virtual machine images for CentOS 5, 6, and 7, and Ubuntu 12.04, and 14.04, and eventually Windows 10 and maybe OS X (only Server can be virtualized) for testing the installers. The goal is to simulate what a user would get from a fresh install of an OS and test that our installers work correctly. And after each test, we can revert the virtual machine to an earlier snapshotted state. This should help in debugging some of the recent installation issues. Since our installers include the GUI, the Linux images will have their respective desktop GUIs installed (desktop version instead of server version). We thought about going with minimal server installs, but there are external dependencies, namely X11/Xorg, that we expect the users to have, and installing the desktop GUI would be the easiest way to satisfy any external dependency. Some highlights from the updates, 1) Python Imaging Library has not been updated since 2009. There is an active fork called Pillow that seems to be mostly backwards compatible. I have noticed one change in the API (replace PIL.Image.fromstring with PIL.Image.frombytes), but more testing is needed. 2) The updated version of ReportLab depends on Pillow. 3) GTK2 was not updated to GTK3 since it seems like wxPython only added support for GTK3 in 3.0.2. GTK2 is updated to the latest version. Dependencies for GTK2 (cairo, pango, etc.) are updated. An additional package, HarfBuzz, is needed. 4) libPNG was updated only to 1.5.x, not 1.6.x, because 1.6.x has some stricter checks on the image. Warnings are raised in the GUI. Eventually, this can be upgraded to 1.6.x, but I would need to reprocess all of the PNG icons to remove the offending bits. 5) libFFI was added for the updated version of glib. The updated dependencies have been built on CentOS 5, 6, and 7, and OS X 10.10, but not all the tests have been run yet. The 32-bit version of CentOS 5 is going to require some more work before the updated packages will compile. Let me know if there are any questions or suggestions. Thanks! -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

1 0

mon_lib dependency
by Roger Martin 01 Feb '16

01 Feb '16

Hi, looking at the prebuilts(http://cci.lbl.gov/cctbx_build/dev-565/cctbx-installer-dev-565-in… for CentOS I see the dependency for mon_lib in verify_mon_lib_data.py. Is mon_lib available stand alone or comes in the full ccp4 package only? Where do most of you get a copy of mon_lib?

8 10

FW: Mac builds currently broken on Jenkins
by markus.gerstel＠diamond.ac.uk 29 Jan '16

29 Jan '16

Sorry, this should have gone onto cctbx list. This is running bootstrap.py --builder=dials build on MacOS X, but I don't think it is specific to dials. -Markus From: Gerstel, Markus (DLSLtd,RAL,LSCI) Sent: 22 January 2016 08:51 To: dials-support(a)lists.sourceforge.net Subject: Mac builds currently broken on Jenkins During the build step this happens: (..) /usr/bin/c++ -o annlib_adaptbx/self_include/bd_fix_rad_search.o -c -DBOOST_ALL_NO_LIB -fPIC -fno-strict-aliasing -Wno-c++0x-extensions -Wno-array-bounds -DNDEBUG -O3 -ffast-math -I/Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/build/include -I/Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/cctbx_project -I/Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/cctbx_project -I/Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/boost -I/Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/annlib -I/Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/annlib/src -I/Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/annlib/include -I/Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/annlib_adaptbx/include -I/Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/build/annlib_adaptbx/include annlib_adaptbx/self_include/bd_fix_rad_search.cpp In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/annlib_adaptbx/ann/ann_adaptor.cpp:3: In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/annlib_adaptbx/include/annlib_adaptbx/ann_adaptor.h:4: In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/cctbx_project/scitbx/array_family/flex_types.h:5: In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/cctbx_project/scitbx/array_family/accessors/flex_grid.h:5: In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/cctbx_project/scitbx/array_family/small.h:4: In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/cctbx_project/scitbx/array_family/small_plain.h:4: In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/cctbx_project/scitbx/array_family/tiny.h:4: In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/cctbx_project/scitbx/array_family/tiny_plain.h:5: In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/cctbx_project/scitbx/array_family/detail/misc.h:4: In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/cctbx_project/scitbx/array_family/type_traits.h:29: In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/boost/boost/type_traits.hpp:13: In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/boost/boost/type_traits/add_const.hpp:16: In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/boost/boost/type_traits/detail/type_trait_def.hpp:14: In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/boost/boost/type_traits/detail/template_arity_spec.hpp:10: In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/boost/boost/mpl/int.hpp:17: In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/boost/boost/mpl/int_fwd.hpp:17: In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/boost/boost/mpl/aux_/adl_barrier.hpp:17: /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/boost/boost/mpl/aux_/config/adl.hpp:17:10: fatal error: 'boost/mpl/aux_/config/msvc.hpp' file not found #include <boost/mpl/aux_/config/msvc.hpp> ^ /usr/bin/c++ -o annlib_adaptbx/self_include/bd_pr_search.o -c -DBOOST_ALL_NO_LIB -fPIC -fno-strict-aliasing -Wno-c++0x-extensions -Wno-array-bounds -DNDEBUG -O3 -ffast-math -I/Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/build/include -I/Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/cctbx_project -I/Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/cctbx_project -I/Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/boost -I/Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/annlib -I/Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/annlib/src -I/Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/annlib/include -I/Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/annlib_adaptbx/include -I/Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/build/annlib_adaptbx/include annlib_adaptbx/self_include/bd_pr_search.cpp /usr/bin/c++ -o annlib_adaptbx/self_include/bd_search.o -c -DBOOST_ALL_NO_LIB -fPIC -fno-strict-aliasing -Wno-c++0x-extensions -Wno-array-bounds -DNDEBUG -O3 -ffast-math -I/Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/build/include -I/Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/cctbx_project -I/Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/cctbx_project -I/Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/boost -I/Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/annlib -I/Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/annlib/src -I/Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/annlib/include -I/Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/annlib_adaptbx/include -I/Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/build/annlib_adaptbx/include annlib_adaptbx/self_include/bd_search.cpp In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/annlib_adaptbx/ann/boost_python/annlib_ext.cpp:2: In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/annlib_adaptbx/include/annlib_adaptbx/ann_adaptor.h:4: In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/cctbx_project/scitbx/array_family/flex_types.h:5: In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/cctbx_project/scitbx/array_family/accessors/flex_grid.h:5: In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/cctbx_project/scitbx/array_family/small.h:4: In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/cctbx_project/scitbx/array_family/small_plain.h:4: In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/cctbx_project/scitbx/array_family/tiny.h:4: In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/cctbx_project/scitbx/array_family/tiny_plain.h:5: In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/cctbx_project/scitbx/array_family/detail/misc.h:4: In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/cctbx_project/scitbx/array_family/type_traits.h:29: In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/boost/boost/type_traits.hpp:13: In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/boost/boost/type_traits/add_const.hpp:16: In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/boost/boost/type_traits/detail/type_trait_def.hpp:14: In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/boost/boost/type_traits/detail/template_arity_spec.hpp:10: In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/boost/boost/mpl/int.hpp:17: In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/boost/boost/mpl/int_fwd.hpp:17: In file included from /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/boost/boost/mpl/aux_/adl_barrier.hpp:17: /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/boost/boost/mpl/aux_/config/adl.hpp:17:10: fatal error: 'boost/mpl/aux_/config/msvc.hpp' file not found #include <boost/mpl/aux_/config/msvc.hpp> ^ 1 error generated. scons: *** [annlib_adaptbx/ann/ann_adaptor.o] Error 1 1 error generated. scons: *** [annlib_adaptbx/ann/boost_python/annlib_ext.o] Error 1 scons: building terminated because of errors. usr+sys time: 3.36 seconds, ticks: 13399823, micro-seconds/tick: 0.251 wall clock time: 15.72 seconds Performing actions: build ===== Running in .: deleting dist, tests, tmp removing .pyc files in /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules removed 3 files ===== Running in build: run configure.py ===== Running in build: save configure command ===== Running in build: libtbx.scons Process failed with return code 2 Build step 'Execute shell' marked build as failure -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 4

dependency on phenix
by Roger Martin 26 Jan '16

26 Jan '16

Hi, looking through http://sourceforge.net/p/cctbx/code/HEAD/tree/trunk/mmtbx/refinement/minimi… shows at line 61 from phenix.refinement import weight_xray_chem and http://sourceforge.net/p/cctbx/code/HEAD/tree/trunk/mmtbx/refinement/occupa… line 134 Is weight_xray_chem.py code that could move to cctbx? (cctbx_project/mmtbx/refinement/weight_xray_chem.py) Are there other cross dependencies?

2 2

Git (again)
by Graeme.Winter＠diamond.ac.uk 15 Jan '16

15 Jan '16

Hi Folks so, while this is tangential to the discussion of whether or not to move to git, I will observe that our chosen repo supplier appears to be taking some down time again: [gw56@ws133 format]$ svn commit -m "Comment about application of this format" Sending format/FormatSMVRigakuEiger1M.py Transmitting file data .svn: Commit failed (details follow): svn: Can't create directory '/svn/p/cctbx/code/db/transactions/23592-1.txn': Permission denied Over the last year this has not been such a rare occurrence and is always annoying when it does happen. The sourceforge.net website is temporarily in static offline mode. Only a very limited set of project pages are available until the main website returns to service. While this is I agree unrelated to the revision control we elect to use, it should perhaps be part of the same discussion. Meanwhile, all builds for dials will be gone until sf.net is once again live... awesome! Cheerio Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1