- cctbxbb - phenix-online.org

incorrect url for downloading fasta files
by R.D. Oeffner 28 Jun '17

28 Jun '17

hi, It would appear that the URL in cctbx_projectiotbxpdbfetch.py for fetching fasta files has been changed by RCSB. So fetch.py therefore downloads garbage when trying to download a fasta file. The correct URL is not documented by RCSB but can be inferred from the RCSB web site when downloading a fasta file. I've updated the URL in fetch.py accordingly. Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234

2 1

please read before your next commit
by markus.gerstel＠diamond.ac.uk 27 Jun '17

27 Jun '17

Good morning, recently a couple of large files were erroneously committed to the cctbx repository and then deleted thereafter. Git always keeps a full history, so to avoid cruft aggregating over time we decided to cut those files out of the repository history. Unfortunately this means everyone will have to do a couple of steps: 1. Please ensure you have run libtbx.git_avoid_merges on all of your development machines 2. Run git pull --rebase the next time you update your repository instead of git pull - again, do this on all development machines. 3. If you keep any local branches you may need to rebase them onto master (git fetch; git checkout $branchname; git rebase origin/master) If you don't follow these instructions you may end up recommitting those large files, and we have to start again. Thanks -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

Housekeeping / roadmap for future
by Graeme.Winter＠diamond.ac.uk 21 Jun '17

21 Jun '17

Hello all, Appreciate a phenix release coming shortly & not suggesting anything any time soon However we were discussing earlier what the road map is for cctbx - making it ready for the future - and I was wondering what is already on the table? Is there a road map somewhere? I recall Billy mentioning Python3 recently, as an example... Thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

4 5

Re: [cctbxbb] bootstrap.py on Windows
by R. D. Oeffner 20 Jun '17

20 Jun '17

Instructions for building on windows are here: https://github.com/cctbx/cctbx_project/wiki Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234 -----Original Message----- From: R. D. Oeffner Sent: Tuesday, June 20, 2017 10:37 PM To: horst.puschmann(a)gmail.com ; cctbxbb(a)phenix-online.org Cc: oleg(a)olexsys.org Subject: Re: [cctbxbb] bootstrap.py on Windows Dear Horst, There's a writeup of instructions on how to do CCTBX builds from bootstrap.py on Windows. No authentication is needed for building CCTBX on any platform. Please follow these instructions and get in touch if you have further problems. CCTBX and other projects depending on CCTBX is tested through regression test nightly on LBL machines on all platforms supported. See http://cci-vm-6.lbl.gov:8010/one_line_per_build for log files of these tests. Regards, Robert -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234 -----Original Message----- From: markus.gerstel(a)diamond.ac.uk Sent: Tuesday, June 20, 2017 9:25 PM To: horst.puschmann(a)gmail.com ; cctbxbb(a)phenix-online.org Cc: oleg(a)olexsys.org Subject: Re: [cctbxbb] bootstrap.py on Windows Hello Horst, I think the automated tests aren't really covering the bootstrap process. While we extensively test bootstrap at our place, we do not run any tests on Windows. I don't know if LBL runs Windows tests, but if they do they probably run it within buildbot, which means it is not run the same way you run it. Best wishes -Markus ________________________________ From: cctbxbb-bounces(a)phenix-online.org [cctbxbb-bounces(a)phenix-online.org] on behalf of Horst Puschmann [horst.puschmann(a)gmail.com] Sent: Tuesday, April 04, 2017 09:59 To: cctbxbb(a)phenix-online.org Cc: Oleg Dolomanov (oleg(a)olexsys.org) Subject: [cctbxbb] bootstrap.py on Windows Hello I am trying to install the cctbx on a Windows 7 64 bit machine using the following bootstrap.py file: https://raw.githubusercontent.com/cctbx/cctbx_project/master/libtbx/auto_bu… The script will fail, unless I comment out lines 1101 and 1102: #if self.isPlatformWindows(): #tarurl, arxname, dirpath = MODULES.get_module(module)().get_tarauthenticated(auth=self.get_auth()) If it tries to execute the commented lines, the error will be "KeyError: 'cciuser' in line 590" After commenting this out, it will fail in line 271, with "AttributeError: 'module' object has no attribute '_create_unverified_context'" I can get round that with disabling line 247: if sys.platform == "win32": (i.e. change this to if sys.platform == "xxx":) After that, things start downloading. All appears well until the same thing happens again in another bootstrap.py file in \modules\cctbx_project\libtbx\auto_build\bootstrap.py If I disable *that*, it goes further, but fails finally with ===== Running in build: run configure.py Traceback (most recent call last): File "bootstrap.py", line 2113, in <module> run() File "bootstrap.py", line 2108, in run enable_shared=options.enable_shared, File "bootstrap.py", line 1036, in run i.run() File "bootstrap.py", line 189, in run raise RuntimeError("Could not run %s: File not found" % executable) RuntimeError: Could not run base\bin\python: File not found I guess the automatic tests haven't picked this up, because no authentication is needed, maybe? Greetings Horst -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org http://phenix-online.org/mailman/listinfo/cctbxbb --- This email has been checked for viruses by AVG. http://www.avg.com

1 0

bootstrap.py on Windows
by Horst Puschmann 20 Jun '17

20 Jun '17

Hello I am trying to install the cctbx on a Windows 7 64 bit machine using the following bootstrap.py file: https://raw.githubusercontent.com/cctbx/cctbx_project/ master/libtbx/auto_build/bootstrap.py The script will fail, unless I comment out lines 1101 and 1102: #if self.isPlatformWindows(): #tarurl, arxname, dirpath = MODULES.get_module(module)().get_tarauthenticated(auth=self.get_auth()) If it tries to execute the commented lines, the error will be "KeyError: 'cciuser' in line 590" After commenting this out, it will fail in line 271, with "AttributeError: 'module' object has no attribute '_create_unverified_context'" I can get round that with disabling line 247: if sys.platform == "win32": (i.e. change this to if sys.platform == "xxx":) After that, things start downloading. All appears well until the same thing happens again in another bootstrap.py file in \modules\cctbx_project\libtbx\auto_build\bootstrap.py If I disable *that*, it goes further, but fails finally with ===== Running in build: run configure.py Traceback (most recent call last): File "bootstrap.py", line 2113, in <module> run() File "bootstrap.py", line 2108, in run enable_shared=options.enable_shared, File "bootstrap.py", line 1036, in run i.run() File "bootstrap.py", line 189, in run raise RuntimeError("Could not run %s: File not found" % executable) RuntimeError: Could not run base\bin\python: File not found I guess the automatic tests haven't picked this up, because no authentication is needed, maybe? Greetings Horst

3 2

bioinformatics.SE
by Marcin Wojdyr 16 Jun '17

16 Jun '17

Hello, just to let you know, a few weeks ago StackExchange started a Bioinformatics Q&A site. Like Biostars, it's mostly about sequences, but sometimes people pop up with questions that could be answered using cctbx, for example a few minutes ago: https://bioinformatics.stackexchange.com/questions/783/how-can-we-find-the-…

1 0

Fwd: [git/cctbx] master: Making cif behavior exactly the same as pdb (throwing away SCALE records if crystal_symmetry is supplied to obtain xray_structure). With test. (bdf35cf1c)
by Graeme.Winter＠diamond.ac.uk 16 Jun '17

16 Jun '17

Please could people refrain from making cctbx tests dependent on phenix tools? E.g. not easy_run.call("phenix.cif_as_pdb %s.cif > junk.out"%prefix) Obliged in anticipation Cheers Graeme Begin forwarded message: From: CCTBX Commit via DLS Jenkins <graeme.winter(a)GMAIL.COM<mailto:[email protected]>> Subject: [git/cctbx] master: Making cif behavior exactly the same as pdb (throwing away SCALE records if crystal_symmetry is supplied to obtain xray_structure). With test. (bdf35cf1c) Date: 15 June 2017 at 19:43:57 BST To: <CCTBX-COMMIT(a)JISCMAIL.AC.UK<mailto:[email protected]>> Reply-To: <cctbxbb(a)PHENIX-ONLINE.ORG<mailto:[email protected]>> Repository : ssh://g18-sc-serv-04.diamond.ac.uk/cctbx On branch : master ________________________________ commit bdf35cf1c69d235d2578d915836aad2a739c1c15 Author: Oleg Sobolev <osobolev(a)lbl.gov<mailto:[email protected]>> Date: Thu Jun 15 11:43:52 2017 -0700 Making cif behavior exactly the same as pdb (throwing away SCALE records if crystal_symmetry is supplied to obtain xray_structure). With test. ________________________________ bdf35cf1c69d235d2578d915836aad2a739c1c15 iotbx/pdb/mmcif.py | 24 +++++----- iotbx/regression/tst_xray_scale.py | 93 ++++++++++++++++++++++++++++++++++++++ iotbx/run_tests.py | 1 + 3 files changed, 107 insertions(+), 11 deletions(-) diff --git a/iotbx/pdb/mmcif.py b/iotbx/pdb/mmcif.py index 6a5221a6a..2b79df99a 100644 --- a/iotbx/pdb/mmcif.py +++ b/iotbx/pdb/mmcif.py @@ -508,17 +508,19 @@ class cif_input(iotbx.pdb.pdb_input_mixin): return extract_tls_from_cif_block(self.cif_block, hierarchy) def scale_matrix(self): - fractionalization_matrix = [ - self.cif_block.get('_atom_sites.fract_transf_matrix[%s][%s]' %(i, j)) - for i,j in ('11', '12', '13', '21', '22', '23', '31','32', '33')] - if fractionalization_matrix.count(None) == 9: - return None - assert fractionalization_matrix.count(None) == 0 - fractionalization_vector = [ - self.cif_block.get('_atom_sites.fract_transf_vector[%s]' %i) for i in '123'] - assert fractionalization_matrix.count(None) == 0 - return [[float(i) for i in fractionalization_matrix], - [float(i) for i in fractionalization_vector]] + if (not hasattr(self, "_scale_matrix")): + fractionalization_matrix = [ + self.cif_block.get('_atom_sites.fract_transf_matrix[%s][%s]' %(i, j)) + for i,j in ('11', '12', '13', '21', '22', '23', '31','32', '33')] + if fractionalization_matrix.count(None) == 9: + return None + assert fractionalization_matrix.count(None) == 0 + fractionalization_vector = [ + self.cif_block.get('_atom_sites.fract_transf_vector[%s]' %i) for i in '123'] + assert fractionalization_matrix.count(None) == 0 + self._scale_matrix = [[float(i) for i in fractionalization_matrix], + [float(i) for i in fractionalization_vector]] + return self._scale_matrix def model_ids(self): return flex.std_string([model.id for model in self.hierarchy.models()]) diff --git a/iotbx/regression/tst_xray_scale.py b/iotbx/regression/tst_xray_scale.py new file mode 100644 index 000000000..e1a4ec9f3 --- /dev/null +++ b/iotbx/regression/tst_xray_scale.py @@ -0,0 +1,93 @@ +from __future__ import division +from libtbx import easy_run +import iotbx.pdb +from cctbx import crystal +from libtbx.test_utils import approx_equal + +cif_str=""" +data_5JUP +# +_atom_sites.entry_id 5JUP +_atom_sites.fract_transf_matrix[1][1] 1.000000 +_atom_sites.fract_transf_matrix[1][2] 0.000000 +_atom_sites.fract_transf_matrix[1][3] 0.000000 +_atom_sites.fract_transf_matrix[2][1] 0.000000 +_atom_sites.fract_transf_matrix[2][2] 1.000000 +_atom_sites.fract_transf_matrix[2][3] 0.000000 +_atom_sites.fract_transf_matrix[3][1] 0.000000 +_atom_sites.fract_transf_matrix[3][2] 0.000000 +_atom_sites.fract_transf_matrix[3][3] 1.000000 +_atom_sites.fract_transf_vector[1] 0.00000 +_atom_sites.fract_transf_vector[2] 0.00000 +_atom_sites.fract_transf_vector[3] 0.00000 +# +loop_ +_atom_type.symbol +C +MG +N +O +P +S +# +loop_ +_atom_site.group_PDB +_atom_site.id +_atom_site.type_symbol +_atom_site.label_atom_id +_atom_site.label_alt_id +_atom_site.label_comp_id +_atom_site.label_asym_id +_atom_site.label_entity_id +_atom_site.label_seq_id +_atom_site.pdbx_PDB_ins_code +_atom_site.Cartn_x +_atom_site.Cartn_y +_atom_site.Cartn_z +_atom_site.occupancy +_atom_site.B_iso_or_equiv +_atom_site.auth_seq_id +_atom_site.auth_comp_id +_atom_site.auth_asym_id +_atom_site.auth_atom_id +_atom_site.pdbx_PDB_model_num +ATOM 9 C "C3'" . U A 1 1 ? 175.229 258.091 229.127 1.00 113.67 1 U A "C3'" 1 +ATOM 7 C "C4'" . U A 1 1 ? 175.388 257.913 227.616 1.00 113.21 1 U A "C4'" 1 +ATOM 6 C "C5'" . U A 1 1 ? 174.124 257.916 226.769 1.00 114.11 1 U A "C5'" 1 +ATOM 10 O "O3'" . U A 1 1 ? 176.253 257.344 229.761 1.00 112.89 1 U A "O3'" 1 +ATOM 8 O "O4'" . U A 1 1 ? 176.291 258.974 227.185 1.00 112.70 1 U A "O4'" 1 +ATOM 5 O "O5'" . U A 1 1 ? 173.735 259.270 226.430 1.00 115.09 1 U A "O5'" 1 +ATOM 3 O OP1 . U A 1 1 ? 174.056 259.302 223.918 1.00 117.16 1 U A OP1 1 +ATOM 4 O OP2 . U A 1 1 ? 172.661 261.090 225.058 1.00 104.87 1 U A OP2 1 +ATOM 1 O OP3 . U A 1 1 ? 171.796 258.709 224.899 1.00 116.23 1 U A OP3 1 +ATOM 2 P P . U A 1 1 ? 173.022 259.610 224.994 1.00 116.34 1 U A P 1 +# +""" + +def run(prefix="iotbx_tst_xray_scale"): + """ + Exercise obtaining exactly the same xray structure from pdb and cif. + Caveate is SCALE records (fract_transf_matrix). In pdb they explicitly + suppressed when crystal_symmetry is supplied, see + iotbx/pdb/__init__.py:939 (def xray_structure_simple): + if(crystal_symmetry is not None): self._scale_matrix = None + """ + fo = open("%s.cif" % prefix,"w") + print >> fo, cif_str + fo.close() + # crystal symmetry from map + cs = crystal.symmetry((419., 419., 419., 90.0, 90.0, 90.0), 1) + # xrs from mmCIF + cif_inp = iotbx.pdb.input(file_name="%s.cif" % prefix) + # print "======== doing xrs from cif ==============" + xrs_cif = cif_inp.xray_structure_simple(crystal_symmetry = cs) + # xrs from PDB + easy_run.call("phenix.cif_as_pdb %s.cif > junk.out"%prefix) + pdb_inp = iotbx.pdb.input(file_name="%s.pdb" % prefix) + # print "======== doing xrs from pdb ==============" + xrs_pdb = pdb_inp.xray_structure_simple(crystal_symmetry = cs) + # + assert approx_equal(xrs_cif.sites_cart()[0], xrs_pdb.sites_cart()[0]) + +if (__name__ == "__main__"): + run() diff --git a/iotbx/run_tests.py b/iotbx/run_tests.py index 52eb3d2e4..9f3e50a98 100644 --- a/iotbx/run_tests.py +++ b/iotbx/run_tests.py @@ -93,6 +93,7 @@ tst_list = ( "$D/regression/tst_show_systematic_absences.py", "$D/regression/tst_miller_sort_asu.py", "$D/regression/tst_reflection_file_reader.py", + "$D/regression/tst_xray_scale.py", ) def run(): -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 1

Backwards compatibility; Python 2.3?
by Nicholas Devenish 12 Jun '17

12 Jun '17

Hi All, I'm starting to go through and learn some of the structure of these packages, starting with libtbx. What's the supported targets supposed to be? There seems to be code specifically marked as backwards compatibility for Python 2.3 (2003), and packages explicitly included that were merged into the standard library around the same time (e.g. Optik). Is this really as far back as we need to support? Thanks, Nick Devenish

7 12

bz2 support on Ubuntu
by David Waterman 08 Jun '17

08 Jun '17

Hi folks, My bootstrap build of cctbx/DIALS on Ubuntu 16.04.2 LTS produces a base Python that does not have bz2 support. This is despite the fact that the package "libbz2-dev is already the newest version (1.0.6-8)." Looking in Python_install_log I see Python build finished, but the necessary bits to build these modules were not found: _bsddb _curses _curses_panel _sqlite3 _tkinter bsddb185 bz2 dbm dl gdbm imageop readline sunaudiodev Has anyone got any ideas? Anyone done a boostrap build on Ubuntu where bz2 can be imported in Python? Also, what's the procedure to rebuild from base, recovering all the currently configured modules? My guess is that something like this would do it, but is there a better way? export MYMODULES=$(libtbx.list_modules | tr '\n' ' ') rm -rf base_tmp/ base/ build/ python bootstrap.py --builder=dials base python bootstrap.py --builder=dials build cd build source setpaths.sh libtbx.configure $MYMODULES make Cheers -- David

2 3

simtbx
by Graeme.Winter＠diamond.ac.uk 05 Jun '17

05 Jun '17

Hi Folks We have noticed that simtbx, part of the core cctbx now, depends on dials This means if you do not have a dials build your cctbx will be broken This is probably not the way we want things? I thought simtbx was a separate repo? Cheerio Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

7 17

cctbx has acquired a hidden dependency on iotbx
by Luc Bourhis 27 May '17

27 May '17

Hi, C:\Users\luc\Documents\Developer\cctbx\modules\cctbx_project\cctbx\maptbx\SConscript, line 5-8: env = env_base.Clone( SHLINKFLAGS=env_etc.shlinkflags, LIBS=["iotbx_xplor", "cctbx_sgtbx_asu", "cctbx", "omptbx", env_etc.libm] ) This is not supposed to be: the cctbx shall be usable with only what’s defined in its libtbx_config. Best wishes, Luc

3 2

shall iotbx really depends on dxtbx?
by Luc Bourhis 26 May '17

26 May '17

Hi, iotbx/libtbx_config: { "modules_required_for_use": ["cctbx", "fable", "ucif", "smtbx", "dxtbx"], "optional_modules": ["ccp4io", "cbflib"], } A search for dxtbx in the source code of iotbx does not return anything. Did I miss something? Moreover dxtbx/libtbx_config: { "modules_required_for_use" : [ "iotbx" ], } The problem is that dxtbx pulls in HDF5 and that it fails to compiles on Windows 64 with a 64-bit Python, a configuration we need for Olex 2. If the solution was as simple as dropping the dependence of iotbx on dxtbx, that would make my day! Thoughts? Best wishes, Luc Bourhis

3 4

probe/reduce repositories
by Hattne, Johan 04 May '17

04 May '17

Dear all; I appear to be missing probe in my source build. On running phenix.model_vs_data I get the traceback posted below. The error message appears to be a tad misleading, because having probe in PATH is not enough: the actual test in the code is if (not libtbx.env.has_module(name="probe")): which leads me to believe that I’m supposed to have something probe-like in my source tree, but the duke-server I used to get these kinds of things from (quidditch?) seems unresponsive from here. This is the first time in weeks that I’m doing this, so it is very much likely that whatever happened to break my setup happened quite some time ago. Cluebat, anyone? // Cheers; Johan Traceback (most recent call last): File "/mnt/gonen/home/hattnej/projects/phenix-build/../phenix-src/phenix/phenix/command_line/model_vs_data.py", line 6, in <module> show_geometry_statistics = True) File "/mnt/gonen/home/hattnej/projects/phenix-src/cctbx_project/mmtbx/model_vs_data.py", line 710, in run atom_selections = atom_selections) File "/mnt/gonen/home/hattnej/projects/phenix-src/cctbx_project/mmtbx/model_vs_data.py", line 346, in show_geometry restraints_manager = restraints_manager) File "/mnt/gonen/home/hattnej/projects/phenix-src/cctbx_project/mmtbx/model_statistics.py", line 138, in __init__ molprobity_scores=molprobity_scores) File "/mnt/gonen/home/hattnej/projects/phenix-src/cctbx_project/mmtbx/model_statistics.py", line 51, in __init__ self.clashscore = clashscore(pdb_hierarchy=pdb_hierarchy).get_clashscore() File "/mnt/gonen/home/hattnej/projects/phenix-src/cctbx_project/mmtbx/validation/clashscore.py", line 90, in __init__ "http://kinemage.biochem.duke.edu/") RuntimeError: Probe could not be detected on your system. Please make sure Probe is in your path. Probe is available at http://kinemage.biochem.duke.edu/ Research Specialist @ Gonen Lab ____________________________________________________ Janelia Research Campus * 19700 Helix Drive Ashburn, VA 20147 * +1 (571) 209-4000 extension 3376

2 5

normalize() is not safe
by Oleg Sobolev 02 May '17

02 May '17

Dear colleagues, I recently discovered that normalize() function of scitbx::vec3<double> is not safe, i.e. this case scitbx::vec3<double> a(0,0,0); a.normalize(); it will produce division by zero and fail. If function contains this being called from python the traceback would be rather vague: <...> Floating-point error (Python and libc call stacks above) This crash may be due to a problem in any imported Python module, including modules which are not part of the cctbx project. To disable the traps leading to this message, define these environment variables (e.g. assign the value 1): BOOST_ADAPTBX_FPE_DEFAULT BOOST_ADAPTBX_SIGNALS_DEFAULT This will NOT solve the problem, just mask it, but may allow you to proceed in case it is not critical. The code that generates this error sits in scitbx/vec3.h, lines 143-153 with a comment that this was done intentionally and a pointer to an appropriate function to do safe normalization. This function is not wrapped to be available in python directly. Similar functionality (but coded in a different place) accessible via python by: >>> from scitbx import matrix >>> b = matrix.col([0,0,0]) >>> b.normalize() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/net/anaconda/raid1/olegs/phenix_queue/modules/cctbx_project/scitbx/matrix/__init__.py", line 270, in normalize return self / abs(self) File "/net/anaconda/raid1/olegs/phenix_queue/modules/cctbx_project/scitbx/matrix/__init__.py", line 158, in __truediv__ return rec([e/other for e in self.elems], self.n) ZeroDivisionError: float division by zero Fast grepping shows that normalize() is widely used in xfel, smtbx, rstbx, dxtbx, mmtbx/hydrogens, scitbx/rigid_body, and in some other places. My questions to CCTBX community are: 1. Were you aware of this issue? 2. Should we do something about it: - make the function more safe and slow; - leave it like it is and let everybody know about the issue and let developers deal with it individually; 3. Any other input on the matter? Best regards, Oleg Sobolev.

5 5

Boost Python 1.56
by Luc Bourhis 14 Apr '17

14 Apr '17

Hi, I have improvements to the smtbx on their way to be committed which require Boost version 1.56. This is related to Boost.Threads, whose support I re-activated a few months ago on Nick’s request. I need the function boost::thread::physical_concurrency which returns the number of physical cores on the machine, as opposed to virtual cores when hyperthreading is enabled (which it is by default on any Intel machine). That function is not available in Boost 1.55 which is the version currently used in the nightly tests: it appeared in 1.56. So, would it be possible to move to Boost 1.56? Otherwise, I will need to backport that function. Not too difficult but not thrilling. Best wishes, Luc

14 36

Re: [cctbxbb] [git/cctbx] master: Replacing illegal C++ code with proper dynamic array allocation. (a31eb2bf0)
by Nicholas Sauter 12 Apr '17

12 Apr '17

Rob, Just as a general thing, "new" statements in C++ need to be paired with "delete" statements (delete pk, delete a_sum, and delete akml). Unless this is done there will be a memory leak. Nick Nicholas K. Sauter, Ph. D. Senior Scientist, Molecular Biophysics & Integrated Bioimaging Division Lawrence Berkeley National Laboratory 1 Cyclotron Rd., Bldg. 33R0345 Berkeley, CA 94720 (510) 486-5713 On Tue, Apr 11, 2017 at 3:31 AM, CCTBX Commit via DLS Jenkins < graeme.winter(a)gmail.com> wrote: > Repository : ssh://g18-sc-serv-04.diamond.ac.uk/cctbx > On branch : master > > ------------------------------ > > > commit a31eb2bf0d5bb129425bb759466f45622c773fd4 > Author: Robert Oeffner <rdo20(a)cam.ac.uk> > Date: Tue Apr 11 11:31:24 2017 +0100 > > Replacing illegal C++ code with proper dynamic array allocation. > > > ------------------------------ > > > a31eb2bf0d5bb129425bb759466f45622c773fd4 > scitbx/lbfgs.h | 6 +++--- > 1 file changed, 3 insertions(+), 3 deletions(-) > > diff --git a/scitbx/lbfgs.h b/scitbx/lbfgs.h > index 41be8e8cc..f0000143d 100755 > --- a/scitbx/lbfgs.h > +++ b/scitbx/lbfgs.h > @@ -917,9 +917,9 @@ namespace lbfgs { > int info = 5; > brackt = false; > FloatType sxnorm = std::sqrt(ddot(SizeType(n), sx, sx)); > - FloatType pk[n]; > - FloatType a_sum[n]; > - FloatType akm1[n]; > + FloatType* pk = new FloatType[n]; > + FloatType* a_sum = new FloatType[n]; > + FloatType* akm1 = new FloatType[n]; > for (SizeType i=0; i < n; i++) { > pk[i] = sx[i]/sxnorm; > akm1[i] = -gx[i]/sxnorm; >

3 2

libtiff dependency on macOS
by Billy Poon 11 Apr '17

11 Apr '17

Hi everyone, I just updated install_base_packages.py to build libtiff generally instead of just for Linux. So if you update your sources and get build errors relating to TIFF, please rebuild your base directory. Thanks! -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

1 0

Re: [cctbxbb] bug fix in cctbx_project/scitbx/lbfgs.h
by R.D. Oeffner 11 Apr '17

11 Apr '17

There is now a fatal warning in libtbx/SConscript if gcc encounters variable length arrays as this is not portable code and we've had broken builds on more than one occasion due to this recently. Rob On 11/04/2017 12:01, Pavel Afonine wrote: > Hi Rob, > > thanks for the fix. This is was Min's code (remember her from the past workshop?). I simply applied her patch without looking into it too much, and made sure all t96 pass. > > Again, thanks for fixing it. > > Pavel > > On 4/11/17 03:48, R.D. Oeffner wrote: > >> Hi Pavel, I have made a change to that file as it didn't compile on Windows. In fact I can't understand how it would compile with other compilers before as the syntax to me seemed illegal given that n is not known at compile time. I'm not sure how a dynamically created FloatType should be destroyed as I couldn't find any examples of that being done throughout this file. Rob

2 1

bootstrap.py on windows
by Horst Puschmann 07 Apr '17

07 Apr '17

Hello I am trying to install the cctbx on a Windows 7 64 bit machine using the following bootstrap.py file: https://raw.githubusercontent.com/cctbx/cctbx_project/master /libtbx/auto_build/bootstrap.py The script will fail, unless I comment out lines 1101 and 1102: #if self.isPlatformWindows(): #tarurl, arxname, dirpath = MODULES.get_module(module)(). get_tarauthenticated(auth=self.get_auth()) If it tries to execute the commented lines, the error will be "KeyError: 'cciuser' in line 590" After commenting this out, it will fail in line 271, with "AttributeError: 'module' object has no attribute '_create_unverified_context'" I can get round that with disabling line 247: if sys.platform == "win32": (i.e. change this to if sys.platform == "xxx":) After that, things start downloading. All appears well until the same thing happens again in another bootstrap.py file in \modules\cctbx_project\ libtbx\auto_build\bootstrap.py If I disable *that*, it goes further but fails finally with ===== Running in build: run configure.py Traceback (most recent call last): File "bootstrap.py", line 2113, in <module> run() File "bootstrap.py", line 2108, in run enable_shared=options.enable_shared, File "bootstrap.py", line 1036, in run i.run() File "bootstrap.py", line 189, in run raise RuntimeError("Could not run %s: File not found" % executable) RuntimeError: Could not run base\bin\python: File not found I guess the automatic tests haven't picked this up, because no authentication is needed, maybe? Greetings Horst

6 16

Fw: cctbx developer guidelines
by R. D. Oeffner 06 Apr '17

06 Apr '17

Hi Pavel and others, Attached is the original word document you sent out if that's useful for anyone. Rob -----Original Message----- From: Pavel Afonine Sent: Saturday, July 11, 2015 7:12 PM To: Paul Adams ; developers(a)phenix-online.org Subject: Re: cctbx developer guidelines Hi Paul, looks great! Attached is slightly edited copy. Pavel On 7/10/15 11:14, Paul Adams wrote: > Hi Everyone, > > we have developed a cctbx developer guidance document (much of which > also applies to Phenix development). I’m attaching it for your review and > comments. If you have any suggestions let me know. The goal is to > distribute this to all developers who are contributing to the cctbx and to > put it somewhere prominent on the web. > > Cheers, > Paul >

1 0

bootstrap.py not overwriting modified sources
by R.D. Oeffner 01 Apr '17

01 Apr '17

Hi all, It appears that when bootstrap.py is invoked to update sources then unsurprisingly it doesn't overwrite existing sources if these are more recent, as in someone has modified them and not committed the changes. Occasionally this leads to broken builds. To prevent this from happening would it not be a good idea to change bootstrap.py to default to overwrite any uncommitted changes unless some flag is set on the command line? Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234 mobile: +44(0)7712 887162

4 5

Berkeley Server Downtime (3/23 - 3/24)
by Billy Poon 24 Mar '17

24 Mar '17

Dear all, On Thursday, March 23, our servers will be shut down starting at 5:00 pm Pacific for some emergency electrical work near our building. We expect that the servers will be available the next day (3/24) by 1:00 pm Pacific. There is a possibility that the outage will last longer. This will affect, - The Phenix web site (http://phenix-online.org/) - The DIALS West web site (http://dials.lbl.gov/) - The CCI web site (http://cci.lbl.gov/) - The Structural Biology Technology Portal (http://technology.sbkb.org/) Bug reports and emails requesting help should be queued and we will receive them once the servers are back online. We apologize for any inconvenience and the short notice. -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

1 1

Re: [cctbxbb] cctbxbb Digest, Vol 96, Issue 3
by Allan Lyckegaard 21 Mar '17

21 Mar '17

Hi Billy, Thanks, it works! It now compiles and when adding it to the path, “source build/setpaths.sh”, it can be imported in python. A few questions, though: - is it necessary to create a new conda environment to make it work? My first try where I got the error was in an existing environment. - can I update any packages in my new environment without breaking cctbx? I would i.e. be wonderful to have access to ipython Cheers, Allan ******************************************************** Allan Lyckegaard, PhD, Scientist Xnovo Technology ApS Galoche Alle 15 4600 Køge CVR: 3489 4221 Mail: alyckegaard(a)xnovotech.com <mailto:[email protected]> Mobile: +45 5373 8581 ******************************************************* The information in this email, including any attachments, is confidential and intended only for the recipient(s) listed. Any use of this email for any other purpose is prohibited. If you have received this email in error, please notify me immediately by reply email, delete this email, and do not disclose its contents to anyone. > On 21 Mar 2017, at 20.00, cctbxbb-request(a)phenix-online.org wrote: > > Send cctbxbb mailing list submissions to > cctbxbb(a)phenix-online.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://phenix-online.org/mailman/listinfo/cctbxbb > or, via email, send a message with subject or body 'help' to > cctbxbb-request(a)phenix-online.org > > You can reach the person managing the list at > cctbxbb-owner(a)phenix-online.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of cctbxbb digest..." > > > Today's Topics: > > 1. Re: Build error, bootstrap.py (Billy Poon) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 20 Mar 2017 12:38:31 -0700 > From: Billy Poon <bkpoon(a)lbl.gov> > To: cctbx mailing list <cctbxbb(a)phenix-online.org> > Subject: Re: [cctbxbb] Build error, bootstrap.py > Message-ID: > <CAHmYUcY2oJp=D9w_de=S++fn4D9rzanB+zFtqU8ME9OgWmSu-Q(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi Allan, > > Your command should have installed the hdf5 dependency among the other > dependencies. The hdf5 dependency is required because there are functions > for reading image data. Was there an error in compiling HDF5? There should > be a log file in <cctbx installation directory>/base_tmp named > HDF5_install_log. > > Are you using a recent version of bootstrap.py? I just tested this on macOS > 10.12.3 with conda 4.3.11 and the command-line tools for Xcode 8.2.1. The > steps I followed (in a clean directory with just bootstrap.py) are, > > 1) conda create -n test python=2 > 2) source activate test > 3) python bootstrap.py --with-python=`which python` --builder=cctbx > --nproc=8 > 4) cd build; make > > The `which python` command in step 3 just returns the path for the python > in the "test" environment. Step 4 is just to make sure everything is > compiled. > > Let us know if you it still does not work. Thanks! > > -- > Billy K. Poon > Research Scientist, Molecular Biophysics and Integrated Bioimaging > Lawrence Berkeley National Laboratory > 1 Cyclotron Road, M/S 33R0345 > Berkeley, CA 94720 > Tel: (510) 486-5709 > Fax: (510) 486-5909 > Web: https://phenix-online.org > > On Fri, Mar 17, 2017 at 7:27 AM, Allan Lyckegaard <alyckegaard(a)xnovotech.com >> wrote: > >> Hi all, >> >> I am trying to install cctbx on my Mac (Sierra) using bootstrap.py with my >> Anaconda python. >> >> When executing 'python bootstrap.py --with-python=/Users/myuser/ >> miniconda2/envs/python2/bin/python --builder=cctbx? I get the following >> error: >> >> /usr/bin/c++ -o dxtbx/format/boost_python/nexus_ext.o -c >> -DBOOST_HAS_INT128=1 -DBOOST_ALL_NO_LIB -DBOOST_PYTHON_MAX_BASES=2 -Wmost >> -Wno-unknown-pragmas -Wno-logical-op-parentheses -Wno-unused-local-typedefs >> -I/Users/myuser/Development/cctbx/modules/boost -fPIC >> -fno-strict-aliasing -Wno-c++0x-extensions -Wno-array-bounds -DNDEBUG -O3 >> -ffast-math -I/Users/myuser/miniconda2/envs/python2/include >> -I/Users/myuser/Development/cctbx/base/include >> -I/Users/myuser/Development/cctbx/build/include >> -I/Users/myuser/Development/cctbx/modules/cctbx_project >> -I/Users/myuser/miniconda2/envs/python2/include/python2.7 >> /Users/myuser/Development/cctbx/modules/cctbx_project/ >> dxtbx/format/boost_python/nexus_ext.cc >> warning: unknown warning option '-Wno-unused-local-typedefs' >> [-Wunknown-warning-option] >> 1 warning generated. >> /usr/bin/c++ -o lib/dxtbx_format_nexus_ext.so -w -bundle -undefined >> dynamic_lookup dxtbx/format/boost_python/nexus_ext.o >> -L/Users/myuser/Development/cctbx/base/lib -lm -lhdf5 >> ld: library not found for -lhdf5 >> clang: error: linker command failed with exit code 1 (use -v to see >> invocation) >> scons: *** [lib/dxtbx_format_nexus_ext.so] Error 1 >> scons: building terminated because of errors. >> usr+sys time: 8.69 seconds >> wall clock time: 21.70 seconds >> Process failed with return code 2 >> >> Any suggestions to how I solve this? I have not seen anywhere in the >> documentation that an hdf5 installation is a prerequisite. >> >> Cheers, >> /Allan >> >> >> _______________________________________________ >> cctbxbb mailing list >> cctbxbb(a)phenix-online.org >> http://phenix-online.org/mailman/listinfo/cctbxbb >> >

2 1

Build error, bootstrap.py
by Allan Lyckegaard 20 Mar '17

20 Mar '17

Hi all, I am trying to install cctbx on my Mac (Sierra) using bootstrap.py with my Anaconda python. When executing 'python bootstrap.py --with-python=/Users/myuser/miniconda2/envs/python2/bin/python --builder=cctbx’ I get the following error: /usr/bin/c++ -o dxtbx/format/boost_python/nexus_ext.o -c -DBOOST_HAS_INT128=1 -DBOOST_ALL_NO_LIB -DBOOST_PYTHON_MAX_BASES=2 -Wmost -Wno-unknown-pragmas -Wno-logical-op-parentheses -Wno-unused-local-typedefs -I/Users/myuser/Development/cctbx/modules/boost -fPIC -fno-strict-aliasing -Wno-c++0x-extensions -Wno-array-bounds -DNDEBUG -O3 -ffast-math -I/Users/myuser/miniconda2/envs/python2/include -I/Users/myuser/Development/cctbx/base/include -I/Users/myuser/Development/cctbx/build/include -I/Users/myuser/Development/cctbx/modules/cctbx_project -I/Users/myuser/miniconda2/envs/python2/include/python2.7 /Users/myuser/Development/cctbx/modules/cctbx_project/dxtbx/format/boost_python/nexus_ext.cc warning: unknown warning option '-Wno-unused-local-typedefs' [-Wunknown-warning-option] 1 warning generated. /usr/bin/c++ -o lib/dxtbx_format_nexus_ext.so -w -bundle -undefined dynamic_lookup dxtbx/format/boost_python/nexus_ext.o -L/Users/myuser/Development/cctbx/base/lib -lm -lhdf5 ld: library not found for -lhdf5 clang: error: linker command failed with exit code 1 (use -v to see invocation) scons: *** [lib/dxtbx_format_nexus_ext.so] Error 1 scons: building terminated because of errors. usr+sys time: 8.69 seconds wall clock time: 21.70 seconds Process failed with return code 2 Any suggestions to how I solve this? I have not seen anywhere in the documentation that an hdf5 installation is a prerequisite. Cheers, /Allan

2 1

hkl and F
by James Holton 04 Mar '17

04 Mar '17

Hello all, Is there any standard way to get the "C++ signature" of a python object? I'm trying to get my _ext.cpp working, and its' maddening trying to figure out how to declare objects that need to be exposed via boost. -James

2 1