- cctbxbb - phenix-online.org

cctbx.miller problem
by 李选选 05 Dec '17

05 Dec '17

> > Dear cctbx developers, > > I’m Xuanxuan, and I’m using cctbx to develop a crystallography-related algorithm. Recently, I found there is something wrong when I use miller package to generate miller indices for given space group and unit cell parameters. Maybe it’s my mistake, or it’s a bug. > > Here are some details. I just use the following code to generate a list of miller indices for a Fd-3m crystal. > > ms = miller.build_set( > crystal_symmetry=crystal.symmetry( > space_group_symbol='Fd-3m', > unit_cell=(5.4307,5.4307,5.4307,90.00,90.0,90.00)), > anomalous_flag=False, > d_min=0.4) > > And I found some reflections are missing, like 220, 311, which should exist according to the reflection conditions listed in this page http://img.chem.ucl.ac.uk/sgp/large/227az2.htm <http://img.chem.ucl.ac.uk/sgp/large/227az2.htm>. > > Looking forward to your response. > > Thanks, > Xuanxuan

5 4

rotate_translate_map function causes "splinching"
by Pearce, N.M. (Nick) 29 Nov '17

29 Nov '17

Hi all, I’m having a problem with the rotate_translate_map function from cctbx.maptbx. I have a 3D array (a mask of 0s and 1s, in this case) that I’m writing as a CCP4 map file. If I write it out “on the origin” (without running rotate_translate_map), then it’s fine (see attached “before.png”). However, if I apply the rotate_translate_map function and then write it out, odd features appear at the edge of the unit cell (see “after.png”). It is NOT the case that the map is going through the “periodic boundary conditions” of the unit cell (the shift is less than one unit cell, as shown by the structure which is transformed by the same amount). The minimal code is: ——————————— array = cctbx.maptbx.rotate_translate_map( unit_cell=self.unit_cell(), map_data=array, rotation_matrix=scitbx.matrix.rec([1,0,0,0,1,0,0,0,1], (3,3)).elems, translation_vector=(-1.0*scitbx.matrix.rec((X,Y,Z), (3,1))).elems ) # Write the map iotbx.ccp4_map.write_ccp4_map( file_name=f_name, unit_cell=self.unit_cell(), space_group=self.space_group(), map_data=array, labels=flex.std_string(['’]) ) ——————————— where (X,Y,Z) is a tuple for the origin shift. Is there something stupid that I’m missing? Has anyone else had the same problem? The only code for a rotate_translate_map function I could find was actually in maptbx/average_densities.h and I couldn’t see anything obviously “wrong”… though I’m not that familiar with C++. The maps and files can be found at https://www.dropbox.com/sh/xqrsk5lcn603416/AABiVO74g6A_jZMGn_DjxOmka?dl=0 all “grid” files are without translation, and all “ref” files are after translation. both are written out exactly the same (the iotbx.ccp4_map call above). I’m using the cctbx bundled with a ccp4 nightly from 2016-10-22. Thanks, Nick [cid:[email protected]][cid:[email protected]]

2 1

cctbx versions causing indentation crash?
by Magnus Andersson 10 Nov '17

10 Nov '17

Hi all, I have a script that runs fine on my old computer, but crashes on my new computer with the error message: IndentationError: unexpected indent This script is (amongst other things) calling the softwatre Coot such as: subprocess.call(["""coot --no-graphics<<end/n (handle-read-draw-molecule-with-recentre \""""+str(inf)+"""\" 1) (set-refinement-immediate-replacement 1) (set-refine-max-residues 40) (regularize-zone 0 \" \" """+str(res1-2)+""" """+str(res3)+"""\" \") (accept-regularizement) (regularize-zone 0 \" \" """+str(res3)+""" """+str(res2+2)+"""\" \") (accept-regularizement) (write-pdb-file 0 \""""+str(outf)+"""\") (coot-real-exit 0)"""], shell=True) which is ok on my old system. I am trying to find the most likely source of this error. Here are the differences: - OLD cctbx = uses Python 2.7.10, NEW cctbx 2.7.14 - OLD OSX = 10.10.5 (i.e. Yosemite), NEW OSX = 10.11.5 (i.e. El Capitan) - OLD Coot = vers 0.8, NEW Coot = vers 0.8.8 [old Coot install was from Fink but this shouldn’t matter I think] Perhaps someone has encountered something similar or has an educated guess on whether different cctbx versions can cause such errors? Thanks, Magnus #### Magnus Andersson, Docent, Ph.D. Email: magnus.andersson(a)scilifelab.se Phone: 0769469872 Mailing address: Science For Life Laboratory Alpha 6 Box 1031 171 21 Solna Visiting address: Tomtebodavägen 23A 171 65 Solna

2 2

scipy module
by Magnus Andersson 06 Nov '17

06 Nov '17

Hello, I am not normally using cctbx but I’m trying to run a script that imports modules as follows: import sys,io,os import subprocess import numpy import pickle import math import pylab #import pymol import time import scipy import iotbx.pdb import cctbx import mmtbx.model from scipy.optimize import fmin, fmin_cg, leastsq from scipy.interpolate import interp1d import datetime from numpy.random import random_sample from cctbx.array_family import flex from scitbx.math import superpose I have installed cctbx and run with cctbx.python. How can I import scipy into the cctbx python? Cheers, Magnus #### Magnus Andersson, Docent, Ph.D. Email: magnus.andersson(a)scilifelab.se Phone: 0769469872 Mailing address: Science For Life Laboratory Alpha 6 Box 1031 171 21 Solna Visiting address: Tomtebodavägen 23A 171 65 Solna

3 3

Re: [cctbxbb] [git/cctbx] master: libtbx.clean_clutter runs isort in supported locations (612869130)
by Aaron Brewster 26 Oct '17

26 Oct '17

Hi Markus, could isort be added as a dependency instead? If I understand this commit right, when running libtbx.clean_clutter, if .isort.cfg is present and isort is not installed, pip is invoked to install isort. l'm nervous about pip being imported anywhere in our code, especially to install things when running code unrelated to installation. It's important to keep installation of stuff separate from running of stuff. Otherwise, it's impossible to keep dependencies straight. -Aaron On Thu, Oct 26, 2017 at 2:29 AM, CCTBX Commit via DLS Jenkins < graeme.winter(a)gmail.com> wrote: > Repository : ssh://g18-sc-serv-04.diamond.ac.uk/cctbx > On branch : master > > ------------------------------ > > > commit 612869130a6210d49a573a3cff04416a9de67808 > Author: Markus Gerstel <markus.gerstel(a)diamond.ac.uk> > Date: Thu Oct 26 10:29:05 2017 +0100 > > libtbx.clean_clutter runs isort in supported locations > > > ------------------------------ > > > 612869130a6210d49a573a3cff04416a9de67808 > libtbx/command_line/clean_clutter.py | 26 ++++++++++++++++++++++++++ > 1 file changed, 26 insertions(+) > > diff --git a/libtbx/command_line/clean_clutter.py > b/libtbx/command_line/clean_clutter.py > index d49bf4c25..8fc2e53c8 100644 > --- a/libtbx/command_line/clean_clutter.py > +++ b/libtbx/command_line/clean_clutter.py > @@ -32,6 +32,24 @@ def clean_clutter_in(files, tabsize=8): > os.remove(tmpname) > raise > > +def isort(path): > + try: > + import mock > + from isort.main import main > + except ImportError: > + # Install package if necessary > + import pip > + pip.main(['install', 'isort', 'mock']) > + import mock > + from isort.main import main > + with mock.patch.object(sys, 'argv', ['isort', '-y', '-ac', '-vb']): > + oldcwd = os.getcwd() > + try: > + os.chdir(path) > + main() > + finally: > + os.chdir(oldcwd) > + > def run(): > opt_parser = (option_parser( > usage=""" > @@ -60,6 +78,7 @@ by running svn commit.""") > if co.committing and files: > opt_parser.show_help() > exit(1) > + run_isort_in_path = False > if co.committing: > try: > files = list(subversion.marked_for_commit()) > @@ -72,7 +91,14 @@ by running svn commit.""") > else: dir = files[0] > files = [ c.path for c in libtbx.file_clutter.gather([dir]) > if c.is_cluttered(flag_x=False) ] > + if os.path.exists(os.path.join(dir, '.isort.cfg')): > + run_isort_in_path = dir > clean_clutter_in(files, tabsize=co.tabsize) > + if run_isort_in_path: > + try: > + isort(run_isort_in_path) > + except Exception as e: > + print("Did not run isort (%s)" % str(e)) > > if (__name__ == "__main__"): > import sys >

2 2

Re: [cctbxbb] Scons for python3 released
by markus.gerstel＠diamond.ac.uk 20 Oct '17

20 Oct '17

That's 3 weeks from now. If we do a 3 week freeze for every DIALS release, plus for phenix releases etc, cctbx would be trapped in perma-frost :) Feel free to start on it now - if it causes issues I can always pick a stable point in the past to branch off from. -Markus On 10 Oct 2017 6:34 pm, Billy Poon <bkpoon(a)lbl.gov> wrote: Great! There's a DIALS release on October 31, so I can start making changes after that. -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org On Tue, Oct 10, 2017 at 1:54 AM, R. D. Oeffner <rdo20(a)cam.ac.uk<mailto:[email protected]>> wrote: http://scons.org/scons-300-is-available.html Perhaps start using this to easy upgrading cctbx to python 3 -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html<http://www.cimr.cam.ac.uk/investigators/read/index.html> tel: +44(0)1223 763234 _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org<mailto:[email protected]> http://phenix-online.org/mailman/listinfo/cctbxbb -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

5 6

Update to allsvn script?
by Graeme.Winter＠diamond.ac.uk 20 Oct '17

20 Oct '17

Folks, phenix_regression/allsvn assumes if you are using git that you are using git/svn These days probably not true ;o) Viz: cd cctbx_project; git svn rebase Unable to determine upstream SVN information from working tree history ccp4io: At revision 160. ccp4io_adaptbx: At revision 142. cd dials; git svn rebase Unable to determine upstream SVN information from working tree history someone on phenix-land want to have a look at this? Obviously not a blocker but is untidy! Thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 2

Re: [cctbxbb] [git/cctbx] master: Add --print_statement checking capability to libtbx.find_clutter (d499b973b)
by Aaron Brewster 19 Oct '17

19 Oct '17

Hi, should this have been from __future__ import print_function ? Thanks, -Aaron On Thu, Oct 19, 2017 at 1:33 PM, CCTBX Commit via DLS Jenkins < graeme.winter(a)gmail.com> wrote: > Repository : ssh://g18-sc-serv-04.diamond.ac.uk/cctbx > On branch : master > > ------------------------------ > > > commit d499b973be4c0443e607fbad13e495bb450938a3 > Author: Markus Gerstel <markus.gerstel(a)diamond.ac.uk> > Date: Thu Oct 19 21:33:54 2017 +0100 > > Add --print_statement checking capability to libtbx.find_clutter > > > ------------------------------ > > > d499b973be4c0443e607fbad13e495bb450938a3 > libtbx/command_line/find_clutter.py | 12 +++++++++++- > libtbx/file_clutter.py | 23 +++++++++++++++++++---- > 2 files changed, 30 insertions(+), 5 deletions(-) > > diff --git a/libtbx/command_line/find_clutter.py > b/libtbx/command_line/find_clutter.py > index 27baa6d7b..66f4b46eb 100644 > --- a/libtbx/command_line/find_clutter.py > +++ b/libtbx/command_line/find_clutter.py > @@ -13,6 +13,7 @@ def run(args): > only_dos = False > only_future = False > flag_absolute_import = False > + flag_print_statement = False > # > paths = [] > for arg in args: > @@ -34,6 +35,8 @@ def run(args): > only_future = True > elif (arg == "--absolute_import"): > flag_absolute_import = True > + elif (arg == "--print_statement"): > + flag_print_statement = True > else: > paths.append(arg) > if (len(paths) == 0): paths = ["."] > @@ -45,9 +48,12 @@ def run(args): > n_too_many_from_future_import_division = 0 > n_missing_from_future_import_absolute_import = 0 > n_too_many_from_future_import_absolute_import = 0 > + n_missing_from_future_import_print_statement = 0 > + n_too_many_from_future_import_print_statement = 0 > n_bad_indentation = 0 > for info in gather(paths=paths, find_unused_imports=not flag_ni, > - find_bad_indentation=flag_indentation, flag_absolute_import=flag_ > absolute_import): > + find_bad_indentation=flag_indentation, flag_absolute_import=flag_ > absolute_import, > + flag_print_statement=flag_print_statement): > if (info.is_cluttered(flag_x=flag_x)): > n_is_cluttered += 1 > if (info.n_bare_excepts > 0): > @@ -62,6 +68,10 @@ def run(args): > n_missing_from_future_import_absolute_import += 1 > elif info.n_from_future_import_absolute_import > 1: > n_too_many_from_future_import_absolute_import += 1 > + if info.n_from_future_import_print_statement == 0: > + n_missing_from_future_import_print_statement += 1 > + elif info.n_from_future_import_print_statement > 1: > + n_too_many_from_future_import_print_statement += 1 > if (info.bad_indentation is not None) and (flag_indentation) : > n_bad_indentation += 1 > info.show( > diff --git a/libtbx/file_clutter.py b/libtbx/file_clutter.py > index 8af972751..768f06491 100644 > --- a/libtbx/file_clutter.py > +++ b/libtbx/file_clutter.py > @@ -11,9 +11,12 @@ class file_clutter(object): > '^ from [ ]+ __future__ [ ]+ import [ \w,]+ division', re.VERBOSE) > from_future_import_absolute_import_pat = re.compile( > '^ from [ ]+ __future__ [ ]+ import [ \w,]+ absolute_import', > re.VERBOSE) > + from_future_import_print_statement_pat = re.compile( > + '^ from [ ]+ __future__ [ ]+ import [ \w,]+ print_statement', > re.VERBOSE) > > def __init__(self, path, find_unused_imports=False, > - find_bad_indentation=True, flag_absolute_import=False): > + find_bad_indentation=True, flag_absolute_import=False, > + flag_print_statement=False): > self.path = path > self.is_executable = os.access(path, os.X_OK) > self.dos_format = False > @@ -25,6 +28,8 @@ class file_clutter(object): > self.n_from_future_import_division = None > self.flag_absolute_import = flag_absolute_import > self.n_from_future_import_absolute_import = None > + self.flag_print_statement = flag_print_statement > + self.n_from_future_import_print_statement = None > self.bad_indentation = None > self.file_should_be_empty = False > bytes = open(path, "rb").read() > @@ -45,6 +50,7 @@ class file_clutter(object): > if (path.endswith(".py")): > self.n_from_future_import_division = 0 > self.n_from_future_import_absolute_import = 0 > + self.n_from_future_import_print_statement = 0 > py_lines = bytes.splitlines() > self.file_should_be_empty = True > for line in py_lines: > @@ -54,6 +60,8 @@ class file_clutter(object): > self.n_from_future_import_division += 1 > if self.from_future_import_absolute_import_pat.search(line): > self.n_from_future_import_absolute_import += 1 > + if self.from_future_import_print_statement_pat.search(line): > + self.n_from_future_import_print_statement += 1 > ls = line.strip() > if ( ls.startswith("except") > and ls[6:].strip().startswith(":") > @@ -103,7 +111,8 @@ class file_clutter(object): > if (self.has_unused_imports()): > sapp("unused imports=%d" % len(self.unused_imports)) > if self.file_should_be_empty: > - if self.n_from_future_import_division == 0 and > self.n_from_future_import_absolute_import == 0: > + if self.n_from_future_import_division == 0 and > self.n_from_future_import_absolute_import == 0 and \ > + self.n_from_future_import_print_statement: > sapp("file is empty, should be 0 byte file") > else: > sapp("file contains only 'from __future__ import' and should be > empty instead") > @@ -116,6 +125,11 @@ class file_clutter(object): > sapp("missing 'from __future__ import absolute_import'") > elif self.n_from_future_import_absolute_import > 1: > sapp("more than one appearance of 'from __future__ import > absolute_import'") > + if self.flag_print_statement and not self.file_should_be_empty: > + if self.n_from_future_import_print_statement == 0: > + sapp("missing 'from __future__ import print_statement'") > + elif self.n_from_future_import_print_statement > 1: > + sapp("more than one appearance of 'from __future__ import > print_statement'") > if (self.bad_indentation is not None) and (flag_indentation) : > n_tab, n_space = self.bad_indentation > sapp("non-standard indentation: %d space, %d tab" % (n_space, > n_tab)) > @@ -144,11 +158,12 @@ def is_text_file(file_name): > if (name.endswith(extension)): return True > return False > > -def gather(paths, find_unused_imports=False, find_bad_indentation=False, > flag_absolute_import=False): > +def gather(paths, find_unused_imports=False, find_bad_indentation=False, > flag_absolute_import=False, flag_print_statement=False): > clutter = [] > def capp(): > clutter.append(file_clutter(path, find_unused_imports, > - find_bad_indentation=find_bad_indentation, > flag_absolute_import=flag_absolute_import)) > + find_bad_indentation=find_bad_indentation, > flag_absolute_import=flag_absolute_import, > + flag_print_statement=flag_print_statement)) > for path in paths: > if (not os.path.exists(path)): > print >> sys.stderr, "No such file or directory:", path >

2 1

Scons for python3 released
by R. D. Oeffner 10 Oct '17

10 Oct '17

http://scons.org/scons-300-is-available.html Perhaps start using this to easy upgrading cctbx to python 3 -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234

2 1

Re: [cctbxbb] [git/cctbx] master: Comment out simage test for moment as no one interested (apparently) in fixing test (767a45560)
by Nicholas Sauter 05 Oct '17

05 Oct '17

Graeme, Not true that we are not interested in fixing it. Billy just diagnosed the problem as of yesterday. I will provide a fix. Nick Nicholas K. Sauter, Ph. D. Senior Scientist, Molecular Biophysics & Integrated Bioimaging Division Lawrence Berkeley National Laboratory 1 Cyclotron Rd., Bldg. 33R0345 Berkeley, CA 94720 (510) 486-5713 On Wed, Oct 4, 2017 at 10:57 PM, CCTBX Commit via DLS Jenkins < graeme.winter(a)gmail.com> wrote: > Repository : ssh://g18-sc-serv-04.diamond.ac.uk/cctbx > On branch : master > > ------------------------------ > > > commit 767a45560e5f5816d77e9f82e3f846a724967e81 > Author: graeme-winter <graeme.winter(a)gmail.com> > Date: Thu Oct 5 06:57:25 2017 +0100 > > Comment out simage test for moment as no one interested (apparently) > in fixing test > > > ------------------------------ > > > 767a45560e5f5816d77e9f82e3f846a724967e81 > rstbx/run_tests.py | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/rstbx/run_tests.py b/rstbx/run_tests.py > index ea9de29e1..abaf74b4c 100644 > --- a/rstbx/run_tests.py > +++ b/rstbx/run_tests.py > @@ -19,7 +19,7 @@ def run(): > print "OK" > > tst_list = ( > - "$D/simage/tst.py", > + #"$D/simage/tst.py", > "$D/dps_core/tst_iotbx_converter.py", > "$D/diffraction/tst_predict.py", > "$D/diffraction/tst_ewald_sphere.py", >

1 0

SQLite extension library for calculating histograms
by R.D. Oeffner 04 Oct '17

04 Oct '17

Hi, Although peripheral to most developers using the CCTBX I would like to announce an extension library for SQLite that allows creating histogram tables, tables of average values of histogram bins as well as tables of ratios between histogram bins from existing SQLite database files. https://github.com/Oeffner/SQLiteHistograms In our group we often use SQLite databases for statistical analysis of results. This extension library permits you to create histograms of an existing table with one line of SQL. Regards, Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234

2 1

Compiling CCTBX with newer GCC on Mac
by Lance Westerhoff 04 Oct '17

04 Oct '17

Hello All- How does one introduce an alternate gcc compiler for CCTBX on Mac? I must be missing it in the documentation. I don’t want to use the old one in /usr/bin (and instead I want to you my newer one in /opt/gcc). I’ve tried all of the standard methods used by other build systems (change the $PATH, set $CXX and $CC, etc) but to no avail. These environment variables seem to be disregarded. We do use this compiler for building CCTBX on linux without too much trouble, so I think the support should be there. Specifically: mac-build64:build jenkins$ env | grep CXX CXX=/opt/gcc/current/bin/g++ mac-build64:build jenkins$ env | grep CC CC=/opt/gcc/current/bin/gcc mac-build64:build jenkins$ which gcc /opt/gcc/current/bin/gcc mac-build64:build jenkins$ g++ -v Using built-in specs. COLLECT_GCC=g++ COLLECT_LTO_WRAPPER=/opt/gcc/gcc-6.3.0-08092017/libexec/gcc/x86_64-apple-darwin11.4.2/6.3.0/lto-wrapper Target: x86_64-apple-darwin11.4.2 Configured with: ../gcc-6.3.0/configure --prefix=/opt/gcc/gcc-6.3.0-08092017 --enable-languages=c,c++,fortran,objc --with-gmp=/opt/gcc/gcc-6.3.0-08092017 --with-mpfr=/opt/gcc/gcc-6.3.0-08092017 --with-mpc=/opt/gcc/gcc-6.3.0-08092017 --enable-libgomp --disable-multilib Thread model: posix gcc version 6.3.0 (GCC) mac-build64:build jenkins$ /opt/python/current/bin/python ../modules/cctbx_project/libtbx/configure.py cctbx Python: 2.7.9 "/opt/python/current/bin/python" Compiler: default .... This script is only applicable to Linux - exiting. Done. mac-build64:build jenkins$ make ./bin/libtbx.scons -j "`./bin/libtbx.show_number_of_processors`" scons: Reading SConscript files ... On MacOS, using clang 4.2.0 Xcode 4.6.3 Build version 4H1503 ... Thanks for your help! -Lance ____________________ Lance M. Westerhoff, Ph.D. President and General Manager QuantumBio Inc. WWW: http://www.quantumbioinc.com Email: lance(a)quantumbioinc.com

3 7

Should we enable boost threads in bootstrap?
by Graeme.Winter＠diamond.ac.uk 28 Sep '17

28 Sep '17

Afternoon all, Should we do this? Any opinions? Could be useful for threads in a semi-portable way... Thanks & cheerio Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

7 16

Should we enable boost threads in bootstrap?
by Tristan Croll 27 Sep '17

27 Sep '17

Sorry for not replying directly to the existing thread, but I've only just subscribed to cctbxbb. Since I started the discussion of the Global Interpreter Lock (GIL) yesterday, I thought I should give a quick run-down of why it's important and where/how it can be released. In brief, the much-maligned GIL exists because much of the Python API proper (in particular, reference counting) is not thread-safe. Its purpose is to ensure that only one thread can ever be using Python objects at any given time. In practice this means that a naive implementation of Python threads gives you the worst of all possible worlds: parallel logic, but slower-than-single-threaded performance. Python regularly tries to swap between all threads it has running, but only the one that currently holds the GIL can go forward. So if one thread runs a method that takes 10 seconds without releasing the GIL, *all* threads will hang for 10 seconds. To be clear, this *doesn't* prevent you from using C++ threads, as long as those threads are not acting on Python objects. The thing is, in most production code the heavy-duty computation is *not* done on Python objects - it's done in C++ on objects that Python simply holds a pointer to. All such functions can safely release the GIL as long as they reacquire it before returning to Python. In the context of the example above, that means all other threads are able to continue on doing their own thing while that 10-second function is running. So why use threads rather than multiprocessing? Three key reasons: - threads are trivial to set up and run in the same way on Linux, macOS and Windows - threads share memory by default, making life much easier when they need to communicate regularly - It turns out that OpenCL/CUDA do not play at all well with forked processes. A real-world example: in ISOLDE I'm currently bringing together two key packages with their own Python APIs: ChimeraX for the molecular graphics, OpenMM for MD simulation. Since this is an interactive application, speed is of the essence. Graphics performance needs to be independent of simulation performance (to allow the user to rotate/translate etc. smoothly no matter how fast the simulation runs) so parallelism is mandatory. There is constant back-and-forth communication needed (the simulation needs to update coordinates; interactions need to be sent back to the simulation), which easier/faster in shared memory. The simulation is running on a GPU. My initial implementation used Python's multiprocessing module applying os.fork() - this worked in Linux under the very specific circumstances where the GPU had not previously been used for OpenCL or CUDA by the master process, but failed on the Mac and is of course impossible on Linux. Switching from multiprocessing to threading (with no other changes to my code) gave me an implementation that works equally well on Linux and Mac with no OS-specific code, and should work just as well on Windows once I get around to doing a build. The performance is effectively equal to what I was getting from multiprocessing, since all the major libraries I'm using (ChimeraX, OpenMM, NumPy) release the GIL in all their C++ calls. At the moment, though, CCTBX functions don't - but there's no reason why they can't. See https://wiki.python.org/moin/boost.python/HowTo#Multithreading_Support_for_…. A simple toy example using Python threads to speed up a heavyish computation: from multiprocessing.pool import ThreadPool import numpy from math import ceil from time import sleep def f(a, target, start_i, end_i): arr = a[start_i:end_i] target[start_i:end_i] = numpy.exp(numpy.cos(arr)+numpy.sin(arr)) def test_threads(a, num_threads): l = len(a) ret = numpy.empty(l, a.dtype) stride = int(ceil(l/num_threads)) with ThreadPool(processes=num_threads) as p: for i in range(num_threads): start = stride*i end = stride*(i+1) if end > l: end = l p.apply_async(f, (a, ret, start, end)) p.close() p.join() return ret a = numpy.random.rand(50000000) target = numpy.empty(len(a), a.dtype) %timeit f(a, target, 0, len(a)) 2.35 s ± 102 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) %timeit test_threads(a, 1): 2.55 s ± 30.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) %timeit test_threads(a, 2): 1.5 s ± 5.69 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) %timeit test_threads(a, 3): 1.2 s ± 111 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

1 1

rstbx test is failing without indication in test_all_parallel
by Oleg Sobolev 22 Sep '17

22 Sep '17

Dear colleagues, Accidentally I found out that this command: rstbx.simage.solver d_min=5 lattice_symmetry=P422 intensity_symmetry=P4 index_and_integrate=True multiprocessing=True finishes with traceback. It is run as part of phenix_regression/misc/tst_rstbx.csh The second problem is that test_all_parallel script fails to detect an actual failure. It would be great if somebody who is in charge of rstbx and testing could look at this. Best regards, Oleg Sobolev.

7 9

Massive commit just now
by Graeme.Winter＠diamond.ac.uk 08 Sep '17

08 Sep '17

Folks, Earlier a branch which was fairly long lived was merged in perhaps a non-ideal way, apparently replaying every commit from around July to now. While this is confusing and has (a little) messed with history, it has not actually broken anything... However, if you have a big branch you want to merge, I think a pull request makes this a little more obvious Just some background to what happened... Cheers Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 3

rna3p error
by Haiguang Liu 30 Aug '17

30 Aug '17

Dear All, I recently run into an error while processing DNA molecules using pdb_interpreter. The error message is as the following: File "/Users/hgliu/test/cctbx/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 5419, in __init__ restraints_loading_flags=restraints_loading_flags) File "/Users/hgliu/test/cctbx/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 3221, in __init__ log=log, File "/Users/hgliu/test/cctbx/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 2507, in __init__ m_j=mm, File "/Users/hgliu/test/cctbx/modules/cctbx_project/mmtbx/monomer_library/pdb_interpretation.py", line 1693, in get_lib_link return mon_lib_srv.link_link_id_dict["rna3p"] KeyError: 'rna3p' After searching through the threads, I found the same error has been reported, but no solution posted. Any suggestion is appreciated. Best, haiguang

2 2

Berkeley servers are back in service
by Billy Poon 03 Aug '17

03 Aug '17

Hi everybody, The servers at Berkeley are now available. Please let us know of any issues. Thanks! -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

1 0

Emergency shutdown
by Nigel Moriarty 02 Aug '17

02 Aug '17

Folks Phenix servers are shutting down due to a grass fire in the area. Cheers Nigel --- Nigel W. Moriarty Building 33R0349, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709 Email : NWMoriarty(a)LBL.gov Fax : 510-486-5909 Web : CCI.LBL.gov

1 0

Changing boost versions
by Nicholas Devenish 01 Aug '17

01 Aug '17

Hi All, Graeme's thread reminded me to ask about something I've been struggling with; is there a built-in/easy way to specify a different installation of boost to use? The build seems to construct lib boost_python manually, for some reason, so I don't know how easy it is to disentangle. Thanks, Nick

3 2

Pillow has landed
by markus.gerstel＠diamond.ac.uk 25 Jul '17

25 Jul '17

Hello everyone, Bootstrap now installs Pillow as the new replacement library of the Python Imaging Library (PIL), which has been deprecated/abandonware for a couple of years now. You can update your local installation by running libtbx.python -m pip install Pillow I ran tests locally, and they appeared to run fine. Although this is pretty much a drop-in replacement there may be some import statements which need updating. If you do come across problems you may find this to be of use: https://pillow.readthedocs.io/en/4.2.x/porting.html -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

smtbx.refine not working correctly?
by Graeme.Winter＠diamond.ac.uk 20 Jul '17

20 Jul '17

Not sure who to ask... Tried smtbx.refine from latest phenix & dials builds, both give [gw56@cs03r-sc-serv-16 Refine]$ smtbx.refine lcy.ins lcy.hkl lcy_smtbx.res Traceback (most recent call last): File "/dls_sw/apps/dials/dials-v1-6-4/build/../modules/cctbx_project/smtbx/command_line/refine.py", line 152, in <module> run(args, options) File "/dls_sw/apps/dials/dials-v1-6-4/build/../modules/cctbx_project/smtbx/command_line/refine.py", line 67, in run strictly_shelxl=False) File "/dls_sw/apps/dials/dials-v1-6-4/modules/cctbx_project/smtbx/refinement/__init__.py", line 10, in from_shelx return _(cls, *args, **kwds) File "/dls_sw/apps/dials/dials-v1-6-4/modules/cctbx_project/iotbx/shelx/__init__.py", line 59, in smtbx_refinement_model_from builder.weighting_scheme, AttributeError: 'smtbx_builder' object has no attribute 'weighting_scheme' [gw56@cs03r-sc-serv-16 Refine]$ module load phenix/1.12 Setting up phenix environment and PATH... [gw56@cs03r-sc-serv-16 Refine]$ smtbx.refine lcy.ins lcy.hkl lcy_smtbx.res Traceback (most recent call last): File "/dls_sw/apps/phenix/1.12/phenix-1.12-2829/build/../modules/cctbx_project/smtbx/command_line/refine.py", line 152, in <module> run(args, options) File "/dls_sw/apps/phenix/1.12/phenix-1.12-2829/build/../modules/cctbx_project/smtbx/command_line/refine.py", line 67, in run strictly_shelxl=False) File "/dls_sw/apps/phenix/1.12/phenix-1.12-2829/modules/cctbx_project/smtbx/refinement/__init__.py", line 10, in from_shelx return _(cls, *args, **kwds) File "/dls_sw/apps/phenix/1.12/phenix-1.12-2829/modules/cctbx_project/iotbx/shelx/__init__.py", line 59, in smtbx_refinement_model_from builder.weighting_scheme, AttributeError: 'smtbx_builder' object has no attribute 'weighting_scheme' who's best to ask about this? Thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 3

New error appeared recently, do not know who it belongs to...
by Graeme.Winter＠diamond.ac.uk 17 Jul '17

17 Jul '17

Exhibit A: libtbx.python "/Users/graeme/svn/cctbx/modules/cctbx_project/rstbx/simage/tst.py" rstbx.simage.explore_completeness d_min=10 rstbx.simage.explore_completeness d_min=10 intensity_symmetry=P4 use_symmetry=True multiprocessing=True rstbx.simage.solver d_min=5 rstbx.simage.solver d_min=5 lattice_symmetry=R32:R intensity_symmetry=R3:R rstbx.simage.solver d_min=5 lattice_symmetry=R32:R intensity_symmetry=P1 rstbx.simage.solver d_min=5 lattice_symmetry=P422 intensity_symmetry=P4 index_and_integrate=True multiprocessing=True Traceback (most recent call last): File "/Users/graeme/svn/cctbx/modules/cctbx_project/rstbx/simage/tst.py", line 273, in <module> run(args=sys.argv[1:]) File "/Users/graeme/svn/cctbx/modules/cctbx_project/rstbx/simage/tst.py", line 269, in run exercise_solver() File "/Users/graeme/svn/cctbx/modules/cctbx_project/rstbx/simage/tst.py", line 255, in exercise_solver "index_and_integrate=True", "multiprocessing=True"]) File "/Users/graeme/svn/cctbx/modules/cctbx_project/rstbx/simage/tst.py", line 213, in run command=cmd, stdout_splitlines=False).raise_if_errors().stdout_buffer File "/Users/graeme/svn/cctbx/modules/cctbx_project/libtbx/easy_run.py", line 39, in raise_if_errors raise Error(msg) RuntimeError: child process stderr output: command: 'rstbx.simage.solver d_min=5 lattice_symmetry=P422 intensity_symmetry=P4 index_and_integrate=True multiprocessing=True' Traceback (most recent call last): File "/Users/graeme/svn/cctbx/build/../modules/cctbx_project/rstbx/command_line/simage.solver.py", line 5, in <module> run(args=sys.argv[1:]) File "/Users/graeme/svn/cctbx/modules/cctbx_project/rstbx/simage/solver.py", line 1161, in run return run_fresh(args) File "/Users/graeme/svn/cctbx/modules/cctbx_project/rstbx/simage/solver.py", line 1145, in run_fresh process(work_params, i_calc) File "/Users/graeme/svn/cctbx/modules/cctbx_project/rstbx/simage/solver.py", line 1088, in process process_core(work_params, i_calc.p1_anom, reindexing_assistant, image_mdls) File "/Users/graeme/svn/cctbx/modules/cctbx_project/rstbx/simage/solver.py", line 985, in process_core work_params, reindexing_assistant, image_mdls, usables) File "/Users/graeme/svn/cctbx/modules/cctbx_project/rstbx/simage/solver.py", line 883, in build_image_cluster raise RuntimeError("Insufficient connectivity between images.") RuntimeError: Insufficient connectivity between images. usr+sys time: 0.78 seconds, ticks: 777467, micro-seconds/tick: 1.003 wall clock time: 6.99 seconds OS X & RHEL6 Any volunteers? Cheers Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 5

Re: [cctbxbb] duplicate spam
by Aaron Brewster 10 Jul '17

10 Jul '17

More commit spam incoming. This is from merging a bunch of commits from an LCLS beamtime into the main branch. Sorry for the flood. -Aaron On Tue, Jun 27, 2017 at 12:30 AM, Markus Gerstel < markus.gerstel(a)diamond.ac.uk> wrote: > Due to rewriting git history there are now a number of duplicate mails > coming through here. > > Apologies. > > > > -Markus > > > > -- > > This e-mail and any attachments may contain confidential, copyright and or > privileged material, and are for the use of the intended addressee only. If > you are not the intended addressee or an authorised recipient of the > addressee please notify us of receipt by returning the e-mail and do not > use, copy, retain, distribute or disclose the information in or attached to > the e-mail. > Any opinions expressed within this e-mail are those of the individual and > not necessarily of Diamond Light Source Ltd. > Diamond Light Source Ltd. cannot guarantee that this e-mail or any > attachments are free from viruses and we cannot accept liability for any > damage which you may sustain as a result of software viruses which may be > transmitted in or with the message. > Diamond Light Source Limited (company no. 4375679). Registered in England > and Wales with its registered office at Diamond House, Harwell Science and > Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom > >

2 2

Building cctbx & friends with intel compilers
by Graeme.Winter＠diamond.ac.uk 29 Jun '17

29 Jun '17

Greetings all, Do we have a recipe for compiling cctbx etc with icc? To pick a random example, this version: icc (ICC) 17.0.4 20170411 Copyright (C) 1985-2017 Intel Corporation. All rights reserved. Thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 2