- cctbxbb - phenix-online.org

scitbx without numpy - still supported?
by Nicholas Devenish 24 Apr '18

24 Apr '18

Hi All, I've been tweaking the code to allow versions of boost higher than 1.63, which has involved poking at the scitbx-numpy bridge code. There seems to be some level of branching in the code for supporting building without numpy - but on the other hand numpy seems to be used exclusively for all flex.type.__init__ functions that I can find? Is building without numpy still supported? Thanks, Nick

5 6

Misuse of libtbx/configure.py
by Robert Oeffner 09 Apr '18

09 Apr '18

Hi, I'm not sure when this has been put into configure.py as I have had taken my eyes of the ball lately regarding Windows builds. But it seems that during bootstrap build the configure.py step is now doing more than just configuring modules. On my PC the excerpt below from stdout shows an error during the configure.py step Processing: "c:\busers\oeffner\nightlybuild\x32\current\modules\dials\libtbx_refresh.py" requires mock>=2.0, has 2.0.0 requirement orderedset is not currently met, package not installed attempting install of orderedset... Collecting orderedset Using cached orderedset-2.0.1.tar.gz Installing collected packages: orderedset Running setup.py install for orderedset: started Running setup.py install for orderedset: finished with status 'error' Complete output from command C:\Busers\oeffner\NightlyBuild\x32\Current\base\bin\python\python.exe -u -c "import setuptools, tokenize;__file__='c:\\users\\oeffner\\appdata\\local\\temp\\pip-build-smhifw\\orderedset\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record c:\users\oeffner\appdata\local\temp\pip-0seb8q-record\install-record.txt --single-version-externally-managed --compile: running install running build running build_py creating build creating build\lib.win32-2.7 creating build\lib.win32-2.7\orderedset copying lib\orderedset\__init__.py -> build\lib.win32-2.7\orderedset running build_ext building 'orderedset._orderedset' extension error: Microsoft Visual C++ 9.0 is required (Unable to find vcvarsall.bat). Get it from http://aka.ms/vcpython27 ---------------------------------------- Command "C:\Busers\oeffner\NightlyBuild\x32\Current\base\bin\python\python.exe -u -c "import setuptools, tokenize;__file__='c:\\users\\oeffner\\appdata\\local\\temp\\pip-build-smhifw\\orderedset\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record c:\users\oeffner\appdata\local\temp\pip-0seb8q-record\install-record.txt --single-version-externally-managed --compile" failed with error code 1 in c:\users\oeffner\appdata\local\temp\pip-build-smhifw\orderedset\ You are using pip version 9.0.1, however version 9.0.3 is available. You should consider upgrading via the 'python -m pip install --upgrade pip' command. install failed. please check manually My first concern with this approach is that the bootstrap process doesn't stop. It carries on and eventually builds the Phenix software suite. This may or may not be a broken build as it is not clear whether the above failure to build orderedset was a showstopper for something. Secondly, I thought that modules would not be installed anywhere else than when building base components or when downloading sources. For a small project it is a nice and sneaky solution to have certain build steps second guessing what dependencies are missing and then installing them "on the fly", if not covertly. But for a large project such as CCTBX with several teams using that as a common framework this becomes messy I would argue. A quick and dirty fix for the above error might be just to put vcvarsall.bat in the PATH environment. But that is not a solution since bootstrap.py has been fashioned to use different versions of Visual C++, some of which are not invoked through a vcvarsall.bat file. I appreciate the need to install additional component modules for python. But would it not be more tidy to install these during the base step of the bootstrap process? Is there any reason why this cannot happen? Regards, Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234

5 5

simtbx - Pilatus example available
by James Holton 08 Apr '18

08 Apr '18

Took most of last week, but I finally got it to work. tst_nanoBragg_pilatus.py will create a Pilatus-like image, complete with bad pixels, window panes, thickness effects, module-to-module misalignment and 0.1 deg of phi rotation. It takes one optional argument, which is the image number in a sequence of 3600. I've processed this simulated data set with xia2 and XDS and found the anomalous signal has the right sign. This required some edits and additions to the simTBX code as well as FormatCBFMini.as_file() that I think fixed a beam center consistency bug. Now if you dxtbx.load() and FormatCBFMiniPilatus.as_file() you get the same beam center in both input and output headers. I also enhanced the header to make it look more like the ones in DIALS regression. A new "SIM.F000" value is now available for marking the "direct beam" on the image itself. I spent most of Friday and all of this morning trying to follow the check-in procedure we laid out during the DIALS workshop. Not sure how, but it suddenly started working just now. I believe I have checked in this code into the simtbx_sandbox branch of cctbx_project. Is Jenkins looking at this yet? Is anyone looking at this yet? -James Holton MAD Scientist

3 3

print dxtbx.model.Scan()
by James Holton 05 Apr '18

05 Apr '18

The following sequence of commands: % source build/setpaths.csh % libtbx.python >>> import dxtbx >>> from dxtbx import model >>> print dxtbx.model.Scan() seems to cause Python itself to core dump. Is this expected? show_stack(1): <stdin>(1) <module> libc backtrace (25 frames, most recent call last): /home/jamesh/projects/cctbx-dev/build/../base/bin/python2.7() [0x400659] /lib64/libc.so.6(__libc_start_main+0xfd) [0x34e961ed1d] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(Py_Main+0xc74) [0x7f93eb5c2b74] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyRun_AnyFileExFlags+0x4c) [0x7f93eb5ad78c] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyRun_InteractiveLoopFlags+0x4e) [0x7f93eb5ad04e] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyRun_InteractiveOneFlags+0x195) [0x7f93eb5ace35] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCode+0x32) [0x7f93eb58d4e2] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x88e) [0x7f93eb58d3ce] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x3ef0) [0x7f93eb589de0] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyFile_WriteObject+0x78) [0x7f93eb503b58] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(+0x958f2) [0x7f93eb5288f2] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyObject_Str+0xa) [0x7f93eb52879a] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(_PyObject_Str+0x7b) [0x7f93eb5286ab] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(+0xb5a1e) [0x7f93eb548a1e] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x43) [0x7f93eb584fd3] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f93eb4dc093] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(+0x5b6cf) [0x7f93eb4ee6cf] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f93eb4dc093] /home/jamesh/projects/cctbx-dev/build/lib/libboost_python.so(+0x2d088) [0x7f93e3f23088] /home/jamesh/projects/cctbx-dev/build/lib/libboost_python.so(boost::python::handle_exception_impl(boost::function0<void>)+0x43) [0x7f93e3f34013] /home/jamesh/projects/cctbx-dev/build/lib/libboost_python.so(+0x2fe08) [0x7f93e3f25e08] /home/jamesh/projects/cctbx-dev/build/lib/libboost_python.so(boost::python::objects::function::call(_object*, _object*) const+0xef) [0x7f93e3f251ff] /home/jamesh/projects/cctbx-dev/build/lib/dxtbx_model_ext.so(boost::python::objects::caller_py_function_impl<boost::python::detail::caller<std::string (*)(dxtbx::model::Scan const&), boost::python::default_call_policies, boost::mpl::vector2<std::string, dxtbx::model::Scan const&> > >::operator()(_object*, _object*)+0x83) [0x7f93db88a7a3] /home/jamesh/projects/cctbx-dev/build/lib/dxtbx_model_ext.so(dxtbx::model::boost_python::scan_to_string(dxtbx::model::Scan const&)+0x319) [0x7f93db871929] /lib64/libc.so.6() [0x34e9632510] Segmentation fault (Python and libc call stacks above)

2 1

Exceptions squashed by easy_mp (revenge of)
by Graeme.Winter＠Diamond.ac.uk 03 Apr '18

03 Apr '18

Folks, Following up on user reports again of errors within easy_mp - all that gets logged is “something went wrong” i.e. Using multiprocessing with 10 parallel job(s) Traceback (most recent call last): File "/home/user/bin/dials-installer/build/../modules/dials/command_line/integrate.py", line 613, in <module> halraiser(e) File "/home/user/bin/dials-installer/build/../modules/dials/command_line/integrate.py", line 611, in <module> script.run() File "/home/user/bin/dials-installer/build/../modules/dials/command_line/integrate.py", line 341, in run reflections = integrator.integrate() File "/home/user/bin/dials-installer/modules/dials/algorithms/integration/integrator.py", line 1214, in integrate self.reflections, _, time_info = processor.process() File "/home/user/bin/dials-installer/modules/dials/algorithms/integration/processor.py", line 271, in process preserve_exception_message = True) File "/home/user/bin/dials-installer/modules/dials/util/mp.py", line 171, in multi_node_parallel_map preserve_exception_message = preserve_exception_message) File "/home/user/bin/dials-installer/modules/dials/util/mp.py", line 53, in parallel_map preserve_exception_message = preserve_exception_message) File "/home/user/bin/dials-installer/modules/cctbx_project/libtbx/easy_mp.py", line 627, in parallel_map result = res() File "/home/user/bin/dials-installer/modules/cctbx_project/libtbx/scheduling/result.py", line 119, in __call__ self.traceback( exception = self.exception() ) File "/home/user/bin/dials-installer/modules/cctbx_project/libtbx/scheduling/stacktrace.py", line 115, in __call__ self.raise_handler( exception = exception ) File "/home/user/bin/dials-installer/modules/cctbx_project/libtbx/scheduling/mainthread.py", line 100, in poll value = target( *args, **kwargs ) File "/home/user/bin/dials-installer/modules/dials/util/mp.py", line 91, in __call__ preserve_exception_message = self.preserve_exception_message) File "/home/user/bin/dials-installer/modules/cctbx_project/libtbx/easy_mp.py", line 627, in parallel_map result = res() File "/home/user/bin/dials-installer/modules/cctbx_project/libtbx/scheduling/result.py", line 119, in __call__ self.traceback( exception = self.exception() ) File "/home/user/bin/dials-installer/modules/cctbx_project/libtbx/scheduling/stacktrace.py", line 86, in __call__ raise exception RuntimeError: Please report this error to dials-support(a)lists.sourceforge.net: exit code = -9 I forget why it was decided that keeping the proper stack trace was a bad thing, but could this be revisited? It would greatly help to see it in the output of the program (if as is the case here I do not have the user data) My email-fu is not strong enough to dig out the previous conversation Cheers Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 6

Re: [cctbxbb] [git/cctbx] master: New command: libtbx.install (914c74bf1)
by Pavel Afonine 13 Mar '18

13 Mar '18

Just out of curiosity: what's your definition of "real world"? Pavel On 3/13/18 03:49, CCTBX commit wrote: > Repository : ssh://g18-sc-serv-04.diamond.ac.uk/cctbx > On branch : master > > ------------------------------------------------------------------------ > > > commit 914c74bf195b5e5d4fa2c4145da3e5f89c7e5872 > Author: Markus Gerstel <markus.gerstel(a)diamond.ac.uk> > Date: Tue Mar 13 10:43:22 2018 +0000 > > New command: libtbx.install > > This downloads and configures from a select list of cctbx/libtbxish > modules in a similar manner to pip in the real world. This will not > update or install any libraries in the base build, but assuming > those libraries are present libtbx.install should work. > > > ------------------------------------------------------------------------ > > > 914c74bf195b5e5d4fa2c4145da3e5f89c7e5872 > libtbx/command_line/install.py | 118 > +++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 118 insertions(+) > > diff --git a/libtbx/command_line/install.py > b/libtbx/command_line/install.py > new file mode 100644 > index 000000000..47ad94eb5 > --- /dev/null > +++ b/libtbx/command_line/install.py > @@ -0,0 +1,118 @@ > +from __future__ import absolute_import, division, print_function > + > +import collections > +import os > +import sys > +from optparse import SUPPRESS_HELP, OptionParser > + > +import procrunner > + > +import libtbx.load_env > +from libtbx.auto_build.bootstrap import Toolbox > + > +# Basically 'pip' for selected libtbx/cctbx modules. > + > + > +def is_source_repository(path): > + return (path / '.git').isdir() or (path / '.svn').isdir() > + > +def run(args): > + parser = OptionParser(usage="libtbx.install [package]", > + description="Installs an additional cctbx > package") > + parser.add_option("-?", action="help", help=SUPPRESS_HELP) > + options, args = parser.parse_args(args) > + > + modules_directory = list(filter(lambda dir: not > is_source_repository(dir), libtbx.env.repository_paths)) > + if not modules_directory: > + sys.exit("No repository path candidate found. Can't install > modules without an installation root.") > + if len(modules_directory) > 1: > + print("More than one repository path candidate found.") > + installation_root = modules_directory[0] > + print("Using %s as installation root" % abs(installation_root)) > + > + packages_to_configure = set() > + > + errors = False > + for package in args: > + if (installation_root / package).isdir(): > + print("Skipping package %s: Directory already exists in > installation root" % package) > + errors = True > + continue > + if package not in warehouse: > + print("Skipping package %s: Never heard of this before" % package) > + errors = True > + continue > + > + downloaded = False > + for mech, call in mechanisms.items(): > + if mech in warehouse[package]: > + print("Attempting to obtain %s using %s..." % (package, > mech), end='') > + sys.stdout.flush() > + if call(abs(installation_root / package), > warehouse[package][mech]): > + assert (installation_root / package).isdir(), "Installation > failed" > + downloaded = True > + print("success") > + break > + else: > + assert not (installation_root / package).isdir(), "Install > mechanism %s did not fail cleanly" % mech > + print("failed") > + if not downloaded: > + print("Skipping package %s: Could not install" % package) > + errors = True > + continue > + > + packages_to_configure.add(package) > + > + if packages_to_configure: > + packages_to_configure = sorted(packages_to_configure) > + print("Configuring packages %s" % ", ".join(packages_to_configure)) > + os.chdir(abs(libtbx.env.build_path)) > + result = procrunner.run_process(['libtbx.configure'] + > packages_to_configure) > + if result['exitcode']: errors = True > + > + if errors: > + sys.exit(1) > + > +def install_git(location, source): > + try: > + result = procrunner.run_process(['git', 'clone', '--recursive', > source, location], print_stdout=False, print_stderr=False) > + if result['exitcode']: > + return False > + Toolbox.set_git_repository_config_to_rebase(os.path.join(location, > '.git', 'config')) > + return True > + except OSError: > + return False # git may not be installed > + > +def install_zip(location, source): > + os.mkdir(location) > + tempfile = os.path.join(location, '.tmp.zip') > + etagfile = os.path.join(location, '..tmp.zip.etag') > + def cleanup(): > + try: os.remove(tempfile) > + except OSError: pass > + try: os.remove(etagfile) > + except OSError: pass > + if Toolbox.download_to_file(source['url'], tempfile) <= 0: > + cleanup() > + os.rmdir(location) > + return False > + Toolbox.unzip(tempfile, location, trim_directory=source.get('trim', 0)) > + cleanup() > + return True > + > +mechanisms = collections.OrderedDict(( > + ('git-auth', install_git), > + ('git-anon', install_git), > + ('http-zip', install_zip), > +)) > + > +warehouse = { > + 'i19': { > + 'git-auth': '[email protected]:/xia2/i19', > + 'git-anon': 'https://github.com/xia2/i19.git', > + 'http-zip': { 'url': > 'https://github.com/xia2/i19/archive/master.zip', 'trim': 1 }, > + }, > +} > + > +if __name__ == '__main__': > + run(sys.argv[1:])

2 1

Re: [cctbxbb] [git/cctbx] master: rename test files, remove them from run_tests (23a4a6fe4)
by Pavel Afonine 05 Mar '18

05 Mar '18

Just to make sure: you are converting to use pytest this particular codes (fable), correct? Pavel P.S.: I'm allergic to pytest. On 3/3/18 07:46, CCTBX commit wrote: > This in preparation for pytestification.

5 8

Revisiting cctbx/iotbx/dxtbx dependencies
by Aaron Brewster 20 Feb '18

20 Feb '18

Hey folks, especially Rob and Luc. So after a quick change to iotbx/libtbx_config, it turns out there's a way to get a super minimum version of cctbx, without configuring dxtbx, and it takes advantage of existing mechanisms: - python bootstrap.py --builder=cctbx hot update - python modules/cctbx_project/libtbx/auto_build/install_base_packages.py openssl python python_compatibility certifi numpy cython png libsvm pytest pythonextra jinja2 junitxml biopython docutils sphinx pyopengl misc freetype matplotlib pillow gettext glib expat fontconfig render pixman tiff cairo gtk fonts wxpython - mkdir build; cd build - ../base/bin/python ../modules/cctbx_project/libtbx/configure.py cctbx iotbx - make Now, some details as to why this works. First, the cctbx builder in boostrap.py doesn't download cbflib so cbflib is never configured. Next, the base step of bootstrap.py is simply a call to install_base_packages.py with a list of required packages. You can just run it standalone and get the packages you need. Here, I ran "python bootstrap.py --builder=cctbx base", killed it after I saw the list of packages it was going to install, then ran install_base_packages manually after removing hdf5 and lz4 (which depends on hdf5) from the list. One could imagine removing other packages depending on your use case. With that, I could run configure.py directly. I can't use cctbx by itself as the cctbx/maptbx/SConscript file has a dependency on iotbx, but configuring cctbx and iotbx together seems to work. The key trick was to make iotbx only depend on dxtbx if dxtbx is explicitly configured. I used a keyword in libtbx_configure added by Luc a bit ago: optional_modules_only_if_explicit_request. I committed that change just now. Outstanding questions: 1. Is there interest in making this a builder? It would be straightforward to add a builder named cctbx_minimal which would just do the above steps 2. Is there further need for making cbflib and hdf5 optional in dxtbx? I purport there is not as the above procedure seems to satisfy the use case, but I can be persuaded otherwise. Thanks, -Aaron

4 5

Re: [cctbxbb] [git/cctbx] master: use larger tolerance (0b81ca799)
by Graeme.Winter＠diamond.ac.uk 12 Feb '18

12 Feb '18

Pavel, This commit did not fix test on OS X High Sierra libtbx.python "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/maptbx/tst_interpolation.py" [FAIL] 1.4s Time: 1.39 Return code: 1 OKs: 0 Standard error: Traceback (most recent call last): File "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/maptbx/tst_interpolation.py", line 128, in <module> exercise(f = f, gx = gx, gy = gy, gz = gz, function_type=function_type) File "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/maptbx/tst_interpolation.py", line 99, in exercise assert approx_equal(it, 0.0, 1.e-4) AssertionError cheers Graeme On 11 Feb 2018, at 00:14, CCTBX commit <diamondlightsource.jenkins(a)GMAIL.COM<mailto:[email protected]>> wrote: Repository : ssh://g18-sc-serv-03.diamond.ac.uk/cctbx On branch : master ________________________________ commit 0b81ca7992f9cf7730eb007aff6e8c30bd2ba71b Author: Pavel <pafonine(a)lbl.gov<mailto:[email protected]>> Date: Sat Feb 10 16:14:51 2018 -0800 use larger tolerance ________________________________ 0b81ca7992f9cf7730eb007aff6e8c30bd2ba71b cctbx/maptbx/tst_interpolation.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/cctbx/maptbx/tst_interpolation.py b/cctbx/maptbx/tst_interpolation.py index 918ba68fd..90dec8a46 100644 --- a/cctbx/maptbx/tst_interpolation.py +++ b/cctbx/maptbx/tst_interpolation.py @@ -96,15 +96,15 @@ def exercise(f,gx,gy,gz,function_type, eps = 1.e-6): if(function_type=="L"): cntr+=1 for it in [tl,gxl,gyl,gzl, tq,gxq,gyq,gzq, tt,gxt,gyt,gzt]: - assert approx_equal(it, 0.0) + assert approx_equal(it, 0.0, 1.e-4) if(function_type=="Q"): cntr+=1 for it in [tq,gxq,gyq,gzq, tt,gxt,gyt,gzt]: - assert approx_equal(it, 0.0) + assert approx_equal(it, 0.0, 1.e-4) if(function_type=="C"): cntr+=1 for it in [tt,gxt,gyt,gzt]: - assert approx_equal(it, 0.0) + assert approx_equal(it, 0.0, 1.e-4) if(abs(tl)>eps and abs(tq)>eps): assert tl >=tq if(function_type=="F"): cntr+=1 -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

Re: [cctbxbb] [Dials-support] indexing success rates
by Aaron Brewster 05 Feb '18

05 Feb '18

Right, typically if I don't know the cell, I'll index a bunch of images with no target, then use cluster.unit_cell on the results to get a consensus cell, then re-index with that cell. For dials.stills_process, the default of fft1d works pretty well generally, but to get the best results, indexing.method_list=fft1d,fft3d might be better. There, if the first method fails the second is tried. real_space_grid_search can also be used. Getting a specific image in a composite file (such as hdf5) isn't a use case I've hit often, so I don't have an easy solution there. It's possible an image range parameter to dials.import would be useful? -Aaron On Sat, Feb 3, 2018 at 1:02 AM, Graeme Winter <graeme.winter(a)gmail.com> wrote: > Hi Aaron > > (and DIALS) > > For stills / single images - should dials.index switch by magic to default > fft1d? > > Of is that the responsibility of e.g. xia2 / cctbx.xfel to worry about? > > It has to be said, more often than not, particularly for screening, we do > not know cell / symmetry a priori (at diamond anyways) > > A related question > > If I have a data block / strong pickle from 1000 stills, how can I index > e.g. image 672? Do I have to split this into 1000 separate things? > > @JH - thanks for this - I think some rigorous testing will be useful here. > > Cheers Graeme > > On 3 Feb 2018, at 08:37, Aaron Brewster <asbrewster(a)lbl.gov> wrote: > > Hi James, thanks. I looked at random image 15 and after trying a bunch of > stuff it indexes fine if I use indexing.method=fft3d. The default for > stills processing is fft1d, so I find this super interesting. > > For what it's worth, I pretty much never process stills without a target > cell. I generated ~30 random images using your script and indexed them, > then I got the unit cells with "dials.show *refined*.json | grep cell". > They were all about the same. So I re-indexed random image 15 using fft1d > but I also specified known_symmetry.cell=50,60,70,90,90,90 and it indexed > fine. > > So, either indexing.method=fft3d or known_symmetry.cell=50,60,70,90,90,90 > is sufficient to index random image 15. Certainly for experimental data I > would recommend using known_symmetry as the results are always better. > However, I'd like to look deeper at fft1d vs fft3d at some point. > > Cc'ing dials-support. > > Thanks! > -Aaron > > > > On Fri, Feb 2, 2018 at 1:35 PM, James Holton <jmholton(a)lbl.gov> wrote: > >> While testing the new simTBX diffraction image simulator Aaron and I >> found that some rather beautiful-looking images still won't index. >> Subsequently, I followed up with mosflm, which does succeed in these cases, >> even without prior knowledge of the cell or space group. >> >> I have now run this 1000 times, varying only the crystal orientation, and >> found 171 cases of dials.stills_process failing to index the simulated >> image whereas mosflm had no trouble, and 5 cases where mosflm failed and >> dials.stills_process succeeded. There were no overlapping cases where both >> programs failed. >> >> I have tarballed up the relevant files here: >> http://bl831.als.lbl.gov/~jamesh/bugreports/dials_index_020218.tgz >> >> The tarball contains a "runme.com" shell script for reproducing these >> results, and also the tst_nanoBragg_forindex.py jiffy for making a test >> image given a provided random-number seed. For example: >> >> libtbx.python ./tst_nanoBragg_forindex.py random 15 >> >> will create a file called noiseimage_001.cbf that dials.stills_process >> cannot index (at least in my hands). For those of you who don't have >> mosflm installed, the tarball incluses a copy, and the "autoindex.com" >> script should run on any linux system. >> >> Any ideas as to why there is this discrepancy? Feels like an opportunity >> for a non-trivial improvement. >> >> Something to ponder over the weekend, I suppose. Note that I recently >> checked in a few bug fixes to simTBX, but this analysis is unaffected by >> them, so you shouldn't have to update your build to reproduce this. >> >> Cheers, >> >> -James Holton >> MAD Scientist >> >> >> >> _______________________________________________ >> cctbxbb mailing list >> cctbxbb(a)phenix-online.org >> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> > ------------------------------------------------------------ > ------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot______ > _________________________________________ > DIALS-support mailing list > DIALS-support(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/dials-support > > >

2 1

indexing success rates
by James Holton 03 Feb '18

03 Feb '18

While testing the new simTBX diffraction image simulator Aaron and I found that some rather beautiful-looking images still won't index. Subsequently, I followed up with mosflm, which does succeed in these cases, even without prior knowledge of the cell or space group. I have now run this 1000 times, varying only the crystal orientation, and found 171 cases of dials.stills_process failing to index the simulated image whereas mosflm had no trouble, and 5 cases where mosflm failed and dials.stills_process succeeded. There were no overlapping cases where both programs failed. I have tarballed up the relevant files here: http://bl831.als.lbl.gov/~jamesh/bugreports/dials_index_020218.tgz The tarball contains a "runme.com" shell script for reproducing these results, and also the tst_nanoBragg_forindex.py jiffy for making a test image given a provided random-number seed. For example: libtbx.python ./tst_nanoBragg_forindex.py random 15 will create a file called noiseimage_001.cbf that dials.stills_process cannot index (at least in my hands). For those of you who don't have mosflm installed, the tarball incluses a copy, and the "autoindex.com" script should run on any linux system. Any ideas as to why there is this discrepancy? Feels like an opportunity for a non-trivial improvement. Something to ponder over the weekend, I suppose. Note that I recently checked in a few bug fixes to simTBX, but this analysis is unaffected by them, so you shouldn't have to update your build to reproduce this. Cheers, -James Holton MAD Scientist

2 1

Fwd: [git/cctbx] master: Allow compiling standalone statically linked executable of Phaser on Windows and MinGW platforms requires: static linking of some boost libraries, additional preprocessor defines for Windows build, additional library specified ...
by Graeme.Winter＠diamond.ac.uk 02 Feb '18

02 Feb '18

Tried sending before got rejected ... Begin forwarded message: From: Graeme Winter <graeme.winter(a)gmail.com<mailto:[email protected]>> Date: 1 February 2018 at 18:51:58 GMT To: cctbxbb(a)PHENIX-ONLINE.ORG<mailto:[email protected]> Cc: CCTBX-COMMIT(a)JISCMAIL.AC.UK<mailto:[email protected]> Subject: Re: [git/cctbx] master: Allow compiling standalone statically linked executable of Phaser on Windows and MinGW platforms requires: static linking of some boost libraries, additional preprocessor defines for Windows build, additional library specified for rstbx build (29fc06f71) Just after this commit all our builds died at diamond Is build bot happy? Curious if local issue Cheers Graeme On 1 Feb 2018, at 18:07, CCTBX commit <diamondlightsource.jenkins(a)GMAIL.COM<mailto:[email protected]>> wrote: Repository : ssh://g18-sc-serv-04.diamond.ac.uk/cctbx On branch : master ________________________________ commit 29fc06f714e1059cb9f4eaac35cc2024178237dc Author: Robert Oeffner <rdo20(a)cam.ac.uk<mailto:[email protected]>> Date: Thu Feb 1 18:07:49 2018 +0000 Allow compiling standalone statically linked executable of Phaser on Windows and MinGW platforms requires: static linking of some boost libraries, additional preprocessor defines for Windows build, additional library specified for rstbx build ________________________________ 29fc06f714e1059cb9f4eaac35cc2024178237dc boost_adaptbx/SConscript | 13 +++++++------ libtbx/SConscript | 4 +++- rstbx/SConscript | 2 +- 3 files changed, 11 insertions(+), 8 deletions(-) diff --git a/boost_adaptbx/SConscript b/boost_adaptbx/SConscript index 8696c1510..b8b58ce2d 100644 --- a/boost_adaptbx/SConscript +++ b/boost_adaptbx/SConscript @@ -32,8 +32,9 @@ print("BOOST_VERSION: %i" %env_etc.boost_version) # any issue. if (libtbx.env.build_options.enable_boost_threads and env_etc.boost_thread_support): - boost_thread_uses_winthreads = (sys.platform == 'win32' and - not env_etc.compiler.endswith('mingw')) + boost_thread_uses_winthreads = (sys.platform == 'win32') + # MinGW compiler 5.3.0 appears to compile boost_threads + # with the same condition as win32. # This environment is to be used latter to build Boost libraries # So we need a clean one without extra library linking. env_boost_shlibs_base = env_base.Copy() @@ -264,7 +265,7 @@ e.g. with "yum install python-dev" or "apt-get install python-dev". libs.append('rt') if boost_thread_uses_winthreads: libs.append('boost_chrono') - env_boost_threads.SharedLibrary( + env_boost_threads.StaticLibrary( target='#lib/boost_thread', source=boost_thread_src, LIBS=libs) @@ -280,7 +281,7 @@ e.g. with "yum install python-dev" or "apt-get install python-dev". # For Windows (c.f. point 2 above) env_boost_system.Append( CPPDEFINES={'BOOST_SYSTEM_DYN_LINK':1}) - env_boost_system.SharedLibrary( + env_boost_system.StaticLibrary( target='#lib/boost_system', source=boost_system_src) @@ -294,7 +295,7 @@ e.g. with "yum install python-dev" or "apt-get install python-dev". os.path.join(env_etc.boost_dist, 'libs', 'chrono', 'src', '*.cpp'))] # For Windows (c.f. point 2 above) env_boost_chrono.Append(CPPDEFINES={'BOOST_CHRONO_DYN_LINK':1}) - env_boost_chrono.SharedLibrary( + env_boost_chrono.StaticLibrary( target='#lib/boost_chrono', source=boost_chrono_src, LIBS=['boost_system']) @@ -308,7 +309,7 @@ e.g. with "yum install python-dev" or "apt-get install python-dev". env_etc.include_registry.append( env=env_boost_threads_test, paths=[env_etc.boost_adaptbx_include]) - env_boost_threads_test.SharedLibrary( + env_boost_threads_test.StaticLibrary( target="#lib/boost_adaptbx_boost_thread_test_ext", source="tests/boost_thread_test.cpp") diff --git a/libtbx/SConscript b/libtbx/SConscript index b4fcc5df9..b956956f2 100644 --- a/libtbx/SConscript +++ b/libtbx/SConscript @@ -364,7 +364,9 @@ For newer versions, please adjust this SConscript. "/D_SECURE_SCL=0", "/D_CRT_SECURE_NO_DEPRECATE", "/DNOMINMAX", - #"/DWIN32", + "/DWIN32", + "/DDLL_EXPORTS", + "/D_WIN32_WINNT=_WIN32_WINNT_WIN7", "/bigobj", # for heavy use of template libraries "/wd4996", # warning C4996: ... was declared deprecated "/wd4068", # warning C4068: unknown pragma diff --git a/rstbx/SConscript b/rstbx/SConscript index 422a9864c..76ba4d1c5 100644 --- a/rstbx/SConscript +++ b/rstbx/SConscript @@ -40,7 +40,7 @@ if (not env_etc.no_boost_python): Import("env_boost_python_ext") env_rstbx_boost_python_ext = env_boost_python_ext.Clone() env_rstbx_boost_python_ext.Prepend( - LIBS=["rstbx", "cctbx", "scitbx_boost_python"]) + LIBS=["rstbx", "ann", "cctbx", "scitbx_boost_python"]) env_rstbx_boost_python_ext.SharedLibrary( target="#lib/rstbx_ext", source="ext.cpp") env_rstbx_boost_python_ext.SharedLibrary( -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 2

Tidy up CCTBX
by Dr. Robert Oeffner 27 Jan '18

27 Jan '18

Hi all, I'm wondering if we could make efforts to refactor code modules in the cctbx_project so as making it have the minimum number of dependencies. This would be helpful to other users of the CCTBX that wishes to develop code on top of it. In the longer term it would also make it easier for us to deploy CCTBX as a Python wheel installer and by implication, spread the gospel of the wonders of the CCTBX. What is currently complicating this is the inclusion of the dxtbx module within the cctbx_project. It is being used by Dials. So my question is if it would be possible to move this project into the Dials code? If the dxtbx was just another small module it wouldn't be much of a problem. But the dxtbx depends on the cbflib and HDF5 components. If you want to build CCTBX from sources that's cumbersome. Why? Because CCTBX as deployed from CCI on http://cci.lbl.gov/cctbx_build/ is only built for a few select platforms. Building CCTBX on a different platform from those with bootstrap compels us to also build cbflib and HDF5 from scratch. Although this is not impossible it does increase the entry barrier for other potential users of the CCTBX who wish to incorporate the CCTBX in their programs: If all a user wants is to calculate allowed origins and symmetry sites for a given space group then an alternative to CCTBX might be preferable rather than having to massively expand their project with the seemingly unrelated HDF5 component. In other words, as it stands now the CCTBX is not straightforward for others to adopt. What do other people think? Regards, Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234

8 16

libtbx.configure crashes RE: new requirements for libtbx_refresh.py
by markus.gerstel＠diamond.ac.uk 23 Jan '18

23 Jan '18

Hi everyone, You may find that with the latest cctbx_project updates your libtbx.configure / make reconf crashes with (...) File "(..)/modules/annlib_adaptbx/libtbx_refresh.py", line 6 print message ^ SyntaxError: invalid syntax make: *** [reconf] Error 1 Solution: Changes are required in annlib_adaptbx, and you do not have the latest version on your system. Please run cd $(libtbx.find_in_repositories annlib_adaptbx)/.. rm -rf annlib_adaptbx git clone https://github.com/cctbx/annlib_adaptbx.git Background: This is related to my earlier email attached below. I couldn't figure out how to update the static tarball of annlib_adaptbx myself, and found it easier to just convert the repository including history. Said repository now lives at https://github.com/cctbx/annlib_adaptbx. Bootstrap has been updated to pull from this location. To avoid duplication the old repository copy we kept at https://github.com/dials/annlib_adaptbx for DIALS release control now points there, too. I note there is another copy at https://github.com/cctbx-xfel/annlib_adaptbx/network where you may also want to change the existing labels to the correct commit tree. Whoever maintains this - let me know if you want any help. Builds are still running but it looks like this is the only bit which may require intervention on your end. -Markus From: cctbxbb-bounces(a)phenix-online.org [mailto:[email protected]] On Behalf Of markus.gerstel(a)diamond.ac.uk Sent: 16 January 2018 09:29 To: cctbxbb(a)phenix-online.org Subject: [cctbxbb] new requirements for libtbx_refresh.py Hello everyone, There will soon be a requirement for all libtbx_refresh.py and run_tests.py of cctbx modules to contain the line from __future__ import absolute_import, division, print_function I have modified those that I have write access to (everything in cctbx_project, all default modules, dials*, labelit*, phenix*, xia2*), but there may be others that I am not aware of/do not use/do not have access to. (While you are at it ideally you would want to make sure these files are valid Python2 and 3 code, but that is not strictly required at this time.) For more information on the patch that requires this change see https://github.com/cctbx/cctbx_project/commit/ecd7192930961fde31c4ba29a049b… -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

GRC / cctbx catch up
by Graeme.Winter＠diamond.ac.uk 16 Jan '18

16 Jan '18

Folks, Who’s going to GRC on diffraction methods this summer? Would it be good to have a cctbx developer’s meeting on the Wednesday afternoon which is “free” - seems like a good opportunity? Cheers Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

new requirements for libtbx_refresh.py
by markus.gerstel＠diamond.ac.uk 16 Jan '18

16 Jan '18

Hello everyone, There will soon be a requirement for all libtbx_refresh.py and run_tests.py of cctbx modules to contain the line from __future__ import absolute_import, division, print_function I have modified those that I have write access to (everything in cctbx_project, all default modules, dials*, labelit*, phenix*, xia2*), but there may be others that I am not aware of/do not use/do not have access to. (While you are at it ideally you would want to make sure these files are valid Python2 and 3 code, but that is not strictly required at this time.) For more information on the patch that requires this change see https://github.com/cctbx/cctbx_project/commit/ecd7192930961fde31c4ba29a049b… -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

How to hack at the autobuild?
by Luc Bourhis 15 Dec '17

15 Dec '17

Hi, I have found a few issues with auto build on recent versions of Fedora. I need to edit files in libtbx/autobuild to fix them but then I want to test that running bootstrap works. But then bootstrap.py always pull from github. I don’t want to push my changes to github until it’s solid though. How do you guys approach that? Best wishes, Luc

3 2

What does get numpy to be install during bootstrap?
by Luc Bourhis 14 Dec '17

14 Dec '17

Hi, I noticed that bootstrap.py does indirectly result in the installation of numpy but a quick swipe of the code did not reveal to me what does get it installed. Since I am sure somebody knows… Best wishes, Luc

2 1

Fwd: [Dials-support] dials.stills_process crash report
by Graeme.Winter＠diamond.ac.uk 13 Dec '17

13 Dec '17

Folks Once again, lack of stack traces in easy_mp is a big annoyance I forget the reasons why printing the actual error was a bad thing, but I do remember a discussion some 18 mo past Not wishing to replay the entire transaction, but could we revisit this? Thanks Graeme Begin forwarded message: From: <Danny.Axford(a)Diamond.ac.uk<mailto:[email protected]>> Subject: [Dials-support] dials.stills_process crash report Date: 12 December 2017 at 17:43:41 GMT To: <dials-support(a)lists.sourceforge.net<mailto:[email protected]>> Cc: martin.appleby(a)diamond.ac.uk<mailto:[email protected]> Perhaps this means something to someone. This is the SACLA data again, it seems to run happily and then hit a problematic frame? Martin and I see the error at pretty much the same point in the dataset, we were both running on local machines rather than the cluster. Cheers, Danny Detector 1 RMSDs by panel: ---------------------------------------------------- | Panel | Nref | RMSD_X | RMSD_Y | RMSD_DeltaPsi | | id | | (px) | (px) | (deg) | ---------------------------------------------------- | 1 | 89 | 0.37442 | 0.48949 | 0.2192 | | 2 | 69 | 0.49627 | 0.4847 | 0.21385 | | 5 | 90 | 0.48267 | 0.59909 | 0.22839 | | 6 | 70 | 0.6094 | 0.55592 | 0.20944 | ---------------------------------------------------- Traceback (most recent call last): File "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20171212/build/../modules/dials/command_line/stills_process.py", line 870, in <module> halraiser(e) File "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20171212/build/../modules/dials/command_line/stills_process.py", line 868, in <module> script.run() File "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20171212/build/../modules/dials/command_line/stills_process.py", line 383, in run preserve_exception_message=True) File "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20171212/modules/cctbx_project/libtbx/easy_mp.py", line 623, in parallel_map result = res() File "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20171212/modules/cctbx_project/libtbx/scheduling/result.py", line 119, in __call__ self.traceback( exception = self.exception() ) File "/dls/science/groups/scisoft/DIALS/CD/latest/dials-dev20171212/modules/cctbx_project/libtbx/scheduling/stacktrace.py", line 86, in __call__ raise exception RuntimeError: Please report this error to dials-support(a)lists.sourceforge.net<mailto:[email protected]>: exit code = -9 -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org<http://slashdot.org/>! http://sdm.link/slashdot _______________________________________________ DIALS-support mailing list DIALS-support(a)lists.sourceforge.net<mailto:[email protected]> https://lists.sourceforge.net/lists/listinfo/dials-support

3 2

Re: [cctbxbb] cctbxbb Digest, Vol 105, Issue 3
by Mooers, Blaine H.M. (HSC) 08 Dec '17

08 Dec '17

Hi Gianluca , You might try getting the sources for 2.7 from here: https://gist.github.com/0x414A/8b64178e69d9f8331938 Best regards, Blaine Blaine Mooers, Ph.D. Associate Professor Department of Biochemistry and Molecular Biology, College of Medicine University of Oklahoma Health Sciences Center 975 NE 10th Street, BRC 466 Oklahoma City, OK 73104-5419 Faculty webpage X-ray lab (LBSF) COBRE in Structural Biology SSRL UEC Facebook Page SSRL UEC SSRL SAC LCLS UEC EasyPyMOL Molecular Graphics Small Angle Scattering office: (405) 271-8300 lab: (405) 271-8313 e-mail: blaine-mooers(a)ouhsc.edu ________________________________________ From: cctbxbb-bounces(a)phenix-online.org [cctbxbb-bounces(a)phenix-online.org] on behalf of cctbxbb-request(a)phenix-online.org [cctbxbb-request(a)phenix-online.org] Sent: Friday, December 08, 2017 2:00 PM To: cctbxbb(a)phenix-online.org Subject: cctbxbb Digest, Vol 105, Issue 3 Send cctbxbb mailing list submissions to cctbxbb(a)phenix-online.org To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai… or, via email, send a message with subject or body 'help' to cctbxbb-request(a)phenix-online.org You can reach the person managing the list at cctbxbb-owner(a)phenix-online.org When replying, please edit your Subject line so it is more specific than "Re: Contents of cctbxbb digest..." Today's Topics: 1. Importing PyQt4 in litbx.python (Gianluca SANTONI) 2. Re: Importing PyQt4 in litbx.python (markus.gerstel(a)diamond.ac.uk) 3. Re: Importing PyQt4 in litbx.python (Gianluca SANTONI) ---------------------------------------------------------------------- Message: 1 Date: Fri, 8 Dec 2017 13:18:13 +0100 From: Gianluca SANTONI <gianluca.santoni(a)esrf.fr> To: cctbx mailing list <cctbxbb(a)phenix-online.org> Subject: [cctbxbb] Importing PyQt4 in litbx.python Message-ID: <09889a85-3126-af38-05f3-8af190cdfd07(a)esrf.fr> Content-Type: text/plain; charset=utf-8; format=flowed Dear all, I'm having some troubles recently in importing pyqt4 in libtbx.python under Debian8. It used to work with a previous build compiled from the sources, but I don't find a reasonable way to make it work from the bootstrap installation. More details: 1) If I do in libtbx.python sys.path.append('/usr/lib/python2.7/dist-packages/') and "from PyQt4 import QtCore", I get an unidentified symbol: PyUnicodeUCS4_AsLatin1String 2) If I try to compile cctbx as I did in the past "python ../libtbx/configure.py cctbx" no make file is produced. Maybe I'm missing something Thanks everybody for your help. Gian -- Gianluca Santoni, PhD ESRF structural biology group 71 avenue des Martyrs 38027 Grenoble cedex (France) ------------------------------ Message: 2 Date: Fri, 8 Dec 2017 12:40:13 +0000 From: <markus.gerstel(a)diamond.ac.uk> To: <cctbxbb(a)phenix-online.org> Subject: Re: [cctbxbb] Importing PyQt4 in litbx.python Message-ID: <f8d53c3e-c8d8-4941-9540-c9b80f95441d(a)email.android.com> Content-Type: text/plain; charset=utf-8 Hi, There is an issue open on UCS4 which is somewhat related: https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_cctbx_cctbx… We currently don't use UCS4, and your system pyqt4 does. You would have to install (and potentially, compile) it for the cctbx python directly. A 3 minute research from my mobile phone suggests that you might be able to download the sources as tar.gz files and then do something along the lines of cctbx.python -m pip install SIP cctbx.python -m pip install pyqt-source.tar.gz but this is untested, and I haven't used pyqt before. -Markus On 8 Dec 2017 12:19, Gianluca SANTONI <gianluca.santoni(a)esrf.fr> wrote: Dear all, I'm having some troubles recently in importing pyqt4 in libtbx.python under Debian8. It used to work with a previous build compiled from the sources, but I don't find a reasonable way to make it work from the bootstrap installation. More details: 1) If I do in libtbx.python sys.path.append('/usr/lib/python2.7/dist-packages/') and "from PyQt4 import QtCore", I get an unidentified symbol: PyUnicodeUCS4_AsLatin1String 2) If I try to compile cctbx as I did in the past "python ../libtbx/configure.py cctbx" no make file is produced. Maybe I'm missing something Thanks everybody for your help. Gian -- Gianluca Santoni, PhD ESRF structural biology group 71 avenue des Martyrs 38027 Grenoble cedex (France) _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai… -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom ------------------------------ Message: 3 Date: Fri, 8 Dec 2017 14:20:26 +0100 From: Gianluca SANTONI <gianluca.santoni(a)esrf.fr> To: cctbxbb(a)phenix-online.org Subject: Re: [cctbxbb] Importing PyQt4 in litbx.python Message-ID: <820c8b08-d380-dd46-e25a-e9ee9e4b076b(a)esrf.fr> Content-Type: text/plain; charset=utf-8; format=flowed Hi Markus, I could fix everyting by compiling both libraries from source with cctbx.python. Hope this helps others with similar problem too. cctbx.python -m pip install SIP cctbx.python -m pip install pyqt-source.tar.gz did not work. There is actually no PyPi package for sip in python2.7 Cheers Gian Gianluca Santoni, PhD ESRF structural biology group 71 avenue des Martyrs 38027 Grenoble cedex (France) On 08/12/2017 13:40, markus.gerstel(a)diamond.ac.uk wrote: > Hi, > > There is an issue open on UCS4 which is somewhat related: https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_cctbx_cctbx… > > We currently don't use UCS4, and your system pyqt4 does. You would have to install (and potentially, compile) it for the cctbx python directly. > > A 3 minute research from my mobile phone suggests that you might be able to download the sources as tar.gz files and then do something along the lines of > > cctbx.python -m pip install SIP > cctbx.python -m pip install pyqt-source.tar.gz > > but this is untested, and I haven't used pyqt before. > > -Markus > > On 8 Dec 2017 12:19, Gianluca SANTONI <gianluca.santoni(a)esrf.fr> wrote: > Dear all, > I'm having some troubles recently in importing pyqt4 in libtbx.python > under Debian8. > It used to work with a previous build compiled from the sources, but I > don't find a reasonable way to make it work from the bootstrap installation. > > More details: > > 1) If I do in libtbx.python > sys.path.append('/usr/lib/python2.7/dist-packages/') and "from PyQt4 > import QtCore", I get an > > unidentified symbol: PyUnicodeUCS4_AsLatin1String > > 2) If I try to compile cctbx as I did in the past "python > ../libtbx/configure.py cctbx" no make file is produced. Maybe I'm > missing something > > Thanks everybody for your help. > Gian > > -- > Gianluca Santoni, PhD > ESRF structural biology group > 71 avenue des Martyrs > 38027 Grenoble cedex (France) > > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai… > > ------------------------------ _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai… End of cctbxbb Digest, Vol 105, Issue 3 ***************************************

1 0

Re: [cctbxbb] Importing PyQt4 in litbx.python
by markus.gerstel＠diamond.ac.uk 08 Dec '17

08 Dec '17

Hi, There is an issue open on UCS4 which is somewhat related: https://github.com/cctbx/cctbx_project/issues/60 We currently don't use UCS4, and your system pyqt4 does. You would have to install (and potentially, compile) it for the cctbx python directly. A 3 minute research from my mobile phone suggests that you might be able to download the sources as tar.gz files and then do something along the lines of cctbx.python -m pip install SIP cctbx.python -m pip install pyqt-source.tar.gz but this is untested, and I haven't used pyqt before. -Markus On 8 Dec 2017 12:19, Gianluca SANTONI <gianluca.santoni(a)esrf.fr> wrote: Dear all, I'm having some troubles recently in importing pyqt4 in libtbx.python under Debian8. It used to work with a previous build compiled from the sources, but I don't find a reasonable way to make it work from the bootstrap installation. More details: 1) If I do in libtbx.python sys.path.append('/usr/lib/python2.7/dist-packages/') and "from PyQt4 import QtCore", I get an unidentified symbol: PyUnicodeUCS4_AsLatin1String 2) If I try to compile cctbx as I did in the past "python ../libtbx/configure.py cctbx" no make file is produced. Maybe I'm missing something Thanks everybody for your help. Gian -- Gianluca Santoni, PhD ESRF structural biology group 71 avenue des Martyrs 38027 Grenoble cedex (France) _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org http://phenix-online.org/mailman/listinfo/cctbxbb -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

Importing PyQt4 in litbx.python
by Gianluca SANTONI 08 Dec '17

08 Dec '17

Dear all, I'm having some troubles recently in importing pyqt4 in libtbx.python under Debian8. It used to work with a previous build compiled from the sources, but I don't find a reasonable way to make it work from the bootstrap installation. More details: 1) If I do in libtbx.python sys.path.append('/usr/lib/python2.7/dist-packages/') and "from PyQt4 import QtCore", I get an unidentified symbol: PyUnicodeUCS4_AsLatin1String 2) If I try to compile cctbx as I did in the past "python ../libtbx/configure.py cctbx" no make file is produced. Maybe I'm missing something Thanks everybody for your help. Gian -- Gianluca Santoni, PhD ESRF structural biology group 71 avenue des Martyrs 38027 Grenoble cedex (France)

1 0

cctbx.miller problem
by 李选选 06 Dec '17

06 Dec '17

> > Dear cctbx developers, > > I’m Xuanxuan, and I’m using cctbx to develop a crystallography-related algorithm. Recently, I found there is something wrong when I use miller package to generate miller indices for given space group and unit cell parameters. Maybe it’s my mistake, or it’s a bug. > > Here are some details. I just use the following code to generate a list of miller indices for a Fd-3m crystal. > > ms = miller.build_set( > crystal_symmetry=crystal.symmetry( > space_group_symbol='Fd-3m', > unit_cell=(5.4307,5.4307,5.4307,90.00,90.0,90.00)), > anomalous_flag=False, > d_min=0.4) > > And I found some reflections are missing, like 220, 311, which should exist according to the reflection conditions listed in this page http://img.chem.ucl.ac.uk/sgp/large/227az2.htm <http://img.chem.ucl.ac.uk/sgp/large/227az2.htm>. > > Looking forward to your response. > > Thanks, > Xuanxuan

5 4

rotate_translate_map function causes "splinching"
by Pearce, N.M. (Nick) 29 Nov '17

29 Nov '17

Hi all, I’m having a problem with the rotate_translate_map function from cctbx.maptbx. I have a 3D array (a mask of 0s and 1s, in this case) that I’m writing as a CCP4 map file. If I write it out “on the origin” (without running rotate_translate_map), then it’s fine (see attached “before.png”). However, if I apply the rotate_translate_map function and then write it out, odd features appear at the edge of the unit cell (see “after.png”). It is NOT the case that the map is going through the “periodic boundary conditions” of the unit cell (the shift is less than one unit cell, as shown by the structure which is transformed by the same amount). The minimal code is: ——————————— array = cctbx.maptbx.rotate_translate_map( unit_cell=self.unit_cell(), map_data=array, rotation_matrix=scitbx.matrix.rec([1,0,0,0,1,0,0,0,1], (3,3)).elems, translation_vector=(-1.0*scitbx.matrix.rec((X,Y,Z), (3,1))).elems ) # Write the map iotbx.ccp4_map.write_ccp4_map( file_name=f_name, unit_cell=self.unit_cell(), space_group=self.space_group(), map_data=array, labels=flex.std_string(['’]) ) ——————————— where (X,Y,Z) is a tuple for the origin shift. Is there something stupid that I’m missing? Has anyone else had the same problem? The only code for a rotate_translate_map function I could find was actually in maptbx/average_densities.h and I couldn’t see anything obviously “wrong”… though I’m not that familiar with C++. The maps and files can be found at https://www.dropbox.com/sh/xqrsk5lcn603416/AABiVO74g6A_jZMGn_DjxOmka?dl=0 all “grid” files are without translation, and all “ref” files are after translation. both are written out exactly the same (the iotbx.ccp4_map call above). I’m using the cctbx bundled with a ccp4 nightly from 2016-10-22. Thanks, Nick [cid:[email protected]][cid:[email protected]]

2 1

cctbx versions causing indentation crash?
by Magnus Andersson 10 Nov '17

10 Nov '17

Hi all, I have a script that runs fine on my old computer, but crashes on my new computer with the error message: IndentationError: unexpected indent This script is (amongst other things) calling the softwatre Coot such as: subprocess.call(["""coot --no-graphics<<end/n (handle-read-draw-molecule-with-recentre \""""+str(inf)+"""\" 1) (set-refinement-immediate-replacement 1) (set-refine-max-residues 40) (regularize-zone 0 \" \" """+str(res1-2)+""" """+str(res3)+"""\" \") (accept-regularizement) (regularize-zone 0 \" \" """+str(res3)+""" """+str(res2+2)+"""\" \") (accept-regularizement) (write-pdb-file 0 \""""+str(outf)+"""\") (coot-real-exit 0)"""], shell=True) which is ok on my old system. I am trying to find the most likely source of this error. Here are the differences: - OLD cctbx = uses Python 2.7.10, NEW cctbx 2.7.14 - OLD OSX = 10.10.5 (i.e. Yosemite), NEW OSX = 10.11.5 (i.e. El Capitan) - OLD Coot = vers 0.8, NEW Coot = vers 0.8.8 [old Coot install was from Fink but this shouldn’t matter I think] Perhaps someone has encountered something similar or has an educated guess on whether different cctbx versions can cause such errors? Thanks, Magnus #### Magnus Andersson, Docent, Ph.D. Email: magnus.andersson(a)scilifelab.se Phone: 0769469872 Mailing address: Science For Life Laboratory Alpha 6 Box 1031 171 21 Solna Visiting address: Tomtebodavägen 23A 171 65 Solna

2 2