June 2018 - cctbxbb - phenix-online.org

Install OpenBLAS or MKL in base
by Luc Bourhis 22 Jun '18

22 Jun '18

Hi, years ago, I had wrote on this discussion list about a project of mine to dramatically accelerate scitbx.lstbx by using optimised BLAS libraries. That work had been dormant on a branch but now we are planning to move forward with the help of Pascal Parlois who will do the actual coding in the coming months. In order for this new code to be exercised by nightly tests, we, the smtbx people, need to write code so that that optimised BLAS library is installed during bootstrap. Right now, as a stopgap, I used conda to install either OpenBLAS and MKL but that won’t do with the cctbx philosophy. There are basically two choices: either OpenBLAS or MKL. When I started this project, I chose OpenBLAS because MKL was not freely redistributable, and I had got the green light to install it as needed. However now MKL has become completely free, thus becoming a possible choice. MKL is more performant on Intel but OpenBLAS is better on AMD (OpenBLAS actually runs on pretty much any processor out there but we don’t care in the context of cctbx): c.f. Julia people conclusions <https://discourse.julialang.org/t/openblas-is-faster-than-intel-mkl-on-amd-…> Now I realise that most of you may not care one way or another!?! But we won’t move forward before we got the green light… Note that I am taking about installing a BLAS usable from C++ here, not getting a BLAS-accelerated numpy. The latter could be an alternative though. But that would require to modify the bootstrap code to compile a MKL- or OpenBLAS-enabled numpy anyway. Best wishes, Luc J Bourhis

6 22

Fwd: buildbot failure in Phenix on cctbx-ci-ci
by Graeme.Winter＠Diamond.ac.uk 05 Jun '18

05 Jun '18

Hi Folks, Am I alone in finding this kind of message ** very ** annoying I did not touch this file I have no business touching this file If I do fix this then it may cause conflicts for others, which will no doubt annoy them far more than the odd extra space in the code However I am also now to blame for breaking the Phenix build, in the run up to a release This is no a happy place Suggestions? Thanks & cheerio Graeme Begin forwarded message: From: phenix(a)phenix-online.org<mailto:[email protected]> Subject: buildbot failure in Phenix on cctbx-ci-ci Date: 4 June 2018 at 12:11:19 BST To: graeme.winter(a)gmail.com<mailto:[email protected]> Cc: ASBrewster(a)LBL.Gov<mailto:[email protected]>, NWMoriarty(a)LBL.Gov<mailto:[email protected]> Build status: FAILURE Buildslave for this Build: ci Complete logs for all build steps: http://cci-vm-6.lbl.gov:8010/builders/cctbx-ci-ci/builds/5580 Build Reason: The SingleBranchScheduler scheduler named 'cctbx-ci.poll' triggered this build Build Source Stamp: clipper: [branch master] HEAD cbflib: [branch master] HEAD cctbx_project: [branch master] 4de5e7f82b0302632282f78c6dda8b999cf6abf6 annlib_adaptbx: [branch master] HEAD tntbx: [branch master] HEAD ccp4io_adaptbx: [branch master] HEAD boost: [branch master] HEAD gui_resources: [branch master] HEAD Blamelist: graeme-winter If you got an email and you didn't touch this file, it may because someone commited this file before you with an error and hasn't fixed it yet. You might have noticed the error if you ran libtbx.find_clutter on the entire tree. Detailed log of last build step: http://cci-vm-6.lbl.gov:8010/builders/cctbx-ci-ci/builds/5580/steps/test ALL clutter/logs/stdio<http://cci-vm-6.lbl.gov:8010/builders/cctbx-ci-ci/builds/5580/steps/test%20…> Last 80 lines of "test ALL clutter.stdio" iota/command_line/iota_single_image.py: tabs or trailing whitespace=2, #10, #11 *** To clean up please use: libtbx.clean_clutter *** -The Buildbot -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

4 5

Re: [cctbxbb] [git/cctbx] master: Generate libtbx.* wrappers from correct directory (ad736e2c8)
by Billy Poon 01 Jun '18

01 Jun '18

I don't think this is working on macOS. The configuration step is generating error messages like, Ignored: [Errno 2] No such file or directory: <command path> In lines 1640-1643, self.write_dispatcher( source_file=libtbx.env.under_base(os.path.join('bin', ep.name)), target_file=os.path.join('bin', 'libtbx.' + ep.name), ) the source_file argument is always being constructed as <base>/bin/<command> instead of using the directory from self.get_setuptools_script_dir(). Should it be the bin_directory variable instead (e.g. source_file=os.path.join(bin_directory, ep.name))? As a side note, conda will put everything in bin because there is no framework. For access to the window manager, you would call pythonw, which points to a python inside a .app bundle. -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org On Fri, Jun 1, 2018 at 3:15 AM CCTBX commit < diamondlightsource.jenkins(a)gmail.com> wrote: > Repository : ssh://g18-sc-serv-04.diamond.ac.uk/cctbx > On branch : master > > ------------------------------ > > > commit ad736e2c8e65407615a7e6604b0544687ed8058c > Author: Markus Gerstel <markus.gerstel(a)diamond.ac.uk> > Date: Thu May 31 08:27:09 2018 +0100 > > Generate libtbx.* wrappers from correct directory > > Python console scripts don't necessarily land in base/bin, specifically > on MacOS this is not the case. Use setuptools etc. to find the actual > Python console script location and use that directory instead. Fixes > #177 > > Add a function to regenerate all Python console scripts which can be > called from an installer to relocate paths. > > > ------------------------------ > > > ad736e2c8e65407615a7e6604b0544687ed8058c > libtbx/env_config.py | 74 +++++++++++++++++++++++++----- > libtbx/fastentrypoints.py | 112 > ++++++++++++++++++++++++++++++++++++++++++++++ > 2 files changed, 176 insertions(+), 10 deletions(-) > > diff --git a/libtbx/env_config.py b/libtbx/env_config.py > index de5fc72dd..25daa2cb3 100644 > --- a/libtbx/env_config.py > +++ b/libtbx/env_config.py > @@ -1562,23 +1562,77 @@ selfx: > source_file=source_file, > target_file=module_name+"."+command) > > + def get_setuptools_script_dir(): > + ''' > + Find the location of python entry point console_scripts, ie. things > like > + 'pip', 'pytest', ... > + This is different from simple /base/bin, eg. on MacOS. > + > + > https://stackoverflow.com/questions/25066084/get-entry-point-script-file-lo… > + ''' > + from setuptools import Distribution > + from setuptools.command.install import install > + class OnlyGetScriptPath(install): > + def run(self): > + # does not call install.run() by design > + self.distribution.install_scripts = self.install_scripts > + dist = Distribution({'cmdclass': {'install': OnlyGetScriptPath}}) > + dist.dry_run = True # not sure if necessary, but to be safe > + dist.parse_config_files() > + command = dist.get_command_obj('install') > + command.ensure_finalized() > + command.run() > + return dist.install_scripts > + > + def regenerate_entry_point_console_scripts(self, verbose=True): > + ''' > + Creates all console_scripts entry point scripts from scratch and > overwrites existing ones. > + This is intended to be used by installers to relocate the entry point > script paths. > + ''' > + try: > + import distutils.dist > + import libtbx.fastentrypoints # monkeypatches setuptools > + import pkg_resources > + import setuptools.command.easy_install > + except ImportError: > + return > + > + # Prepare generic script generator > + distribution = distutils.dist.Distribution({'name': 'setuptools'}) > + command = setuptools.command.easy_install.easy_install(distribution) > + command.args = ['wheel'] # dummy argument > + command.finalize_options() > + > + # Force regeneration of all known console_scripts > + for pkg_resources_dist in pkg_resources.working_set: > + console_scripts = > pkg_resources_dist.get_entry_map().get('console_scripts') > + if console_scripts: > + if verbose: > + print("Regenerating commands for %s: %s" % ( > + pkg_resources_dist, > + list(console_scripts), > + )) > + command.install_wrapper_scripts(pkg_resources_dist) > + > def generate_entry_point_dispatchers(self): > - # Write indirect dispatcher scripts for all console_scripts entry > points > - # that have existing dispatcher scripts in the base/bin directory, but > - # add a 'libtbx.' prefix. > - base_bin_directory = libtbx.env.under_base('bin') > - if not os.path.isdir(base_bin_directory): > + ''' > + Write indirect dispatcher scripts for all console_scripts entry points > + that have existing dispatcher scripts in the base/bin directory, but > + add a 'libtbx.' prefix. > + ''' > + try: > + import pkg_resources > + bin_directory = get_setuptools_script_dir() > + except ImportError: > + return > + if not os.path.isdir(bin_directory): > return # do not create console_scripts dispatchers, only point to > them > > - base_bin_dispatchers = set(os.listdir(base_bin_directory)) > + base_bin_dispatchers = set(os.listdir(bin_directory)) > existing_dispatchers = filter(lambda f: f.startswith('libtbx.'), > self.bin_path.listdir()) > existing_dispatchers = set(map(lambda f: f[7:], existing_dispatchers)) > entry_point_candidates = base_bin_dispatchers - existing_dispatchers > > - try: > - import pkg_resources > - except ImportError: > - return > entry_points = pkg_resources.iter_entry_points('console_scripts') > entry_points = filter(lambda ep: ep.name in entry_point_candidates, > entry_points) > for ep in entry_points: > diff --git a/libtbx/fastentrypoints.py b/libtbx/fastentrypoints.py > new file mode 100644 > index 000000000..9707f74a3 > --- /dev/null > +++ b/libtbx/fastentrypoints.py > @@ -0,0 +1,112 @@ > +# noqa: D300,D400 > +# Copyright (c) 2016, Aaron Christianson > +# All rights reserved. > +# > +# Redistribution and use in source and binary forms, with or without > +# modification, are permitted provided that the following conditions are > +# met: > +# > +# 1. Redistributions of source code must retain the above copyright > +# notice, this list of conditions and the following disclaimer. > +# > +# 2. Redistributions in binary form must reproduce the above copyright > +# notice, this list of conditions and the following disclaimer in the > +# documentation and/or other materials provided with the distribution. > +# > +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS > +# IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED > +# TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A > +# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT > +# HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, > +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED > +# TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR > +# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF > +# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING > +# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS > +# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. > +''' > +Monkey patch setuptools to write faster console_scripts with this format: > + > + import sys > + from mymodule import entry_function > + sys.exit(entry_function()) > + > +This is better. > + > +(c) 2016, Aaron Christianson > +http://github.com/ninjaaron/fast-entry_points > +''' > +from setuptools.command import easy_install > +import re > +TEMPLATE = '''\ > +# -*- coding: utf-8 -*- > +# EASY-INSTALL-ENTRY-SCRIPT: '{3}','{4}','{5}' > +__requires__ = '{3}' > +import re > +import sys > + > +from {0} import {1} > + > +if __name__ == '__main__': > + sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) > + sys.exit({2}())''' > + > + > +@classmethod > +def get_args(cls, dist, header=None): # noqa: D205,D400 > + """ > + Yield write_script() argument tuples for a distribution's > + console_scripts and gui_scripts entry points. > + """ > + if header is None: > + # pylint: disable=E1101 > + header = cls.get_header() > + spec = str(dist.as_requirement()) > + for type_ in 'console', 'gui': > + group = type_ + '_scripts' > + for name, ep in dist.get_entry_map(group).items(): > + # ensure_safe_name > + if re.search(r'[\\/]', name): > + raise ValueError("Path separators not allowed in script > names") > + script_text = TEMPLATE.format( > + ep.module_name, ep.attrs[0], '.'.join(ep.attrs), > + spec, group, name) > + # pylint: disable=E1101 > + args = cls._get_script_args(type_, name, header, script_text) > + for res in args: > + yield res > + > + > +# pylint: disable=E1101 > +easy_install.ScriptWriter.get_args = get_args > + > + > +def main(): > + import os > + import re > + import shutil > + import sys > + dests = sys.argv[1:] or ['.'] > + filename = re.sub('\.pyc$', '.py', __file__) > + > + for dst in dests: > + shutil.copy(filename, dst) > + manifest_path = os.path.join(dst, 'MANIFEST.in') > + setup_path = os.path.join(dst, 'setup.py') > + > + # Insert the include statement to MANIFEST.in if not present > + with open(manifest_path, 'a+') as manifest: > + manifest.seek(0) > + manifest_content = manifest.read() > + if 'include fastentrypoints.py' not in manifest_content: > + manifest.write(('\n' if manifest_content else '') + > + 'include fastentrypoints.py') > + > + # Insert the import statement to setup.py if not present > + with open(setup_path, 'a+') as setup: > + setup.seek(0) > + setup_content = setup.read() > + if 'import fastentrypoints' not in setup_content: > + setup.seek(0) > + setup.truncate() > + setup.write('import fastentrypoints\n' + setup_content) > > ------------------------------ > > To unsubscribe from the CCTBX-COMMIT list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCTBX-COMMIT&A=1 >

1 0

show() not __str__ - why?
by Graeme.Winter＠Diamond.ac.uk 01 Jun '18

01 Jun '18

Folks, Came up in conversation - how come the motif for printing / providing object info is to call show() and pass a file handle rather than the more pythonic __str__? See e.g. https://github.com/dials/dials/blob/b499c6191caa18e04ddaf190321a3da6c14af55… I know in dials we could do it differently however we inherited the motif from cctbx & I am not sure why things work that way - figured there must be a reason... This is also under discussion under https://github.com/dials/dials/pull/567 but I thought it was worth making sure we knew the reasons why cctbx uses it's own way of doing something... (also it's quite annoying that printing things will give you say space_group object i.e. much like printing type(object), when you want to see it's contents - like printing j and being told it is an int!) Thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0