- cctbxbb - phenix-online.org

console
by Nigel Moriarty 27 Jul '15

27 Jul '15

I reverted the version of bootstrap.py on cci-vm-6 and got the attached console GUI. Cheers Nigel --- Nigel W. Moriarty Building 33R0349, Physical Biosciences Division Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709 Email : NWMoriarty(a)LBL.gov Fax : 510-486-5909 Web : CCI.LBL.gov

1 0

Re: [cctbxbb] after SourceForge awakens
by Graeme.Winter＠diamond.ac.uk 23 Jul '15

23 Jul '15

Hi Nigel We're thinking along the same lines moving xia2 across first then maybe dials, depending on how people feel. Moving all of it across in the longer term would work well. Thanks Graeme On 20 Jul 2015 20:42, Nigel Moriarty <nwmoriarty(a)lbl.gov> wrote: Folks I have not read the emails regarding the current SourceForge situation. However, we had been planning to move to github in the near future. SF has been very reliable so I don't think it was too much of pain given the long use we've had from them. It does, however, come at a time when we are trying to get out a major release of Phenix. To that end, it would be helpful if any commits were bug fixes only to minimise the risk of breaking to the codebase. We will be actively evaluating the options regarding to github after Phenix 1.10 is safely away... Cheers Nigel --- Nigel W. Moriarty Building 33R0349, Physical Biosciences Division Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709 Email : NWMoriarty(a)LBL.gov Fax : 510-486-5909 Web : CCI.LBL.gov<http://CCI.LBL.gov> -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 2

after SourceForge awakens
by Nigel Moriarty 21 Jul '15

21 Jul '15

Folks I have not read the emails regarding the current SourceForge situation. However, we had been planning to move to github in the near future. SF has been very reliable so I don't think it was too much of pain given the long use we've had from them. It does, however, come at a time when we are trying to get out a major release of Phenix. To that end, it would be helpful if any commits were bug fixes only to minimise the risk of breaking to the codebase. We will be actively evaluating the options regarding to github after Phenix 1.10 is safely away... Cheers Nigel --- Nigel W. Moriarty Building 33R0349, Physical Biosciences Division Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709 Email : NWMoriarty(a)LBL.gov Fax : 510-486-5909 Web : CCI.LBL.gov

1 0

Re: [cctbxbb] List usage
by Graeme.Winter＠diamond.ac.uk 21 Jul '15

21 Jul '15

Hi Nigel Sensible however unknowable in advance that this is the behaviour & also who is on list! Sorry Graeme On 20 Jul 2015 18:13, Nigel Moriarty <nwmoriarty(a)lbl.gov> wrote: Folks To those of you on the cctbx list (and even those not), as moderator I have received over a dozen requests from the server. One cause of the problem is non-members posting to a members-only list. This is usually caused by people including the ccbx list in a long list of recipients. This leads to another problem. The list complains about too many recipients. I'm sure that some of the discussion regarding the current SourceForge situation has been garbled as a result. There is an easy solution. When sending to the list, include only the list in the recipients. Anybody who is interested should subscribe to the list. Non members such as David Waterman should be encouraged to join the list. Cheers Nigel --- Nigel W. Moriarty Building 33R0349, Physical Biosciences Division Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709 Email : NWMoriarty(a)LBL.gov Fax : 510-486-5909 Web : CCI.LBL.gov<http://CCI.LBL.gov> -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 3

List usage
by Nigel Moriarty 21 Jul '15

21 Jul '15

Folks To those of you on the cctbx list (and even those not), as moderator I have received over a dozen requests from the server. One cause of the problem is non-members posting to a members-only list. This is usually caused by people including the ccbx list in a long list of recipients. This leads to another problem. The list complains about too many recipients. I'm sure that some of the discussion regarding the current SourceForge situation has been garbled as a result. There is an easy solution. When sending to the list, include only the list in the recipients. Anybody who is interested should subscribe to the list. Non members such as David Waterman should be encouraged to join the list. Cheers Nigel --- Nigel W. Moriarty Building 33R0349, Physical Biosciences Division Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709 Email : NWMoriarty(a)LBL.gov Fax : 510-486-5909 Web : CCI.LBL.gov

1 0

Building cctbx bundles without phenix.* dispatcher scripts
by Graeme.Winter＠diamond.ac.uk 03 Jul '15

03 Jul '15

Hi Folks Following on from various conversations going on here about programs in the default dials bundle which necessarily includes cctbx & lots of useful utilities in there, the question was asked why it includes bits of Phenix. In real life it does not include bits of phenix - they are bits of e.g. mmtbx which have dispatcher scripts which ask to be called phenix.xyz for example, as they are *also* used in phenix. Is there a way we can set the dispatcher prefix at the time of running libtbx.configure? That way you could have phenix.xyz or cctbx.xyz in other situations… then in the dials bundles we could make it clear that there are no bits of Phenix in there. Many thanks in advance, Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 2

bootstrap external dependencies
by markus.gerstel＠diamond.ac.uk 17 Jun '15

17 Jun '15

Good morning, last night we ran into a problem with some of the remaining external dependencies in the bootstrap script base install step. The sphinx package will currently download 11 files during its installation (list below). As with all other downloads I would like to be able to download these files beforehand and then pass them to our build systems directly, so that the subsequent builds are no longer dependant on network connectivity or pypi server reliability. I would like to propose a python-extra package for the bootstrap script, that checks the existence of a base_tmp/python_extra directory, and, if found, would install all python packages in that folder. This would be an easy facility to automatically build extra python packages (at DLS we already add a few extra packages, such as junit-xml, so this would actually simplify our build process), and would not affect any other bootstrap users. Alternatively, since the dependencies are fixed as they come from the Sphinx archive on the cci servers, we could package them up as well and put static build instructions into build_sphinx() in install_base_packages.py. Thoughts? -Markus The packages are: https://pypi.python.org/packages/source/J/Jinja2/Jinja2-2.7.3.tar.gz#md5=b9… https://pypi.python.org/packages/source/d/docutils/docutils-0.12.tar.gz#md5… https://pypi.python.org/packages/source/P/Pygments/Pygments-2.0.2.tar.gz#md… https://pypi.python.org/packages/source/M/MarkupSafe/MarkupSafe-0.23.tar.gz… https://pypi.python.org/packages/source/n/nose/nose-1.3.7.tar.gz#md5=4d3ad0… https://pypi.python.org/packages/source/p/pyparsing/pyparsing-2.0.3.tar.gz#… https://pypi.python.org/packages/source/t/tornado/tornado-4.2.tar.gz#md5=c5… https://pypi.python.org/packages/source/p/python-dateutil/python-dateutil-2… https://pypi.python.org/packages/source/c/certifi/certifi-2015.04.28.tar.gz… https://pypi.python.org/packages/source/b/backports.ssl_match_hostname/back… https://pypi.python.org/packages/source/s/six/six-1.9.0.tar.gz#md5=476881ef… -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

Race condition in bootstrap python installation
by markus.gerstel＠diamond.ac.uk 28 May '15

28 May '15

We have recently seen intermittent build failures with the bootstrap base installation at Diamond. These all occurred during the compilation of python and were usually of the form : gcc -pthread -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Wl,-rpath=\$ORIGIN/../lib Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o Parser/parsetok.o Parser/bitset.o Parser/metagrammar.o Parser/firstsets.o Parser/grammar.o Parser/pgen.o Objects/obmalloc.o Python/mysnprintf.o Python/pyctype.o Parser/tokenizer_pgen.o Parser/printgrammar.o Parser/pgenmain.o -lpthread -ldl -lutil -o Parser/pgen : Parser/node.o: file not recognized: File truncated : collect2: ld returned 1 exit status : make[1]: *** [Parser/pgen] Error 1 : make[1]: Leaving directory `/scratch/jenkins_slave/workspace/dials_bootstrap_platforms/compilationtarget/native/label/dials-ws154/build_dials/base_tmp/Python-2.7.8_cci' : make: *** [Include/graminit.h] Error 2 : make: *** Waiting for unfinished jobs.... Traceback (most recent call last): File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 1194, in <module> installer(args=sys.argv, log=sys.stdout) File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 184, in __init__ self.build_dependencies(packages=packages) File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 571, in build_dependencies getattr(self, 'build_%s'%i)() File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 622, in build_python self.call('make -j %s install'%(self.nproc), log=log, cwd=python_dir) File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 264, in call return call(args, log=log, verbose=self.verbose, **kwargs) File "/scratch/jenkins_slave/workspace/dials_bootstrap_platforms/compilationtarget/native/label/dials-ws154/build_dials/modules/cctbx_project/libtbx/auto_build/installer_utils.py", line 81, in call raise RuntimeError("Call to '%s' failed with exit code %d" % (args, rc)) RuntimeError: Call to 'make -j 4 install' failed with exit code 2 Other indicated errors are /usr/bin/ld: final link failed: File truncated or /usr/bin/ld: can not read symbols: File truncated I suspect that these errors are manifestations of a race condition in the python build process, which the bootstrap script now by default runs with a -j 4 parallel make. To test this we are now using --nproc=1 for the base installation step. We haven't seen these build failures since. I suggest we set the python compilation to never run with parallel make. Any other ideas? -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 2

Native python unittests for the libtbx testing framework
by markus.gerstel＠diamond.ac.uk 20 May '15

20 May '15

Dear cctbx developers and users, For a side-project of mine I experimented with native python unit tests (some information available on https://docs.python.org/2/library/unittest.html if you are not familiar with it). The python unittest framework has a couple of advantages over the libtbx testing world. I found that the two most immediate advantages are: Automated test discovery. There is no need to keep a complete list of test files in another python file, which is what we currently do in run_tests.py. Multiple test functions per python test file. At the moment we can run multiple tests via command line parameters to the python test files. This of course means that you need to keep a list of '(test file, parameter)' tuples in run_tests.py, and you need some boilerplate dispatcher in your test file. This can lead to two situations: you may have a number of tests with numeric parameters, and no one knows what the parameters stand for. One example of this can be found in DIALS, where tst_index.py is run with numeric parameters 1 to 4 and 7 to 15, but not 5 and 6. The alternative situation is that you end up with a massive test file, that tests the entire class under test, but figuring out what specifically went wrong is probably more time consuming than necessary. In contrast in Python unittests any function in the file that is named test_* will be identified and run as a separate test. I find that this automatically leads to smaller and thus more readable tests. I understand that libtbx has been developed before Python unittest was around, that there is a substantial amount of code around that relies on current libtbx behaviour, and that there is no pressing need to change anything. However I would like to be able to take advantage of the tools that are out there, so I wrote a function that uses Python unittest for test discovery, and can be placed in run_tests.py. With this function you can then use regular Python unittests and run them with either through libtbx, with the python unittest tools, or directly. In each case you end up with the test results reported in the way you would expect. What I would like to do is integrate this logic in libtbx directly, so that we can use it for all cctbx modules. Since it does not modify libtbx default behaviour and only becomes active after a change in the run_tests.py of the relevant module it should be fully backwards compatible with virtually no downsides. What do you think? -Markus run_tests.py previously: (..) tst_list = ( "$D/test/algorithms/profile_model/nave/tst_model.py", ) (..) run_tests.py new: (..) tst_list = ( "$D/test/algorithms/profile_model/nave/tst_model.py", ) + discover_unittests("dlstbx") (..) discover_unittests function: def discover_unittests(module, pattern='tst_*.py'): try: import inspect import os import sys import unittest except: return tuple([]) dist_dir = libtbx.env.dist_path(module) found_tests = unittest.defaultTestLoader.discover(dist_dir, pattern=pattern) def recursive_TestSuite_to_list(suite): list = [] for t in suite: if isinstance(t, unittest.TestSuite): list.extend(recursive_TestSuite_to_list(t)) elif isinstance(t, unittest.TestCase): module = t.__class__.__module__ if module == 'unittest.loader': # This indicates a loading error. # Regenerate file name and try to run file directly. path = t._testMethodName.replace('.', os.path.sep) list.append("$D/%s.py" % path) else: module = inspect.getsourcefile(sys.modules[module]) function = "%s.%s" % (t.__class__.__name__, t._testMethodName) list.append([module, function]) else: raise Exception("Unknown test object (%s)" % t) return list test_list = recursive_TestSuite_to_list(found_tests) return tuple(test_list) -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 2

Re: [cctbxbb] Native python unittests for the libtbx testing framework
by Jan Marten Simons 20 May '15

20 May '15

Am Freitag 15 Mai 2015 18:07:33 schrieb Pavel Afonine: > Hi Markus, > > lucky you, I wish I have time for side projects! > > I've been working on cctbx (and writing tests!) for the past 12 years > and can't name any problem using "libtbx testing world". Most tests are > written this way. Why I am not so keen to use an alternative? > - why introduce inconsistency for no obvious clear pressing reasons; > - I have no spare time to learn and get used to an alternative > framework (again, for no obvious clear pressing reason); > - some rare developers (mostly postdocs that come and go) used > Python test framework in the past on the code that I work too and that I > have to maintain when people are gone. I find it is an irritating > overhead for me dealing with these tests, so typically I bother to > re-write them to use libtbx tools. Hi Pavel, those are sound and valid reasons from your point of view. But please also consider this alternative view: Coming from other python projects people are likely to have experience with the python unit test framework. As cctbx is using its own testing functionality people might be put off from writing tests or at least have to overcome this additional barrier on entry. I'm a strong supporter of sticking/moving to standards like PEP0008 [1] or pip/whl* based installs as conforming to those helps people to quickly find their way into the code. * wouldn't it be great if one could simply tell people this: "To install cctbx simply type 'pip install cctbx' in your terminal."? [1] https://www.python.org/dev/peps/pep-0008/ With regards, Dipl. Phys. Jan M. Simons Institute of Crystallography RWTH Aachen University

4 3