[cctbxbb] Boost Python 1.56

markus.gerstel at diamond.ac.uk markus.gerstel at diamond.ac.uk
Fri Apr 14 03:17:46 PDT 2017


Hi Billy,

thank you very much. This should help the dials installer greatly, which blew up from 158 MB to 226 MB :)
There are still some nonfunctional files in the boost distribution, which can be removed with

find libs -type d | grep "/\(test\|doc\|dot_net_example\|example\|examples\|tutorial\|xmldoc\)$" | xargs rm -r
find libs -type f | grep "\(.html\|Jamfile.v2\)$" | xargs rm
find . -type d -empty -delete

and by packing it using gzip -9 instead of the default level gzip compression you can get the download size to below 10MB. (simply "export GZIP=-9" before tar cfz ...)

(Incidentally - is there a valid reason why we use gzip instead of at least bzip2 or even xz?)

-Markus



________________________________
From: cctbxbb-bounces at phenix-online.org [cctbxbb-bounces at phenix-online.org] on behalf of Billy Poon [bkpoon at lbl.gov]
Sent: Friday, April 14, 2017 01:02
To: cctbx mailing list
Subject: Re: [cctbxbb] Boost Python 1.56

Hi Graeme,

I hope Markus's commands were helpful. The only tricky thing was the deletion of the build directory. If you had set up the cctbx environment, then python would have pointed to something in the build directory. You would have need to explicitly use ./base/bin/python or the system python with bootstrap.py instead.

Also, I stripped extraneous stuff from the Boost tarball according to how earlier boost tarballs were built. This will improve download times and fix the Windows installation. Basically, only the following things are kept,

LICENSE_1_0.txt
boost/
libs/date_time
libs/detail
libs/filesystem
libs/program_options
libs/python
libs/system
libs/thread

It's available now if you use authenticated access (--cciuser flag for bootstrap.py) and will propagate for unauthenticated access (http://cci.lbl.gov/repositories) by 9:30 pm Pacific time (or 5:30 am UK time). Let me know of any issues. Thanks!

--
Billy K. Poon
Research Scientist, Molecular Biophysics and Integrated Bioimaging
Lawrence Berkeley National Laboratory
1 Cyclotron Road, M/S 33R0345
Berkeley, CA 94720
Tel: (510) 486-5709
Fax: (510) 486-5909
Web: https://phenix-online.org

On Thu, Apr 13, 2017 at 8:02 AM, R.D. Oeffner <rdo20 at cam.ac.uk<mailto:rdo20 at cam.ac.uk>> wrote:

Sorry this was to Billy only. Please ignore.



Rob





On 13/04/2017 16:00, R.D. Oeffner wrote:

Hi Billy,

As you'll know the Windows builds failed with the new boost libraries. this is due to some filenames exceeding the 255 char limit on Windows. It worked fine on my nightly build because the build folder isn't as deeply nested as on pulse and flux. On pulse and flux it dies in tarfile.TarFile._extract_member() which I have made a monkey patch for which appears to work by ignoring the offending files.

However, as all the long filenames seem to come from html folders within boost presumably these are just documentation and it would be better to exclude these altogether in the cci subset of boost. People can always go online if they want to read up on the documentation for boost. Besides bootstrap.py is already quite bloated.

Let me know what you think,

Rob



On 12/04/2017 21:47, Billy Poon wrote:

Hi everyone,

Boost 1.56 is now available and the tests have been updated. To update an existing installtion, you have to delete the "build" directory first. If you don't, I guess some files are not recompiled with the new Boost headers, which will cause  cctbx_project/scitbx/random/tests/tst_random.py to expect the old values. Then run bootstrap.py with hot, update, and build to rebuild.

cd <installation directory>
rm -fr build
python bootstrap.py hot update build --builder=cctbx

Let me know if there are any issues. Thanks!


--
Billy K. Poon
Research Scientist, Molecular Biophysics and Integrated Bioimaging
Lawrence Berkeley National Laboratory
1 Cyclotron Road, M/S 33R0345
Berkeley, CA 94720
Tel: (510) 486-5709<tel:(510)%20486-5709>
Fax: (510) 486-5909<tel:(510)%20486-5909>
Web: https://phenix-online.org

On Fri, Apr 7, 2017 at 1:08 PM, Billy Poon <bkpoon at lbl.gov<mailto:bkpoon at lbl.gov>> wrote:
I can do the switch to Boost 1.56 next Wednesday now that Dials 1.5 has been released. I'm just double-checking some other Phenix tests.

Once the update is live, bootstrap.py should delete the existing modules/boost directory and replace it with the new one. And then scons should recompile anything that depends on boost. If in doubt, you can manually delete the module/boost and build directories.

--
Billy K. Poon
Research Scientist, Molecular Biophysics and Integrated Bioimaging
Lawrence Berkeley National Laboratory
1 Cyclotron Road, M/S 33R0345
Berkeley, CA 94720
Tel: (510) 486-5709<tel:(510)%20486-5709>
Fax: (510) 486-5909<tel:(510)%20486-5909>
Web: https://phenix-online.org
On Thu, Apr 6, 2017 at 9:32 AM, Billy Poon <bkpoon at lbl.gov<mailto:bkpoon at lbl.gov>> wrote:
Hi James,

I agree with Graeme's tests and I would add

cctbx_regression.test_nightly

That command is a shortcut for running the test modules for libtbx, boost_adaptbx, scitbx, cctbx, iotbx, dxtbx, and smtbx. The mmtbx module is also tested if chem_data is available. Should we add rstbx to the shortcut (cctbx_project/cctbx/command_line/cctbx_test_nightly.py)?

--
Billy K. Poon
Research Scientist, Molecular Biophysics and Integrated Bioimaging
Lawrence Berkeley National Laboratory
1 Cyclotron Road, M/S 33R0345
Berkeley, CA 94720
Tel: (510) 486-5709<tel:(510)%20486-5709>
Fax: (510) 486-5909<tel:(510)%20486-5909>
Web: https://phenix-online.org
On Thu, Apr 6, 2017 at 1:37 AM, <richard.gildea at diamond.ac.uk<mailto:richard.gildea at diamond.ac.uk>> wrote:
I've made a start on transcribing this document here:

https://github.com/cctbx/cctbx_project/wiki/cctbx-Developer-Guidance

It probably still needs cleaning up a bit (e.g. I couldn't figure out quickly how to do 3-level list nesting, 1.a.i etc.) and updating to reflect current practice (e.g. git not svn).

Cheers,

Richard

Dr Richard Gildea
Data Analysis Scientist
Tel: +441235 77 8078<tel:%2B441235%2077%208078>

Diamond Light Source Ltd.
Diamond House
Harwell Science & Innovation Campus
Didcot
Oxfordshire
OX11 0DE

________________________________________
From: cctbxbb-bounces at phenix-online.org<mailto:cctbxbb-bounces at phenix-online.org> [cctbxbb-bounces at phenix-online.org<mailto:cctbxbb-bounces at phenix-online.org>] on behalf of Pavel Afonine [pafonine at lbl.gov<mailto:pafonine at lbl.gov>]
Sent: 06 April 2017 09:07
To: cctbx mailing list
Subject: Re: [cctbxbb] Boost Python 1.56

Hi Graeme,

hm.. this is a good question. We've been through back-and-forth
iterations of editing this file and I think the latest I have is from
Paul. But I can't find a non-PDF version of it. Paul: do you have an
editable version of this file?

Thanks,
Pavel

On 4/6/17 13:45, Graeme.Winter at diamond.ac.uk<mailto:Graeme.Winter at diamond.ac.uk> wrote:
> Hi Pavel
>
> These all seem sensible
>
> If you have the original non pdf document it may be easier to transcribe this over..
>
> I also note that it lacks the actual detail on how to run tests! However would be happy to add this once on wiki
>
> Best wishes Graeme
>
> On 6 Apr 2017, at 04:00, Pavel Afonine <pafonine at lbl.gov<mailto:pafonine at lbl.gov><mailto:pafonine at lbl.gov<mailto:pafonine at lbl.gov>>> wrote:
>
> Not sure if that answers your questions but once upon a time we here at Berkeley tried to write a some sort of document that was supposed to answer questions like this. Attached. By no means it is complete, up-to-date, etc, but it might be worth reading for anyone who contributes to cctbx. (Even not sure if I'm sending the latest version).
> Unfortunately, nobody bothered to put it in some central place.
>
> Pavel
>
> On 4/6/17 10:51, James Holton wrote:
> Hey Billy,
>
> On a related note. How do I run these regression tests before committing something into Git? Is there a document on dials regression testing I can't find?
>
> -James
>
> On Apr 5, 2017, at 3:38 PM, Billy Poon <bkpoon at lbl.gov<mailto:bkpoon at lbl.gov><mailto:bkpoon at lbl.gov<mailto:bkpoon at lbl.gov>>> wrote:
>
> Hi all,
>
> I tested Boost 1.56 on our buildbot servers and got some new test failures with
>
> cctbx_project/scitbx/array_family/boost_python/tst_flex.py
> cctbx_project/scitbx/random/tests/tst_random.py
>
> The full log for CentOS 6 can be found at
>
> http://cci-vm-6.lbl.gov:8010/builders/phenix-nightly-intel-linux-2.6-x86_64-centos6/builds/601/steps/test%20cctbx_regression.test_nightly/logs/stdio
>
> It looks like the errors are related to random number generation. For a given seed, would the sequence of numbers change when Boost is changed? I did a diff between Boost 1.56 and the current Boost and could not see any changes that immediately stood out as being related to random numbers.
>
> Are these tests failing for others as well?
>
> --
> Billy K. Poon
> Research Scientist, Molecular Biophysics and Integrated Bioimaging
> Lawrence Berkeley National Laboratory
> 1 Cyclotron Road, M/S 33R0345
> Berkeley, CA 94720
> Tel: (510) 486-5709<tel:%28510%29%20486-5709>
> Fax: (510) 486-5909<tel:%28510%29%20486-5909>
> Web: https://phenix-online.org<https://phenix-online.org/>
>
> On Wed, Apr 5, 2017 at 8:12 AM, Charles Ballard <charles.ballard at stfc.ac.uk<mailto:charles.ballard at stfc.ac.uk><mailto:charles.ballard at stfc.ac.uk<mailto:charles.ballard at stfc.ac.uk>>> wrote:
> FYI,  we (CCP4) have been using 1.56 for building cctbx/phaser/dials for the last while with no issues.  Don't know about 1.60, but 1.59 causes issues with the boost python make_getter and make_setter (initialisation of none const reference if the passed type is a temporary).
>
> Charles
>
> On 3 Apr 2017, at 14:31, Luc Bourhis wrote:
>
> Hi all,
>
> everybody seemed to agree but then it was proposed to move straight to Boost 1.60, and this caused troubles. Could we consider again to move to at least 1.56? As far as I can tell, this does not cause any issue and as stated one year ago, it would help me and Olex 2.
>
> Thanks,
>
> Luc
>
> On 10 Feb 2016, at 15:17, Nicholas Sauter <nksauter at lbl.gov<mailto:nksauter at lbl.gov><mailto:nksauter at lbl.gov<mailto:nksauter at lbl.gov>>> wrote:
>
> Nigel, Billy & Aaron,
>
> I completely endorse this move to Boost 1.56.  Can we update our build?
>
> Nick
>
> Nicholas K. Sauter, Ph. D.
> Computer Staff Scientist, Molecular Biophysics and Integrated Bioimaging Division
> Lawrence Berkeley National Laboratory
> 1 Cyclotron Rd., Bldg. 33R0345
> Berkeley, CA 94720
> (510) 486-5713<tel:%28510%29%20486-5713><tel:%28510%29%20486-5713>
>
> On Wed, Feb 10, 2016 at 2:41 PM, Luc Bourhis <luc_j_bourhis at mac.com<mailto:luc_j_bourhis at mac.com><mailto:luc_j_bourhis at mac.com<mailto:luc_j_bourhis at mac.com>>> wrote:
> Hi,
>
> I have improvements to the smtbx on their way to be committed which require Boost version 1.56. This is related to Boost.Threads, whose support I re-activated a few months ago on Nick's request. I need the function boost::thread::physical_concurrency which returns the number of physical cores on the machine, as opposed to virtual cores when hyperthreading is enabled (which it is by default on any Intel machine). That function is not available in Boost 1.55 which is the version currently used in the nightly tests: it appeared in 1.56.
>
> So, would it be possible to move to Boost 1.56? Otherwise, I will need to backport that function. Not too difficult but not thrilling.
>
> Best wishes,
>
> Luc
>
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org><mailto:cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org><mailto:cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org><mailto:cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org><mailto:cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org><mailto:cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org><mailto:cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
> <cctbx-developer-guidance-08-2015.pdf>_______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org><mailto:cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>

_______________________________________________
cctbxbb mailing list
cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
http://phenix-online.org/mailman/listinfo/cctbxbb

--
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom


_______________________________________________
cctbxbb mailing list
cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
http://phenix-online.org/mailman/listinfo/cctbxbb


_______________________________________________
cctbxbb mailing list
cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
http://phenix-online.org/mailman/listinfo/cctbxbb






_______________________________________________
cctbxbb mailing list
cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
http://phenix-online.org/mailman/listinfo/cctbxbb





More information about the cctbxbb mailing list