[cctbxbb] Should we enable boost threads in bootstrap?

Tue Sep 26 08:17:50 PDT 2017

Highly recommend talking with Luc Bourhis, the most recent developer of
boost::threads support within cctbx.  He can answer these questions.
Nick

Nicholas K. Sauter, Ph. D.
Senior Scientist, Molecular Biophysics & Integrated Bioimaging Division
Lawrence Berkeley National Laboratory
1 Cyclotron Rd., Bldg. 33R0345
Berkeley, CA 94720
(510) 486-5713

On Tue, Sep 26, 2017 at 8:11 AM, Billy Poon <bkpoon at lbl.gov> wrote:

> I just chatted with Tristan Croll from Cambridge at the Phenix developer
> workshop. Would the Global Interpreter Lock be an issue?
>
> https://docs.python.org/2.7/c-api/init.html#thread-state-
> and-the-global-interpreter-lock
>
> It sounds like we should be releasing the lock before doing any threading
> and then reacquiring the lock afterwards.
>
> --
> Billy K. Poon
> Research Scientist, Molecular Biophysics and Integrated Bioimaging
> Lawrence Berkeley National Laboratory
> 1 Cyclotron Road, M/S 33R0345
> Berkeley, CA 94720
> Tel: (510) 486-5709
> Fax: (510) 486-5909
> Web: https://phenix-online.org
>
> On Thu, Aug 3, 2017 at 9:32 AM, Nigel Moriarty <nwmoriarty at lbl.gov> wrote:
>
>> Gaeme
>>
>> We were evacuated yesterday ~2pm but the lab is reopened now and the
>> servers are coming on line.
>>
>> Cheers
>>
>> Nigel
>>
>> ---
>> Nigel W. Moriarty
>> Building 33R0349, Molecular Biophysics and Integrated Bioimaging
>> Lawrence Berkeley National Laboratory
>> Berkeley, CA 94720-8235
>> Phone : 510-486-5709 <(510)%20486-5709>     Email : NWMoriarty at LBL.gov
>> Fax   : 510-486-5909 <(510)%20486-5909>       Web  : CCI.LBL.gov
>>
>> On Wed, Aug 2, 2017 at 10:29 PM, <Graeme.Winter at diamond.ac.uk> wrote:
>>
>>> Hi Rob
>>>
>>> For data processing we are very interested in classic threading because
>>> our calculations include the word “if” - we have looked at pushing some of
>>> the calculations which do not include if (i.e. summed area tables) and
>>> there we get clobbered by the need to push the data of GPU memory also…
>>>
>>> Re: Python3
>>>
>>> Well there is a question. ISTR people have looked (with some success,
>>> apparently) at building cctbx with cmake, which could be an alternative to
>>> using SCons? Ergo not a p3 blocker (other blockers may exist, ymmv etc)
>>>
>>> Cheers Graeme
>>>
>>> PS - any ideas how long the server outage is likely to last? Appreciate
>>> fires are a pretty good reason to take out kit. Not a problem in the UK
>>> ATM, flooding more likely :-\
>>>
>>>
>>>
>>>
>>>
>>> > On 2 Aug 2017, at 22:13, Dr. Robert Oeffner <rdo20 at cam.ac.uk> wrote:
>>> >
>>> > If efficient threading is desired I would have thought that these days
>>> GPUs are all the rage and that it would be worth looking into openCL and
>>> CUDA implementations for doing this.
>>> >
>>> > On an unrelated note are there any thoughts on moving CCTBX to
>>> Python3? One issue, which may not be insurmountable is that SCons does not
>>> yet support Python3.
>>> >
>>> > Rob
>>> >
>>> >
>>> >
>>> > -----Original Message----- From: Graeme.Winter at diamond.ac.uk
>>> > Sent: Tuesday, August 1, 2017 10:16 PM
>>> > To: cctbxbb at phenix-online.org
>>> > Subject: Re: [cctbxbb] Should we enable boost threads in bootstrap?
>>> >
>>> > Lee,
>>> >
>>> > End game for us is moving to “proper” threading i.e. lots of threads /
>>> cores working on one problem in one address space - be it regular 20 core
>>> xeon or 64 core KNL
>>> >
>>> > Boost threads came up in conversation today as a C++11 like threading
>>> model, so I wondered if it would be a stepping stone...
>>> >
>>> > Don’t have this book, maybe should get it….
>>> >
>>> > Cheers Graeme
>>> >
>>> >
>>> >
>>> > On 1 Aug 2017, at 18:14, Lee O'Riordan <loriordan at lbl.gov<mailto:lori
>>> ordan at lbl.gov>> wrote:
>>> >
>>> > Graeme, Nigel,
>>> >
>>> > I would be a little bit worried about Boost threads when it comes to
>>> our KNL port of cctbx. In this instance the use of OpenMP or Intel TBB (at
>>> least accordingly to Intel docs) would be optimal over boost threads (or
>>> pthreads, etc.)[see Intel Xeon Phi High Performance Programming, KNL
>>> edition P155 ---  no ebook, sorry]. That being said, there is no way to
>>> know unless we try it out first, but it isn't something we can test right
>>> now.
>>> >
>>> > As for Threads vs MP, this again falls into our KNL port, where
>>> threads would be better suited (and become a necessity for optimal
>>> performance) when running on high-core count devices. If the OpenMP
>>> functionality exists, then maybe this would be a more portable way of
>>> taking advantage of all cores/hyperthreads.
>>> >
>>> > In this instance I think turning boost threads on for a build-by-build
>>> basis would be better, rather than as a default? Though if I am wrong, feel
>>> free to correct me.
>>> >
>>> > Best,
>>> > Lee.
>>> >
>>> > On Tue, Aug 1, 2017 at 9:36 AM, Nigel Moriarty <nwmoriarty at lbl.gov
>>> <mailto:nwmoriarty at lbl.gov>> wrote:
>>> > Graeme
>>> >
>>> > The short answer is "Why?" but that may start a very long discussion.
>>> There are a number of multiprocessing modules in easy_mp that seem to cover
>>> all the bases. Are there situations where threading is "better" to
>>> multiprocessing?
>>> >
>>> > Articles on multiprocessing in cctbx.
>>> >
>>> > https://www.phenix-online.org/newsletter/CCN_2017_01.pdf#page=6
>>> >
>>> > https://www.phenix-online.org/newsletter/CCN_2013_07.pdf
>>> >
>>> > https://www.phenix-online.org/newsletter/CCN_2013_01.pdf
>>> >
>>> >
>>> > Cheers
>>> >
>>> > Nigel
>>> >
>>> > ---
>>> > Nigel W. Moriarty
>>> > Building 33R0349, Molecular Biophysics and Integrated Bioimaging
>>> > Lawrence Berkeley National Laboratory
>>> > Berkeley, CA 94720-8235
>>> > Phone : 510-486-5709<tel:(510)%20486-5709>     Email :
>>> NWMoriarty at LBL.gov<mailto:NWMoriarty at LBL.gov>
>>> > Fax   : 510-486-5909<tel:(510)%20486-5909>       Web  : CCI.LBL.gov<
>>> http://cci.lbl.gov/>
>>> >
>>> > On Tue, Aug 1, 2017 at 7:54 AM, <Graeme.Winter at diamond.ac.uk<mailto:
>>> Graeme.Winter at diamond.ac.uk>> wrote:
>>> > Afternoon all,
>>> >
>>> > Should we do this? Any opinions? Could be useful for threads in a
>>> semi-portable way...
>>> >
>>> > Thanks & cheerio Graeme
>>> >
>>> > --
>>> > This e-mail and any attachments may contain confidential, copyright
>>> and or privileged material, and are for the use of the intended addressee
>>> only. If you are not the intended addressee or an authorised recipient of
>>> the addressee please notify us of receipt by returning the e-mail and do
>>> not use, copy, retain, distribute or disclose the information in or
>>> attached to the e-mail.
>>> > Any opinions expressed within this e-mail are those of the individual
>>> and not necessarily of Diamond Light Source Ltd.
>>> > Diamond Light Source Ltd. cannot guarantee that this e-mail or any
>>> attachments are free from viruses and we cannot accept liability for any
>>> damage which you may sustain as a result of software viruses which may be
>>> transmitted in or with the message.
>>> > Diamond Light Source Limited (company no. 4375679). Registered in
>>> England and Wales with its registered office at Diamond House, Harwell
>>> Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>>> >
>>> >
>>> > _______________________________________________
>>> > cctbxbb mailing list
>>> > cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
>>> > http://phenix-online.org/mailman/listinfo/cctbxbb
>>> >
>>> >
>>> > _______________________________________________
>>> > cctbxbb mailing list
>>> > cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
>>> > http://phenix-online.org/mailman/listinfo/cctbxbb
>>> >
>>> >
>>> > _______________________________________________
>>> > cctbxbb mailing list
>>> > cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
>>> > http://phenix-online.org/mailman/listinfo/cctbxbb
>>> >
>>> >
>>> > _______________________________________________
>>> > cctbxbb mailing list
>>> > cctbxbb at phenix-online.org
>>> > http://phenix-online.org/mailman/listinfo/cctbxbb
>>> >
>>> >
>>> > ---
>>> > This email has been checked for viruses by AVG.
>>> > http://www.avg.com
>>> > _______________________________________________
>>> > cctbxbb mailing list
>>> > cctbxbb at phenix-online.org
>>> > http://phenix-online.org/mailman/listinfo/cctbxbb
>>>
>>>
>>> _______________________________________________
>>> cctbxbb mailing list
>>> cctbxbb at phenix-online.org
>>> http://phenix-online.org/mailman/listinfo/cctbxbb
>>>
>>
>>
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb at phenix-online.org
>> http://phenix-online.org/mailman/listinfo/cctbxbb
>>
>>
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://phenix-online.org/pipermail/cctbxbb/attachments/20170926/150c2456/attachment-0001.htm>