[cctbxbb] Should we enable boost threads in bootstrap?

Billy Poon bkpoon at lbl.gov
Tue Sep 26 08:11:55 PDT 2017


I just chatted with Tristan Croll from Cambridge at the Phenix developer
workshop. Would the Global Interpreter Lock be an issue?

https://docs.python.org/2.7/c-api/init.html#thread-state-and-the-global-interpreter-lock

It sounds like we should be releasing the lock before doing any threading
and then reacquiring the lock afterwards.

--
Billy K. Poon
Research Scientist, Molecular Biophysics and Integrated Bioimaging
Lawrence Berkeley National Laboratory
1 Cyclotron Road, M/S 33R0345
Berkeley, CA 94720
Tel: (510) 486-5709
Fax: (510) 486-5909
Web: https://phenix-online.org

On Thu, Aug 3, 2017 at 9:32 AM, Nigel Moriarty <nwmoriarty at lbl.gov> wrote:

> Gaeme
>
> We were evacuated yesterday ~2pm but the lab is reopened now and the
> servers are coming on line.
>
> Cheers
>
> Nigel
>
> ---
> Nigel W. Moriarty
> Building 33R0349, Molecular Biophysics and Integrated Bioimaging
> Lawrence Berkeley National Laboratory
> Berkeley, CA 94720-8235
> Phone : 510-486-5709 <(510)%20486-5709>     Email : NWMoriarty at LBL.gov
> Fax   : 510-486-5909 <(510)%20486-5909>       Web  : CCI.LBL.gov
>
> On Wed, Aug 2, 2017 at 10:29 PM, <Graeme.Winter at diamond.ac.uk> wrote:
>
>> Hi Rob
>>
>> For data processing we are very interested in classic threading because
>> our calculations include the word “if” - we have looked at pushing some of
>> the calculations which do not include if (i.e. summed area tables) and
>> there we get clobbered by the need to push the data of GPU memory also…
>>
>> Re: Python3
>>
>> Well there is a question. ISTR people have looked (with some success,
>> apparently) at building cctbx with cmake, which could be an alternative to
>> using SCons? Ergo not a p3 blocker (other blockers may exist, ymmv etc)
>>
>> Cheers Graeme
>>
>> PS - any ideas how long the server outage is likely to last? Appreciate
>> fires are a pretty good reason to take out kit. Not a problem in the UK
>> ATM, flooding more likely :-\
>>
>>
>>
>>
>>
>> > On 2 Aug 2017, at 22:13, Dr. Robert Oeffner <rdo20 at cam.ac.uk> wrote:
>> >
>> > If efficient threading is desired I would have thought that these days
>> GPUs are all the rage and that it would be worth looking into openCL and
>> CUDA implementations for doing this.
>> >
>> > On an unrelated note are there any thoughts on moving CCTBX to Python3?
>> One issue, which may not be insurmountable is that SCons does not yet
>> support Python3.
>> >
>> > Rob
>> >
>> >
>> >
>> > -----Original Message----- From: Graeme.Winter at diamond.ac.uk
>> > Sent: Tuesday, August 1, 2017 10:16 PM
>> > To: cctbxbb at phenix-online.org
>> > Subject: Re: [cctbxbb] Should we enable boost threads in bootstrap?
>> >
>> > Lee,
>> >
>> > End game for us is moving to “proper” threading i.e. lots of threads /
>> cores working on one problem in one address space - be it regular 20 core
>> xeon or 64 core KNL
>> >
>> > Boost threads came up in conversation today as a C++11 like threading
>> model, so I wondered if it would be a stepping stone...
>> >
>> > Don’t have this book, maybe should get it….
>> >
>> > Cheers Graeme
>> >
>> >
>> >
>> > On 1 Aug 2017, at 18:14, Lee O'Riordan <loriordan at lbl.gov<mailto:lori
>> ordan at lbl.gov>> wrote:
>> >
>> > Graeme, Nigel,
>> >
>> > I would be a little bit worried about Boost threads when it comes to
>> our KNL port of cctbx. In this instance the use of OpenMP or Intel TBB (at
>> least accordingly to Intel docs) would be optimal over boost threads (or
>> pthreads, etc.)[see Intel Xeon Phi High Performance Programming, KNL
>> edition P155 ---  no ebook, sorry]. That being said, there is no way to
>> know unless we try it out first, but it isn't something we can test right
>> now.
>> >
>> > As for Threads vs MP, this again falls into our KNL port, where threads
>> would be better suited (and become a necessity for optimal performance)
>> when running on high-core count devices. If the OpenMP functionality
>> exists, then maybe this would be a more portable way of taking advantage of
>> all cores/hyperthreads.
>> >
>> > In this instance I think turning boost threads on for a build-by-build
>> basis would be better, rather than as a default? Though if I am wrong, feel
>> free to correct me.
>> >
>> > Best,
>> > Lee.
>> >
>> > On Tue, Aug 1, 2017 at 9:36 AM, Nigel Moriarty <nwmoriarty at lbl.gov
>> <mailto:nwmoriarty at lbl.gov>> wrote:
>> > Graeme
>> >
>> > The short answer is "Why?" but that may start a very long discussion.
>> There are a number of multiprocessing modules in easy_mp that seem to cover
>> all the bases. Are there situations where threading is "better" to
>> multiprocessing?
>> >
>> > Articles on multiprocessing in cctbx.
>> >
>> > https://www.phenix-online.org/newsletter/CCN_2017_01.pdf#page=6
>> >
>> > https://www.phenix-online.org/newsletter/CCN_2013_07.pdf
>> >
>> > https://www.phenix-online.org/newsletter/CCN_2013_01.pdf
>> >
>> >
>> > Cheers
>> >
>> > Nigel
>> >
>> > ---
>> > Nigel W. Moriarty
>> > Building 33R0349, Molecular Biophysics and Integrated Bioimaging
>> > Lawrence Berkeley National Laboratory
>> > Berkeley, CA 94720-8235
>> > Phone : 510-486-5709<tel:(510)%20486-5709>     Email :
>> NWMoriarty at LBL.gov<mailto:NWMoriarty at LBL.gov>
>> > Fax   : 510-486-5909<tel:(510)%20486-5909>       Web  : CCI.LBL.gov<
>> http://cci.lbl.gov/>
>> >
>> > On Tue, Aug 1, 2017 at 7:54 AM, <Graeme.Winter at diamond.ac.uk<mailto:
>> Graeme.Winter at diamond.ac.uk>> wrote:
>> > Afternoon all,
>> >
>> > Should we do this? Any opinions? Could be useful for threads in a
>> semi-portable way...
>> >
>> > Thanks & cheerio Graeme
>> >
>> > --
>> > This e-mail and any attachments may contain confidential, copyright and
>> or privileged material, and are for the use of the intended addressee only.
>> If you are not the intended addressee or an authorised recipient of the
>> addressee please notify us of receipt by returning the e-mail and do not
>> use, copy, retain, distribute or disclose the information in or attached to
>> the e-mail.
>> > Any opinions expressed within this e-mail are those of the individual
>> and not necessarily of Diamond Light Source Ltd.
>> > Diamond Light Source Ltd. cannot guarantee that this e-mail or any
>> attachments are free from viruses and we cannot accept liability for any
>> damage which you may sustain as a result of software viruses which may be
>> transmitted in or with the message.
>> > Diamond Light Source Limited (company no. 4375679). Registered in
>> England and Wales with its registered office at Diamond House, Harwell
>> Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>> >
>> >
>> > _______________________________________________
>> > cctbxbb mailing list
>> > cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
>> > http://phenix-online.org/mailman/listinfo/cctbxbb
>> >
>> >
>> > _______________________________________________
>> > cctbxbb mailing list
>> > cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
>> > http://phenix-online.org/mailman/listinfo/cctbxbb
>> >
>> >
>> > _______________________________________________
>> > cctbxbb mailing list
>> > cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
>> > http://phenix-online.org/mailman/listinfo/cctbxbb
>> >
>> >
>> > _______________________________________________
>> > cctbxbb mailing list
>> > cctbxbb at phenix-online.org
>> > http://phenix-online.org/mailman/listinfo/cctbxbb
>> >
>> >
>> > ---
>> > This email has been checked for viruses by AVG.
>> > http://www.avg.com
>> > _______________________________________________
>> > cctbxbb mailing list
>> > cctbxbb at phenix-online.org
>> > http://phenix-online.org/mailman/listinfo/cctbxbb
>>
>>
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb at phenix-online.org
>> http://phenix-online.org/mailman/listinfo/cctbxbb
>>
>
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://phenix-online.org/pipermail/cctbxbb/attachments/20170926/e23d001e/attachment.htm>


More information about the cctbxbb mailing list