I just chatted with Tristan Croll from Cambridge at the Phenix developer workshop. Would the Global Interpreter Lock be an issue?

https://docs.python.org/2.7/c-api/init.html#thread-state-and-the-global-interpreter-lock

It sounds like we should be releasing the lock before doing any threading and then reacquiring the lock afterwards.

--

Billy K. Poon

Research Scientist, Molecular Biophysics and Integrated Bioimaging

Lawrence Berkeley National Laboratory

1 Cyclotron Road, M/S 33R0345

Berkeley, CA 94720

Tel: (510) 486-5709

Fax: (510) 486-5909

Web: https://phenix-online.org

On Thu, Aug 3, 2017 at 9:32 AM, Nigel Moriarty <nwmoriarty@lbl.gov> wrote:

Gaeme

We were evacuated yesterday ~2pm but the lab is reopened now and the servers are coming on line.

Cheers

Nigel

---

Nigel W. Moriarty
Building 33R0349, Molecular Biophysics and Integrated Bioimaging

Lawrence Berkeley National Laboratory
Berkeley, CA 94720-8235
Phone : 510-486-5709     Email : NWMoriarty@LBL.gov
Fax   : 510-486-5909       Web  : CCI.LBL.gov

On Wed, Aug 2, 2017 at 10:29 PM, <Graeme.Winter@diamond.ac.uk> wrote:

Hi Rob

For data processing we are very interested in classic threading because our calculations include the word “if” - we have looked at pushing some of the calculations which do not include if (i.e. summed area tables) and there we get clobbered by the need to push the data of GPU memory also…

Re: Python3

Well there is a question. ISTR people have looked (with some success, apparently) at building cctbx with cmake, which could be an alternative to using SCons? Ergo not a p3 blocker (other blockers may exist, ymmv etc)

Cheers Graeme

PS - any ideas how long the server outage is likely to last? Appreciate fires are a pretty good reason to take out kit. Not a problem in the UK ATM, flooding more likely :-\

> On 2 Aug 2017, at 22:13, Dr. Robert Oeffner <rdo20@cam.ac.uk> wrote:
>
> If efficient threading is desired I would have thought that these days GPUs are all the rage and that it would be worth looking into openCL and CUDA implementations for doing this.
>
> On an unrelated note are there any thoughts on moving CCTBX to Python3? One issue, which may not be insurmountable is that SCons does not yet support Python3.
>
> Rob
>
>
>
> -----Original Message----- From: Graeme.Winter@diamond.ac.uk
> Sent: Tuesday, August 1, 2017 10:16 PM
> To: cctbxbb@phenix-online.org
> Subject: Re: [cctbxbb] Should we enable boost threads in bootstrap?
>
> Lee,
>
> End game for us is moving to “proper” threading i.e. lots of threads / cores working on one problem in one address space - be it regular 20 core xeon or 64 core KNL
>
> Boost threads came up in conversation today as a C++11 like threading model, so I wondered if it would be a stepping stone...
>
> Don’t have this book, maybe should get it….
>
> Cheers Graeme
>
>
>
> On 1 Aug 2017, at 18:14, Lee O'Riordan <loriordan@lbl.gov<mailto:loriordan@lbl.gov>> wrote:
>
> Graeme, Nigel,
>
> I would be a little bit worried about Boost threads when it comes to our KNL port of cctbx. In this instance the use of OpenMP or Intel TBB (at least accordingly to Intel docs) would be optimal over boost threads (or pthreads, etc.)[see Intel Xeon Phi High Performance Programming, KNL edition P155 ---  no ebook, sorry]. That being said, there is no way to know unless we try it out first, but it isn't something we can test right now.
>
> As for Threads vs MP, this again falls into our KNL port, where threads would be better suited (and become a necessity for optimal performance) when running on high-core count devices. If the OpenMP functionality exists, then maybe this would be a more portable way of taking advantage of all cores/hyperthreads.
>
> In this instance I think turning boost threads on for a build-by-build basis would be better, rather than as a default? Though if I am wrong, feel free to correct me.
>
> Best,
> Lee.
>
> On Tue, Aug 1, 2017 at 9:36 AM, Nigel Moriarty <nwmoriarty@lbl.gov<mailto:nwmoriarty@lbl.gov>> wrote:
> Graeme
>
> The short answer is "Why?" but that may start a very long discussion. There are a number of multiprocessing modules in easy_mp that seem to cover all the bases. Are there situations where threading is "better" to multiprocessing?
>
> Articles on multiprocessing in cctbx.
>
> https://www.phenix-online.org/newsletter/CCN_2017_01.pdf#page=6
>
> https://www.phenix-online.org/newsletter/CCN_2013_07.pdf
>
> https://www.phenix-online.org/newsletter/CCN_2013_01.pdf
>
>
> Cheers
>
> Nigel
>
> ---
> Nigel W. Moriarty
> Building 33R0349, Molecular Biophysics and Integrated Bioimaging
> Lawrence Berkeley National Laboratory
> Berkeley, CA 94720-8235
> Phone : 510-486-5709<tel:(510)%20486-5709>     Email : NWMoriarty@LBL.gov<mailto:NWMoriarty@LBL.gov>
> Fax   : 510-486-5909<tel:(510)%20486-5909>       Web  : CCI.LBL.gov<http://cci.lbl.gov/>
>
> On Tue, Aug 1, 2017 at 7:54 AM, <Graeme.Winter@diamond.ac.uk<mailto:Graeme.Winter@diamond.ac.uk>> wrote:
> Afternoon all,
>
> Should we do this? Any opinions? Could be useful for threads in a semi-portable way...
>
> Thanks & cheerio Graeme
>
> --
> This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
> Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
> Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
> Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb@phenix-online.org<mailto:cctbxbb@phenix-online.org>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb@phenix-online.org<mailto:cctbxbb@phenix-online.org>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb@phenix-online.org<mailto:cctbxbb@phenix-online.org>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb@phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
> ---
> This email has been checked for viruses by AVG.
> http://www.avg.com
> _______________________________________________
> cctbxbb mailing list
> cctbxbb@phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb


_______________________________________________
cctbxbb mailing list
cctbxbb@phenix-online.org
http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________
cctbxbb mailing list
cctbxbb@phenix-online.org
http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________
cctbxbb mailing list
cctbxbb@phenix-online.org
http://phenix-online.org/mailman/listinfo/cctbxbb