As someone who spent a few days recently banging my head against boost.python bindings, I can’t help but feel any alternative must be more straightforward to use / debug… So, possibly desirable, but I would suspect that cctbx is very closely tied to boost so probably not feasible (at a guess) Cheers Graeme On 2 Oct 2020, at 09:47, David Waterman mailto:[email protected]> wrote: Thanks Tristan, This does look useful indeed. I note you use pybind11 in your example. Should this work just as easily with Boost.Python? On that topic, I've heard many good things about pybind11. Does anyone have a feeling for how much work it would be to convert cctbx to pybind11 bindings rather than Boost.Python. Is this feasible? Is it desirable? Cheers -- David On Thu, 1 Oct 2020 at 16:45, Tristan Croll mailto:[email protected]> wrote: Hi all, For anyone who's interested: following on from my quick demo of the use of std::async to simplify threading of long-running tasks called from Python, I've put some example code at https://github.com/tristanic/async-hello-world/tree/master/src (implemented as a ChimeraX plugin for demo purposes, but the C++ code and bindings are entirely generic). To re-iterate what this is about, and why I think it's important: in brief, for most real-world problems Python-level threading is not particularly useful since (except in specific cases) the Global Interpreter Lock limits the overall performance to no better than - and often much worse than - single-threaded code. While multiprocessing can be an option for some tasks, it's (a) slow to get started, (b) memory-hungry due to the need for each process to have a complete copy of all the data, (c) rather difficult to organise with respect to intercommunication, and (d) not very cross-platform friendly (I'm looking at you, Windows!). The point of this example is to show that with C++11 organising the threading at the C++ level can often end up being both the simplest and fastest approach - it's only a dozen or so extra lines of fairly boilerplate code required to allow the Python programmer to choose between running the exact same function either in the main thread or asynchronously in a new one. Why I think it's important: I've said in the past, and continue to believe, that we should be aiming for (interactive) model building, validation and refinement to merge together into a single integrated process - as much as anything, ISOLDE is meant to be the proof of principle that this is a possibility. But for that to work, high-performance and cross-platform parallelism is a real necessity. The approach described here is how I'm currently getting ChimeraX to maintain 30+ fps graphics performance while simultaneously running interactive molecular dynamics and structure factor calculations. I suspect there are many functions in CCTBX that could potentially benefit from this approach even when not considering GUI applications. Best regards, Tristan _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

Hi All, I seem to remember in most cases for us the python threading isn't too much of an issue because the glue parts in python are relatively light and we have the option of releasing the GIL for the heavy duty parts. On Fri, Oct 2, 2020 at 9:58 AM Tristan Croll wrote:

On PyBind11, I have only good things to say about it. It's modelled off Boost.Python so the "feel" of it is very much the same, but being header-only it comes with no dependency baggage. For an example of a fairly big project wrapped with it, see https://github.com/tristanic/chimerax-clipper/tree/master/src/bindings. Whether it's desirable to go to the effort of converting the entire CCTBX? That's not for me to say.

From: [email protected] on behalf of David Waterman On that topic, I've heard many good things about pybind11. Does anyone have a feeling for how much work it would be to convert cctbx to pybind11 bindings rather than Boost.Python. Is this feasible? Is it desirable?

I really like pybind11 - it's active, has good documentation, and deals with python 3 well. Boost.python is the exact opposite of this - it's effectively dead, tries to pretend python3 doesn't exist, and the documentation is awful - I actually read the pybind11 documentation and then try to work out the differences to boost.python, the core is similar enough. However, I've found almost no documented cases of interoperation or of people porting large projects from one to the other - I've tried to identify small subsets that could be transitioned without doing everything at once, but sooner or later most objects end up getting passed through to different parts of the C++ code, so almost everything needs to be converted at once. An automated conversion would probably work for 95% of the declaration but then cause the remaining 5% to be more complicated. My frustration with boost hasn't reached high enough levels that this has been worth it yet, and we'd need complete agreement across cctbx and dials that this needed to be done. That's a high bar. Nick

Hi Tristan, there is an example of this in the Phaser repo: /phaser/codebase/phaser/boost_python/phaser_ext.cpp (search for ScopedGILRelease) BW, Gabor On Fri, Oct 2, 2020 at 12:37 PM Tristan Croll wrote:

I seem to remember in most cases for us the python threading isn't too much of an issue because the glue parts in python are relatively light and we have the option of releasing the GIL for the heavy duty parts.

In principle, perhaps. In practice, as far as I can tell no part of the CCTBX ever releases the GIL. If I go to the top level of the cctbx repo, then:

grep -ri gil # just returns a bunch of references to Richard Gildea, a series of amino acid sequences, plus a few other odds and ends

# Python C API release-GIL macro grep -r Py_BEGIN_ALLOW_THREADS # nothing

# Python C API release-GIL function grep -r PyEval_SaveThread # nothing

Beyond that, it can be surprisingly tricky to pick exactly when you should and shouldn't release the GIL. If you have a function that *always*​ runs for a long time, then it's a no-brainer. But if you have one that sometimes runs long (i.e. when run over a long dataset) and sometimes very fast (where "very fast" is less than about 5-10 milliseconds) then it's a much trickier decision. The problem with the latter is that when a thread releases the GIL it typically won't get a chance to retrieve it for at least 5 milliseconds... if you run a separate thread that iterates a thousand times over a GIL-releasing method that would normally take 100 microseconds per call, what would normally take 1 second can blow out to around a minute.

Believe me when I say I've been down this route. In my first year or so of building ISOLDE, I tried very hard to do all my threading in Python. I succeeded in the sense that I got something running that was *mostly*​ reasonably smooth, but it would occasionally lock up entirely for a few seconds before continuing and the amount of complexity involved in getting it to work even that well was a bit nuts. After taking the plunge and switching to C++ threading it's all gotten ever so much easier.

------------------------------ *From:* [email protected] < [email protected]> on behalf of Nicholas Devenish < [email protected]> *Sent:* 02 October 2020 11:26 *To:* cctbx mailing list *Subject:* Re: [cctbxbb] Simple C++11 async example

Hi All,

I seem to remember in most cases for us the python threading isn't too much of an issue because the glue parts in python are relatively light and we have the option of releasing the GIL for the heavy duty parts.

...

On PyBind11, I have only good things to say about it. It's modelled off Boost.Python so the "feel" of it is very much the same, but being

On Fri, Oct 2, 2020 at 9:58 AM Tristan Croll wrote: header-only it comes with no dependency baggage. For an example of a fairly big project wrapped with it, see https://github.com/tristanic/chimerax-clipper/tree/master/src/bindings. Whether it's desirable to go to the effort of converting the entire CCTBX? That's not for me to say.

...

From: [email protected] < [email protected]> on behalf of David Waterman < [email protected]> On that topic, I've heard many good things about pybind11. Does anyone have a feeling for how much work it would be to convert cctbx to pybind11 bindings rather than Boost.Python. Is this feasible? Is it desirable?

I really like pybind11 - it's active, has good documentation, and deals with python 3 well. Boost.python is the exact opposite of this - it's effectively dead, tries to pretend python3 doesn't exist, and the documentation is awful - I actually read the pybind11 documentation and then try to work out the differences to boost.python, the core is similar enough. However, I've found almost no documented cases of interoperation or of people porting large projects from one to the other - I've tried to identify small subsets that could be transitioned without doing everything at once, but sooner or later most objects end up getting passed through to different parts of the C++ code, so almost everything needs to be converted at once. An automated conversion would probably work for 95% of the declaration but then cause the remaining 5% to be more complicated. My frustration with boost hasn't reached high enough levels that this has been worth it yet, and we'd need complete agreement across cctbx and dials that this needed to be done. That's a high bar.

Nick

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb