[cctbxbb] Simple C++11 async example

Tristan Croll tic20 at cam.ac.uk
Fri Oct 2 04:36:53 PDT 2020


I seem to remember in most cases for us the python threading isn't too
much of an issue because the glue parts in python are relatively light
and we have the option of releasing the GIL for the heavy duty parts.

In principle, perhaps. In practice, as far as I can tell no part of the CCTBX ever releases the GIL. If I go to the top level of the cctbx repo, then:

grep -ri gil
# just returns a bunch of references to Richard Gildea, a series of amino acid sequences, plus a few other odds and ends

# Python C API release-GIL macro
grep -r Py_BEGIN_ALLOW_THREADS
# nothing

# Python C API release-GIL function
grep -r PyEval_SaveThread
# nothing

Beyond that, it can be surprisingly tricky to pick exactly when you should and shouldn't release the GIL. If you have a function that always​ runs for a long time, then it's a no-brainer. But if you have one that sometimes runs long (i.e. when run over a long dataset) and sometimes very fast (where "very fast" is less than about 5-10 milliseconds) then it's a much trickier decision. The problem with the latter is that when a thread releases the GIL it typically won't get a chance to retrieve it for at least 5 milliseconds... if you run a separate thread that iterates a thousand times over a GIL-releasing method that would normally take 100 microseconds per call, what would normally take 1 second can blow out to around a minute.

Believe me when I say I've been down this route. In my first year or so of building ISOLDE, I tried very hard to do all my threading in Python. I succeeded in the sense that I got something running that was mostly​ reasonably smooth, but it would occasionally lock up entirely for a few seconds before continuing and the amount of complexity involved in getting it to work even that well was a bit nuts. After taking the plunge and switching to C++ threading it's all gotten ever so much easier.

________________________________
From: cctbxbb-bounces at phenix-online.org <cctbxbb-bounces at phenix-online.org> on behalf of Nicholas Devenish <ndevenish at gmail.com>
Sent: 02 October 2020 11:26
To: cctbx mailing list <cctbxbb at phenix-online.org>
Subject: Re: [cctbxbb] Simple C++11 async example

Hi All,

I seem to remember in most cases for us the python threading isn't too
much of an issue because the glue parts in python are relatively light
and we have the option of releasing the GIL for the heavy duty parts.

On Fri, Oct 2, 2020 at 9:58 AM Tristan Croll <tic20 at cam.ac.uk> wrote:
> On PyBind11, I have only good things to say about it. It's modelled off Boost.Python so the "feel" of it is very much the same, but being header-only it comes with no dependency baggage. For an example of a fairly big project wrapped with it, see https://github.com/tristanic/chimerax-clipper/tree/master/src/bindings. Whether it's desirable to go to the effort of converting the entire CCTBX? That's not for me to say.

> From: cctbxbb-bounces at phenix-online.org <cctbxbb-bounces at phenix-online.org> on behalf of David Waterman <dgwaterman at gmail.com>
> On that topic, I've heard many good things about pybind11. Does anyone have a feeling for how much work it would be to convert cctbx to pybind11 bindings rather than Boost.Python. Is this feasible? Is it desirable?

I really like pybind11 - it's active, has good documentation, and
deals with python 3 well. Boost.python is the exact opposite of this -
it's effectively dead, tries to pretend python3 doesn't exist, and the
documentation is awful - I actually read the pybind11 documentation
and then try to work out the differences to boost.python, the core is
similar enough. However, I've found almost no documented cases of
interoperation or of people porting large projects from one to the
other - I've tried to identify small subsets that could be
transitioned without doing everything at once, but sooner or later
most objects end up getting passed through to different parts of the
C++ code, so almost everything needs to be converted at once. An
automated conversion would probably work for 95% of the declaration
but then cause the remaining 5% to be more complicated. My frustration
with boost hasn't reached high enough levels that this has been worth
it yet, and we'd need complete agreement across cctbx and dials that
this needed to be done. That's a high bar.

Nick

_______________________________________________
cctbxbb mailing list
cctbxbb at phenix-online.org
http://phenix-online.org/mailman/listinfo/cctbxbb
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://phenix-online.org/pipermail/cctbxbb/attachments/20201002/a2f39492/attachment.htm>


More information about the cctbxbb mailing list