[cctbxbb] Simple C++11 async example

[Gabor Bunkoczi]

Hi Tristan,

there is an example of this in the Phaser repo:

/phaser/codebase/phaser/boost_python/phaser_ext.cpp

(search for ScopedGILRelease)

BW, Gabor

On Fri, Oct 2, 2020 at 12:37 PM Tristan Croll <tic20 at cam.ac.uk> wrote:

> I seem to remember in most cases for us the python threading isn't too
> much of an issue because the glue parts in python are relatively light
> and we have the option of releasing the GIL for the heavy duty parts.
>
>
> In principle, perhaps. In practice, as far as I can tell no part of the
> CCTBX ever releases the GIL. If I go to the top level of the cctbx repo,
> then:
>
> grep -ri gil
> # just returns a bunch of references to Richard Gildea, a series of amino
> acid sequences, plus a few other odds and ends
>
> # Python C API release-GIL macro
> grep -r Py_BEGIN_ALLOW_THREADS
> # nothing
>
> # Python C API release-GIL function
> grep -r PyEval_SaveThread
> # nothing
>
> Beyond that, it can be surprisingly tricky to pick exactly when you should
> and shouldn't release the GIL. If you have a function that *always*​ runs
> for a long time, then it's a no-brainer. But if you have one that sometimes
> runs long (i.e. when run over a long dataset) and sometimes very fast
> (where "very fast" is less than about 5-10 milliseconds) then it's a much
> trickier decision. The problem with the latter is that when a thread
> releases the GIL it typically won't get a chance to retrieve it for at
> least 5 milliseconds... if you run a separate thread that iterates a
> thousand times over a GIL-releasing method that would normally take 100
> microseconds per call, what would normally take 1 second can blow out to
> around a minute.
>
> Believe me when I say I've been down this route. In my first year or so of
> building ISOLDE, I tried very hard to do all my threading in Python. I
> succeeded in the sense that I got something running that was *mostly*​
> reasonably smooth, but it would occasionally lock up entirely for a few
> seconds before continuing and the amount of complexity involved in getting
> it to work even that well was a bit nuts. After taking the plunge and
> switching to C++ threading it's all gotten ever so much easier.
>
> ------------------------------
> *From:* cctbxbb-bounces at phenix-online.org <
> cctbxbb-bounces at phenix-online.org> on behalf of Nicholas Devenish <
> ndevenish at gmail.com>
> *Sent:* 02 October 2020 11:26
> *To:* cctbx mailing list <cctbxbb at phenix-online.org>
> *Subject:* Re: [cctbxbb] Simple C++11 async example
>
> Hi All,
>
> I seem to remember in most cases for us the python threading isn't too
> much of an issue because the glue parts in python are relatively light
> and we have the option of releasing the GIL for the heavy duty parts.
>
> On Fri, Oct 2, 2020 at 9:58 AM Tristan Croll <tic20 at cam.ac.uk> wrote:
> > On PyBind11, I have only good things to say about it. It's modelled off
> Boost.Python so the "feel" of it is very much the same, but being
> header-only it comes with no dependency baggage. For an example of a fairly
> big project wrapped with it, see
> https://github.com/tristanic/chimerax-clipper/tree/master/src/bindings.
> Whether it's desirable to go to the effort of converting the entire CCTBX?
> That's not for me to say.
>
> > From: cctbxbb-bounces at phenix-online.org <
> cctbxbb-bounces at phenix-online.org> on behalf of David Waterman <
> dgwaterman at gmail.com>
> > On that topic, I've heard many good things about pybind11. Does anyone
> have a feeling for how much work it would be to convert cctbx to pybind11
> bindings rather than Boost.Python. Is this feasible? Is it desirable?
>
> I really like pybind11 - it's active, has good documentation, and
> deals with python 3 well. Boost.python is the exact opposite of this -
> it's effectively dead, tries to pretend python3 doesn't exist, and the
> documentation is awful - I actually read the pybind11 documentation
> and then try to work out the differences to boost.python, the core is
> similar enough. However, I've found almost no documented cases of
> interoperation or of people porting large projects from one to the
> other - I've tried to identify small subsets that could be
> transitioned without doing everything at once, but sooner or later
> most objects end up getting passed through to different parts of the
> C++ code, so almost everything needs to be converted at once. An
> automated conversion would probably work for 95% of the declaration
> but then cause the remaining 5% to be more complicated. My frustration
> with boost hasn't reached high enough levels that this has been worth
> it yet, and we'd need complete agreement across cctbx and dials that
> this needed to be done. That's a high bar.
>
> Nick
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://phenix-online.org/pipermail/cctbxbb/attachments/20201002/97391266/attachment.htm>