<div dir="ltr">Hi Tristan,<div><br></div><div>there is an example of this in the Phaser repo:</div><div><br></div><div>/phaser/codebase/phaser/boost_python/phaser_ext.cpp</div><div><br></div><div>(search for ScopedGILRelease)</div><div><br></div><div>BW, Gabor</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Oct 2, 2020 at 12:37 PM Tristan Croll <<a href="mailto:tic20@cam.ac.uk">tic20@cam.ac.uk</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<blockquote style="border-color:rgb(200,200,200);border-left-width:3px;border-left-style:solid;padding-left:1ex;margin-left:0.8ex;color:rgb(102,102,102)">
<font size="2"><span style="font-size:11pt">I seem to remember in most cases for us the python threading isn't too<br>
much of an issue because the glue parts in python are relatively light<br>
</span></font>
<div><font size="2"><span style="font-size:11pt">and we have the option of releasing the GIL for the heavy duty parts.</span></font></div>
</blockquote>
<div><font size="2"><span style="font-size:11pt"><br>
</span></font></div>
<div><font size="2"><span style="font-size:11pt">In principle, perhaps. In practice, as far as I can tell no part of the CCTBX ever releases the GIL. If I go to the top level of the cctbx repo, then:</span></font></div>
<div><font size="2"><span style="font-size:11pt"><br>
</span></font></div>
<div><font size="2"><span style="font-size:11pt">grep -ri gil</span></font></div>
<div><font size="2"><span style="font-size:11pt"># just returns a bunch of references to Richard Gildea, a series of amino acid sequences, plus a few other odds and ends</span></font></div>
<div><font size="2"><span style="font-size:11pt"><br>
</span></font></div>
<div><font size="2"><span style="font-size:11pt"># Python C API release-GIL macro<br>
</span></font></div>
<div><font size="2"><span style="font-size:11pt">grep -r Py_BEGIN_ALLOW_THREADS</span></font></div>
<div><font size="2"><span style="font-size:11pt"># nothing</span></font></div>
<div><font size="2"><span style="font-size:11pt"><br>
</span></font></div>
<div><font size="2"><span style="font-size:11pt"># Python C API release-GIL function</span></font></div>
<div><font size="2"><span style="font-size:11pt">grep -r PyEval_SaveThread<br>
</span></font></div>
<div><font size="2"><span style="font-size:11pt"># nothing</span></font></div>
<div><font size="2"><span style="font-size:11pt"><br>
</span></font></div>
<div><font size="2"><span style="font-size:11pt">Beyond that, it can be surprisingly tricky to pick exactly when you should and shouldn't release the GIL. If you have a function that
<b>always</b> runs for a long time, then it's a no-brainer. But if you have one that sometimes runs long (i.e. when run over a long dataset) and sometimes very fast (where "very fast" is less than about 5-10 milliseconds) then it's a much trickier decision.
The problem with the latter is that when a thread releases the GIL it typically won't get a chance to retrieve it for at least 5 milliseconds... if you run a separate thread that iterates a thousand times over a GIL-releasing method that would normally take
100 microseconds per call, what would normally take 1 second can blow out to around a minute.
<br>
</span></font></div>
<div><font size="2"><span style="font-size:11pt"><br>
</span></font></div>
<div><font size="2"><span style="font-size:11pt">Believe me when I say I've been down this route. In my first year or so of building ISOLDE, I tried very hard to do all my threading in Python. I succeeded in the sense that I got something running that was
<b>mostly</b> reasonably smooth, but it would occasionally lock up entirely for a few seconds before continuing and the amount of complexity involved in getting it to work even that well was a bit nuts. After taking the plunge and switching to C++ threading
it's all gotten ever so much easier.<br>
</span></font></div>
<div><font size="2"><span style="font-size:11pt"><br>
</span></font></div>
<div id="gmail-m_-6810891950453166390appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_-6810891950453166390divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> <a href="mailto:cctbxbb-bounces@phenix-online.org" target="_blank">cctbxbb-bounces@phenix-online.org</a> <<a href="mailto:cctbxbb-bounces@phenix-online.org" target="_blank">cctbxbb-bounces@phenix-online.org</a>> on behalf of Nicholas Devenish <<a href="mailto:ndevenish@gmail.com" target="_blank">ndevenish@gmail.com</a>><br>
<b>Sent:</b> 02 October 2020 11:26<br>
<b>To:</b> cctbx mailing list <<a href="mailto:cctbxbb@phenix-online.org" target="_blank">cctbxbb@phenix-online.org</a>><br>
<b>Subject:</b> Re: [cctbxbb] Simple C++11 async example</font>
<div> </div>
</div>
<div><font size="2"><span style="font-size:11pt">
<div>Hi All,<br>
<br>
I seem to remember in most cases for us the python threading isn't too<br>
much of an issue because the glue parts in python are relatively light<br>
and we have the option of releasing the GIL for the heavy duty parts.<br>
<br>
On Fri, Oct 2, 2020 at 9:58 AM Tristan Croll <<a href="mailto:tic20@cam.ac.uk" target="_blank">tic20@cam.ac.uk</a>> wrote:<br>
> On PyBind11, I have only good things to say about it. It's modelled off Boost.Python so the "feel" of it is very much the same, but being header-only it comes with no dependency baggage. For an example of a fairly big project wrapped with it, see
<a href="https://github.com/tristanic/chimerax-clipper/tree/master/src/bindings" target="_blank">
https://github.com/tristanic/chimerax-clipper/tree/master/src/bindings</a>. Whether it's desirable to go to the effort of converting the entire CCTBX? That's not for me to say.<br>
<br>
> From: <a href="mailto:cctbxbb-bounces@phenix-online.org" target="_blank">cctbxbb-bounces@phenix-online.org</a> <<a href="mailto:cctbxbb-bounces@phenix-online.org" target="_blank">cctbxbb-bounces@phenix-online.org</a>> on behalf of David Waterman <<a href="mailto:dgwaterman@gmail.com" target="_blank">dgwaterman@gmail.com</a>><br>
> On that topic, I've heard many good things about pybind11. Does anyone have a feeling for how much work it would be to convert cctbx to pybind11 bindings rather than Boost.Python. Is this feasible? Is it desirable?<br>
<br>
I really like pybind11 - it's active, has good documentation, and<br>
deals with python 3 well. Boost.python is the exact opposite of this -<br>
it's effectively dead, tries to pretend python3 doesn't exist, and the<br>
documentation is awful - I actually read the pybind11 documentation<br>
and then try to work out the differences to boost.python, the core is<br>
similar enough. However, I've found almost no documented cases of<br>
interoperation or of people porting large projects from one to the<br>
other - I've tried to identify small subsets that could be<br>
transitioned without doing everything at once, but sooner or later<br>
most objects end up getting passed through to different parts of the<br>
C++ code, so almost everything needs to be converted at once. An<br>
automated conversion would probably work for 95% of the declaration<br>
but then cause the remaining 5% to be more complicated. My frustration<br>
with boost hasn't reached high enough levels that this has been worth<br>
it yet, and we'd need complete agreement across cctbx and dials that<br>
this needed to be done. That's a high bar.<br>
<br>
Nick<br>
<br>
_______________________________________________<br>
cctbxbb mailing list<br>
<a href="mailto:cctbxbb@phenix-online.org" target="_blank">cctbxbb@phenix-online.org</a><br>
<a href="http://phenix-online.org/mailman/listinfo/cctbxbb" target="_blank">http://phenix-online.org/mailman/listinfo/cctbxbb</a><br>
</div>
</span></font></div>
</div>
_______________________________________________<br>
cctbxbb mailing list<br>
<a href="mailto:cctbxbb@phenix-online.org" target="_blank">cctbxbb@phenix-online.org</a><br>
<a href="http://phenix-online.org/mailman/listinfo/cctbxbb" rel="noreferrer" target="_blank">http://phenix-online.org/mailman/listinfo/cctbxbb</a><br>
</blockquote></div>