<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On 27 Sep 2017, at 09:21, <a href="mailto:Graeme.Winter@Diamond.ac.uk" class="">Graeme.Winter@Diamond.ac.uk</a> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="WordSection1" style="page: WordSection1; font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">In principle boost threads is all we need – if we *<b class="">are</b>* calling back from Python then I imagine it would be something like:<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">at start of block: read data in Python (as hack)<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">split of threads, do many calculations<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">join threads<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">move to next block<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">I would hope that anyone who has decided to use threads and uses Python has heard of the GIL, so the fact that this can be a problem should not block us from enabling boost threads</span></div></div></div></blockquote><br class=""></div><div>Well, the standard cctbx pattern goes as follow.</div><div><br class=""></div><div>From the Python side, </div><div><br class=""></div><div><div>from scitbx.array_family import flex</div><div>py_arr = flex.double((1,2,3,4))</div><div>some_function(a)</div><div><br class=""></div><div>and then on the C++ side,</div><div><br class=""></div><div>void some_function(af::const_ref<double, grid_type> const &scitbx_arr) {</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>// work on scitbx_arr</div><div>} </div><div><br class=""></div><div>The C++ code does not call back into the Python interpreter here. There are two separate reference counting: one for the Python object py_arr and one for the C++ handle scitbx_arr. For example, even if py_arr disappears after the call `some_function(a)`, scitbx_arr will not go away because the reference counter of the data underneath is unaffected and it is not zero by construction of the automatic conversions performed through the Boost::Python bridge.</div><div><br class=""></div><div>The only way something can go wrong is if from the C++ `some_function`, one does explicitly call into the C API of the Python interpreter, or it’s Boost::Python interface. There are hardly any place like that. I can think of the bridge between Python and C++ I/O I wrote in boost_adaptbx but of nothing else actually. </div><div><br class=""></div><div>Note of course that if we use Boost::Thread only in `some_function`, breaking the work into blocks in there and distributing it to thread, this is the end of the discussion.</div><div><br class=""></div><div>But you seem to want to write a Python interface to Boost::Thread. First, I would advise against that. I don’t see any need for it but please feel free to convince me otherwise!! </div><div><br class=""></div><div>But for the sake of discussion, let’s see whether this can be done. I guess you (when I say you, I don’t mean you Graeme in particular) want to be able to use it like so, on the Python side,</div><div><br class=""></div><div>for block_index in xrange(…):</div><div> thread_me(some_function, py_arr, block_index, block_index + block_size)</div><div><br class=""></div><div>and on the C++ side,</div><div><br class=""></div><div>void some_function(af::const_ref<double, grid_type> const &scitbx_arr, unsigned int block_start, unsigned int block_end) {</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>// work on the block</div><div>}</div><div><br class=""></div><div>where `some_function` would be auto-magically run in its own thread. The issue here is how to pass the function from Python to C++. Is it possible to write a `thread_me` with the following signature and then have wrap it with Boost::Python?</div><div><br class=""></div><div>typedef void (*worker_type)(af::const_ref<double, grid_type> const &scitbx_arr, unsigned int block_start, unsigned int block_end);</div><div>void thread_me(worker_type const f) {</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>// fire a Boost::Thread and execute f in it</div><div>}</div><div><br class=""></div><div>I would not think so. So dead end I would say.</div><div><br class=""></div><div>Best wishes,</div><div><br class=""></div><div>Luc</div><div><br class=""></div><div><br class=""></div><div><br class=""></div><div><br class=""></div><div><br class=""></div></div></body></html>