[cctbxbb] Should we enable boost threads in bootstrap?

[Luc Bourhis]

> On 27 Sep 2017, at 09:21, Graeme.Winter at Diamond.ac.uk wrote:
> 
> In principle boost threads is all we need – if we *are* calling back from Python then I imagine it would be something like:
>  
> at start of block: read data in Python (as hack)
> split of threads, do many calculations
> join threads
> move to next block
>  
> I would hope that anyone who has decided to use threads and uses Python has heard of the GIL, so the fact that this can be a problem should not block us from enabling boost threads

Well, the standard cctbx pattern goes as follow.

From the Python side, 

from scitbx.array_family import flex
py_arr = flex.double((1,2,3,4))
some_function(a)

and then on the C++ side,

void some_function(af::const_ref<double, grid_type> const &scitbx_arr) {
	// work on scitbx_arr
} 

The C++ code does not call back into the Python interpreter here. There are two separate reference counting: one for the Python object py_arr and one for the C++ handle scitbx_arr. For example, even if py_arr disappears after the call `some_function(a)`, scitbx_arr will not go away because the reference counter of the data underneath is unaffected and it is not zero by construction of the automatic conversions performed through the Boost::Python bridge.

The only way something can go wrong is if from the C++ `some_function`, one does explicitly call into the C API of the Python interpreter, or it’s Boost::Python interface. There are hardly any place like that. I can think of the bridge between Python and C++ I/O I wrote in boost_adaptbx but of nothing else actually. 

Note of course that if we use Boost::Thread only in `some_function`, breaking the work into blocks in there and distributing it to thread, this is the end of the discussion.

But you seem to want to write a Python interface to Boost::Thread. First, I would advise against that. I don’t see any need for it but please feel free to convince me otherwise!! 

But for the sake of discussion, let’s see whether this can be done. I guess you (when I say you, I don’t mean you Graeme in particular) want to be able to use it like so, on the Python side,

for block_index in xrange(…):
  thread_me(some_function, py_arr, block_index, block_index + block_size)

and on the C++ side,

void some_function(af::const_ref<double, grid_type> const &scitbx_arr, unsigned int block_start, unsigned int block_end) {
	// work on the block
}

where `some_function` would be auto-magically run in its own thread. The issue here is how to pass the function from Python to C++. Is it possible to write a `thread_me` with the following signature and then have wrap it with Boost::Python?

typedef void (*worker_type)(af::const_ref<double, grid_type> const &scitbx_arr, unsigned int block_start, unsigned int block_end);
void thread_me(worker_type const f) {
	// fire a Boost::Thread and execute f in it
}

I would not think so. So dead end I would say.

Best wishes,

Luc





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://phenix-online.org/pipermail/cctbxbb/attachments/20170928/16293610/attachment.htm>