[cctbxbb] Should we enable boost threads in bootstrap?

[Graeme.Winter at diamond.ac.uk]

HI Luc

Re:

But you seem to want to write a Python interface to Boost::Thread. First, I would advise against that. I don’t see any need for it but please feel free to convince me otherwise!!

Nope thought had not occurred to me!

Our use case is to split into threads in C++ land and rejoin still in C++ land, to make the C++ code exercise more cores sharing one address space / one set of input read-only data objects.

We may gather inputs in Python land, but that is a different problem

Still, I can see no reason not to start using boost threading here…

Cheers Graeme


From: cctbxbb-bounces at phenix-online.org [mailto:cctbxbb-bounces at phenix-online.org] On Behalf Of Luc Bourhis
Sent: 28 September 2017 08:46
To: cctbx mailing list
Subject: Re: [cctbxbb] Should we enable boost threads in bootstrap?


On 27 Sep 2017, at 09:21, Graeme.Winter at Diamond.ac.uk<mailto:Graeme.Winter at Diamond.ac.uk> wrote:

In principle boost threads is all we need – if we *are* calling back from Python then I imagine it would be something like:

at start of block: read data in Python (as hack)
split of threads, do many calculations
join threads
move to next block

I would hope that anyone who has decided to use threads and uses Python has heard of the GIL, so the fact that this can be a problem should not block us from enabling boost threads

Well, the standard cctbx pattern goes as follow.

From the Python side,

from scitbx.array_family import flex
py_arr = flex.double((1,2,3,4))
some_function(a)

and then on the C++ side,

void some_function(af::const_ref<double, grid_type> const &scitbx_arr) {
            // work on scitbx_arr
}

The C++ code does not call back into the Python interpreter here. There are two separate reference counting: one for the Python object py_arr and one for the C++ handle scitbx_arr. For example, even if py_arr disappears after the call `some_function(a)`, scitbx_arr will not go away because the reference counter of the data underneath is unaffected and it is not zero by construction of the automatic conversions performed through the Boost::Python bridge.

The only way something can go wrong is if from the C++ `some_function`, one does explicitly call into the C API of the Python interpreter, or it’s Boost::Python interface. There are hardly any place like that. I can think of the bridge between Python and C++ I/O I wrote in boost_adaptbx but of nothing else actually.

Note of course that if we use Boost::Thread only in `some_function`, breaking the work into blocks in there and distributing it to thread, this is the end of the discussion.

But you seem to want to write a Python interface to Boost::Thread. First, I would advise against that. I don’t see any need for it but please feel free to convince me otherwise!!

But for the sake of discussion, let’s see whether this can be done. I guess you (when I say you, I don’t mean you Graeme in particular) want to be able to use it like so, on the Python side,

for block_index in xrange(…):
  thread_me(some_function, py_arr, block_index, block_index + block_size)

and on the C++ side,

void some_function(af::const_ref<double, grid_type> const &scitbx_arr, unsigned int block_start, unsigned int block_end) {
            // work on the block
}

where `some_function` would be auto-magically run in its own thread. The issue here is how to pass the function from Python to C++. Is it possible to write a `thread_me` with the following signature and then have wrap it with Boost::Python?

typedef void (*worker_type)(af::const_ref<double, grid_type> const &scitbx_arr, unsigned int block_start, unsigned int block_end);
void thread_me(worker_type const f) {
            // fire a Boost::Thread and execute f in it
}

I would not think so. So dead end I would say.

Best wishes,

Luc






-- 
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. 
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://phenix-online.org/pipermail/cctbxbb/attachments/20170928/ae33683a/attachment-0001.htm>