[cctbxbb] libtbx.easy_mp semantics

[Gabor Bunkoczi]

Hi Markus,

your observation of the facts is correct.

However, I would argue that such behaviour is not the fault of easy_mp 
as such, but of the code executing. If the code is supposed to be 
executed in a parallel fashion, it should not change any of its 
arguments, because it will never end up in a predictable state. In 
Python, it is possible to take liberties because of the way the 
multiprocessing module works, but easy_mp never gave such guarantees (in 
fact, "threading" is an allowed option), and if there is only a single 
thread executing, it is most efficient to use the main thread.

To solve your current problem, you can set LIBTBX_FORCE_PARALLEL, this 
should force easy_mp.parallel_map (I am assuming this is the function in 
question) to spawn new processes for each calculation, and this will 
effectively keep the current state of the arguments (I have not tested 
this, because your test is not present in my current copy, and it would 
not be convenient for me to update right now). Perhaps this can be 
turned into a command line argument to use when such code is executed, 
although I would advise against running such code in parallel. 
Alternative solution is to copy the object, although this is also not 
always an option.

BW, Gabor

On 2016-02-23 10:31, markus.gerstel at diamond.ac.uk wrote:
> Hello everyone,
> 
> Could someone please explain the libtbx.easy_mp semantics to me?
> 
> Currently, if you run easy_mp with multiple processes, objects are
> 'reset' back to their original state, ie. the state of the object in
> the main thread is retained, changes in the processing threads are
> dropped.
> 
> However if you run easy_mp with the number of processes set to 1,
> object states are modified, because the main thread is the executing
> thread.
> 
> How it this supposed to work?
> 
> For a more detailed test please run
> 
> $ libtbx.python libtbx/tst_easy_mp_state.py
> 
> The source code explains the test in more detail.
> 
> This behaviour currently causes dials.integrate to fail on bzipped
> input files.
> 
> Given that this code is used in many places and this particular
> feature is not obvious, it may explain other anecdotal emergent
> behaviour, too.
> 
> -Markus
> 
> --
> 
> This e-mail and any attachments may contain confidential, copyright
> and or privileged material, and are for the use of the intended
> addressee only. If you are not the intended addressee or an authorised
> recipient of the addressee please notify us of receipt by returning
> the e-mail and do not use, copy, retain, distribute or disclose the
> information in or attached to the e-mail.
> Any opinions expressed within this e-mail are those of the individual
> and not necessarily of Diamond Light Source Ltd.
> Diamond Light Source Ltd. cannot guarantee that this e-mail or any
> attachments are free from viruses and we cannot accept liability for
> any damage which you may sustain as a result of software viruses which
> may be transmitted in or with the message.
> Diamond Light Source Limited (company no. 4375679). Registered in
> England and Wales with its registered office at Diamond House, Harwell
> Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United
> Kingdom
> 
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb

-- 
##################################################

      Dr Gabor Bunkoczi

      Cambridge Institute for Medical Research
      Wellcome Trust/MRC Building
      Addenbrooke's Hospital
      Hills Road
      Cambridge CB2 0XY
##################################################