[cctbxbb] easy_mp.multi_core_run function

R.D. Oeffner rdo20 at cam.ac.uk
Mon Dec 19 09:11:49 PST 2016


I needed a way to do multiprocessing job runs in a simple fashion on a 
multicore PC. Gabor provided much of the underlying machinery in libtbx 
for that. But I have often found I'm wasting too much time setting it 
all up. So I crafted a small function, easy_mp.multi_core_run(), out of 
his example in Computational Crystallography Newsletter (2013). 4, p21 
which does exactly that. Unlike easy_mp.parallel_map() this function 
continues if one of the jobs crashes catastrophically.

It takes as arguments name of function to be parallelised,  a list of 
argument tuples and number of processors to run on. As the jobs 
completes it returns an iterator of tuples of arguments with 
corresponding results.


   def RunMyJob(args1, args2):

   def GatherResults(parmres):

   from libtbx import easy_mp
   argstuples = [(args1a, args2a), (args1b, args2b), (args1c, args2c) ]

   for i, parmres in enumerate(easy_mp.multi_core_run( RunMyJob, 
argstuples, nproc)):
     GatherResults( parmres )

I'm interested in comments or if this  kind of simple function is 
already available elsewhere.



Robert Oeffner, Ph.D.
Research Associate, The Read Group
Department of Haematology,
Cambridge Institute for Medical Research
University of Cambridge
Cambridge Biomedical Campus
Wellcome Trust/MRC Building
Hills Road
Cambridge CB2 0XY

tel: +44(0)1223 763234
mobile: +44(0)7712 887162

More information about the cctbxbb mailing list