[cctbxbb] easy_mp.multi_core_run function

R.D. Oeffner rdo20 at cam.ac.uk
Mon Dec 19 09:11:49 PST 2016


Hi,

I needed a way to do multiprocessing job runs in a simple fashion on a 
multicore PC. Gabor provided much of the underlying machinery in libtbx 
for that. But I have often found I'm wasting too much time setting it 
all up. So I crafted a small function, easy_mp.multi_core_run(), out of 
his example in Computational Crystallography Newsletter (2013). 4, p21 
which does exactly that. Unlike easy_mp.parallel_map() this function 
continues if one of the jobs crashes catastrophically.

It takes as arguments name of function to be parallelised,  a list of 
argument tuples and number of processors to run on. As the jobs 
completes it returns an iterator of tuples of arguments with 
corresponding results.

Example:

   def RunMyJob(args1, args2):
     pass

   def GatherResults(parmres):
     pass

   from libtbx import easy_mp
   argstuples = [(args1a, args2a), (args1b, args2b), (args1c, args2c) ]

   for i, parmres in enumerate(easy_mp.multi_core_run( RunMyJob, 
argstuples, nproc)):
     GatherResults( parmres )


I'm interested in comments or if this  kind of simple function is 
already available elsewhere.


Regards,


Rob

-- 
Robert Oeffner, Ph.D.
Research Associate, The Read Group
Department of Haematology,
Cambridge Institute for Medical Research
University of Cambridge
Cambridge Biomedical Campus
Wellcome Trust/MRC Building
Hills Road
Cambridge CB2 0XY

www.cimr.cam.ac.uk/investigators/read/index.html
tel: +44(0)1223 763234
mobile: +44(0)7712 887162


More information about the cctbxbb mailing list