[cctbxbb] easy_mp.multi_core_run function
R.D. Oeffner
rdo20 at cam.ac.uk
Mon Dec 19 09:11:49 PST 2016
Hi,
I needed a way to do multiprocessing job runs in a simple fashion on a
multicore PC. Gabor provided much of the underlying machinery in libtbx
for that. But I have often found I'm wasting too much time setting it
all up. So I crafted a small function, easy_mp.multi_core_run(), out of
his example in Computational Crystallography Newsletter (2013). 4, p21
which does exactly that. Unlike easy_mp.parallel_map() this function
continues if one of the jobs crashes catastrophically.
It takes as arguments name of function to be parallelised, a list of
argument tuples and number of processors to run on. As the jobs
completes it returns an iterator of tuples of arguments with
corresponding results.
Example:
def RunMyJob(args1, args2):
pass
def GatherResults(parmres):
pass
from libtbx import easy_mp
argstuples = [(args1a, args2a), (args1b, args2b), (args1c, args2c) ]
for i, parmres in enumerate(easy_mp.multi_core_run( RunMyJob,
argstuples, nproc)):
GatherResults( parmres )
I'm interested in comments or if this kind of simple function is
already available elsewhere.
Regards,
Rob
--
Robert Oeffner, Ph.D.
Research Associate, The Read Group
Department of Haematology,
Cambridge Institute for Medical Research
University of Cambridge
Cambridge Biomedical Campus
Wellcome Trust/MRC Building
Hills Road
Cambridge CB2 0XY
www.cimr.cam.ac.uk/investigators/read/index.html
tel: +44(0)1223 763234
mobile: +44(0)7712 887162
More information about the cctbxbb
mailing list