<div dir="ltr">Hi Graeme,<div><br></div><div>if all you want to do is print the stacktrace out to stdout/stderr, then the following lines should do this:</div><div><br></div><div>from libtbx.scheduling import stacktrace</div><div>stacktrace.enable()</div><div>parallel_map(...)</div><div><br></div><div>This will install a custom excepthook and print the propagated stacktrace onto the console when the program crashes.</div><div><br></div><div>If you want to accumulate potentially multiple traceback, you need to do what Rob suggests (although I would not actually copy the code, but instead create a wrapping function that fails silently and stores the tracebacks) - it is not immediately clear to me whether this has any advantages over the previous approach, but let me know if you need some help for such a generic solution.</div><div><br></div><div>BW, Gabor</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 3, 2018 at 1:46 PM, Dr. Robert Oeffner <span dir="ltr"><<a href="mailto:rdo20@cam.ac.uk" target="_blank">rdo20@cam.ac.uk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Graeme,<br>
<br>
Had a look at the code again in parallel_map(). I think it may be possible to adapt it to retain stack traces of individual qsub jobs.<br>
In libtbx/easy_mp.py compare the lines 627 with 718. Both are doing<br>
result = res()<br>
But the latter is guarded by a try except block. Any exception is the result of a child process dying and it's stack trace is added as the third member of the parmres tuple which is passed on to the user.<br>
<br>
I think similar could be done with parallel_map(). So I suggest fashioning a parallel_map2() function which is a copy of parallel_map() but with the added exception handler around the result=res() statement.<br>
<br>
As I have no access to a qsub cluster I can't test whether this would work.<br>
<br>
Regards,<br>
<br>
Rob<div class="HOEnZb"><div class="h5"><br>
<br>
<br>
On 03/04/2018 13:23, <a href="mailto:Graeme.Winter@Diamond.ac.uk" target="_blank">Graeme.Winter@Diamond.ac.uk</a> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
HI Rob<br>
<br>
I think this is true … sometimes<br>
<br>
It sets up the qsub every time, but does not always use it - at least it works on my MacBook with no qsub ;-)<br>
<br>
That said, the question remains why exception reports are bad for parallel map… we *are* using preserve_exception_message…<br>
<br>
Cheers Graeme<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On 3 Apr 2018, at 13:20, Dr. Robert Oeffner <<a href="mailto:rdo20@cam.ac.uk" target="_blank">rdo20@cam.ac.uk</a>> wrote:<br>
<br>
Hi Graeme,<br>
<br>
Just had a look at the code in dials/util/mp.py. It seems that you are using parallel_map() on a cluster using qsub. Unfortunately multi_core_run() is not designed for that. It only runs on a single multi core CPU PC.<br>
<br>
Sorry,<br>
<br>
Rob<br>
<br>
<br>
On 03/04/2018 12:44, <a href="mailto:Graeme.Winter@Diamond.ac.uk" target="_blank">Graeme.Winter@Diamond.ac.uk</a> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Thanks Rob, I could not dig out the thread (and the mail list thing does not have search that I could find)<br>
I’ll talk to the crew about swapping this out for dials.* - though is possibly quite a big change?<br>
Cheers Graeme<br>
On 3 Apr 2018, at 12:26, Dr. Robert Oeffner <<a href="mailto:rdo20@cam.ac.uk" target="_blank">rdo20@cam.ac.uk</a><mailto:<a href="mailto:rdo20@cam.ac.uk" target="_blank">rdo20@<wbr>cam.ac.uk</a>>> wrote:<br>
Hi Graeme,<br>
I recall we've been here before,<br>
<a href="http://phenix-online.org/pipermail/cctbxbb/2017-December/001807.html" rel="noreferrer" target="_blank">http://phenix-online.org/piper<wbr>mail/cctbxbb/2017-December/<wbr>001807.html</a><br>
I believe the solution is to use easy_mp.multi_core_run() instead of easy_mp.parallel_map(). The first function preserves stack traces of individual process, unlike easy_mp.parallel_map().<br>
Regards,<br>
Rob<br>
On 03/04/2018 07:16, <a href="mailto:Graeme.Winter@Diamond.ac.uk" target="_blank">Graeme.Winter@Diamond.ac.uk</a><ma<wbr>ilto:<a href="mailto:Graeme.Winter@Diamond.ac.uk" target="_blank">Graeme.Winter@Diamond.ac.<wbr>uk</a>> wrote:<br>
Folks,<br>
Following up on user reports again of errors within easy_mp - all that gets logged is “something went wrong” i.e.<br>
Using multiprocessing with 10 parallel job(s)<br>
Traceback (most recent call last):<br>
File "/home/user/bin/dials-installe<wbr>r/build/../modules/dials/<wbr>command_line/integrate.py", line 613, in <module><br>
halraiser(e)<br>
File "/home/user/bin/dials-installe<wbr>r/build/../modules/dials/<wbr>command_line/integrate.py", line 611, in <module><br>
script.run()<br>
File "/home/user/bin/dials-installe<wbr>r/build/../modules/dials/<wbr>command_line/integrate.py", line 341, in run<br>
reflections = integrator.integrate()<br>
File "/home/user/bin/dials-installe<wbr>r/modules/dials/algorithms/<wbr>integration/integrator.py", line 1214, in integrate<br>
self.reflections, _, time_info = processor.process()<br>
File "/home/user/bin/dials-installe<wbr>r/modules/dials/algorithms/<wbr>integration/processor.py", line 271, in process<br>
preserve_exception_message = True)<br>
File "/home/user/bin/dials-installe<wbr>r/modules/dials/util/mp.py", line 171, in multi_node_parallel_map<br>
preserve_exception_message = preserve_exception_message)<br>
File "/home/user/bin/dials-installe<wbr>r/modules/dials/util/mp.py", line 53, in parallel_map<br>
preserve_exception_message = preserve_exception_message)<br>
File "/home/user/bin/dials-installe<wbr>r/modules/cctbx_project/<wbr>libtbx/easy_mp.py", line 627, in parallel_map<br>
result = res()<br>
File "/home/user/bin/dials-installe<wbr>r/modules/cctbx_project/<wbr>libtbx/scheduling/result.py", line 119, in __call__<br>
self.traceback( exception = self.exception() )<br>
File "/home/user/bin/dials-installe<wbr>r/modules/cctbx_project/<wbr>libtbx/scheduling/stacktrace.<wbr>py", line 115, in __call__<br>
self.raise_handler( exception = exception )<br>
File "/home/user/bin/dials-installe<wbr>r/modules/cctbx_project/<wbr>libtbx/scheduling/mainthread.<wbr>py", line 100, in poll<br>
value = target( *args, **kwargs )<br>
File "/home/user/bin/dials-installe<wbr>r/modules/dials/util/mp.py", line 91, in __call__<br>
preserve_exception_message = self.preserve_exception_messag<wbr>e)<br>
File "/home/user/bin/dials-installe<wbr>r/modules/cctbx_project/<wbr>libtbx/easy_mp.py", line 627, in parallel_map<br>
result = res()<br>
File "/home/user/bin/dials-installe<wbr>r/modules/cctbx_project/<wbr>libtbx/scheduling/result.py", line 119, in __call__<br>
self.traceback( exception = self.exception() )<br>
File "/home/user/bin/dials-installe<wbr>r/modules/cctbx_project/<wbr>libtbx/scheduling/stacktrace.<wbr>py", line 86, in __call__<br>
raise exception<br>
RuntimeError: Please report this error to <a href="mailto:dials-support@lists.sourceforge.net" target="_blank">dials-support@lists.sourceforg<wbr>e.net</a><mailto:<a href="mailto:dials-support@lists.sourceforge.net" target="_blank">dials-support@<wbr>lists.sourceforge.net</a>>: exit code = -9<br>
I forget why it was decided that keeping the proper stack trace was a bad thing, but could this be revisited? It would greatly help to see it in the output of the program (if as is the case here I do not have the user data)<br>
My email-fu is not strong enough to dig out the previous conversation<br>
Cheers Graeme<br>
--<br>
Robert Oeffner, Ph.D.<br>
Research Associate, The Read Group<br>
Department of Haematology,<br>
Cambridge Institute for Medical Research<br>
University of Cambridge<br>
Cambridge Biomedical Campus<br>
Wellcome Trust/MRC Building<br>
Hills Road<br>
Cambridge CB2 0XY<br>
<a href="http://www.cimr.cam.ac.uk/investigators/read/index.html" rel="noreferrer" target="_blank">www.cimr.cam.ac.uk/investigato<wbr>rs/read/index.html</a><<a href="http://www.cimr.cam.ac.uk/investigators/read/index.html" rel="noreferrer" target="_blank">http://www.<wbr>cimr.cam.ac.uk/investigators/<wbr>read/index.html</a>><br>
tel: <a href="tel:%2B44%280%291223%20763234" value="+441223763234" target="_blank">+44(0)1223 763234</a><br>
</blockquote>
<br>
<br>
-- <br>
Robert Oeffner, Ph.D.<br>
Research Associate, The Read Group<br>
Department of Haematology,<br>
Cambridge Institute for Medical Research<br>
University of Cambridge<br>
Cambridge Biomedical Campus<br>
Wellcome Trust/MRC Building<br>
Hills Road<br>
Cambridge CB2 0XY<br>
<br>
<a href="http://www.cimr.cam.ac.uk/investigators/read/index.html" rel="noreferrer" target="_blank">www.cimr.cam.ac.uk/investigato<wbr>rs/read/index.html</a><br>
tel: <a href="tel:%2B44%280%291223%20763234" value="+441223763234" target="_blank">+44(0)1223 763234</a><br>
</blockquote>
<br>
<br>
</blockquote>
<br>
<br>
-- <br>
Robert Oeffner, Ph.D.<br>
Research Associate, The Read Group<br>
Department of Haematology,<br>
Cambridge Institute for Medical Research<br>
University of Cambridge<br>
Cambridge Biomedical Campus<br>
Wellcome Trust/MRC Building<br>
Hills Road<br>
Cambridge CB2 0XY<br>
<br>
<a href="http://www.cimr.cam.ac.uk/investigators/read/index.html" rel="noreferrer" target="_blank">www.cimr.cam.ac.uk/investigato<wbr>rs/read/index.html</a><br>
tel: <a href="tel:%2B44%280%291223%20763234" value="+441223763234" target="_blank">+44(0)1223 763234</a><br></div></div><div class="HOEnZb"><div class="h5">
______________________________<wbr>_________________<br>
cctbxbb mailing list<br>
<a href="mailto:cctbxbb@phenix-online.org" target="_blank">cctbxbb@phenix-online.org</a><br>
<a href="http://phenix-online.org/mailman/listinfo/cctbxbb" rel="noreferrer" target="_blank">http://phenix-online.org/mailm<wbr>an/listinfo/cctbxbb</a><br>
</div></div></blockquote></div><br></div>