<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.hoenzb
{mso-style-name:hoenzb;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Dear Nigel,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Benchmarking on my desktop machine shows that for the python package the build times are between 98.4s (nproc=4), 101.3s (3), 107.1s (2) and 124.9s (1). So
far we had no problems with parallel building any other packages, which is why I would only limit the parallelism of python. We could indeed successively limit it until we no longer observe it, but that does not guarantee race-condition-free compilation for
users.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">As for dropping the default I think this makes sense. Particularly when building inside virtual machines a 4-fold parallel build can very quickly eat up all
the memory and crash. The output provides no hints how the user can fix it.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">One possible fix for both cases would be to run with n processors and then rerun failed steps with 1 processor – but this is far from a clean solution.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I think nproc defaults taking into account available cpus/memory together with a fixed nproc=1 specifically for problematic packages such as base/python is
the most sensible route.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">-Markus<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></a></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> cctbxbb-bounces@phenix-online.org [mailto:cctbxbb-bounces@phenix-online.org]
<b>On Behalf Of </b>Nigel Moriarty<br>
<b>Sent:</b> 27 May 2015 09:20<br>
<b>To:</b> cctbx mailing list<br>
<b>Subject:</b> Re: [cctbxbb] Race condition in bootstrap python installation<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Markus<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I have seen the "make" error on occasion here also. It works fine if I force a build so I agree with the race conditions analysis. However, do we have to drop to 1 when 2 or even 3 may have the same end result and not delay the builds as
much. Either way we should may the default 1 and then we can use which ever value works best at each site.<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">Cheers<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Nigel<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">---<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Nigel W. Moriarty<br>
Building 64R0246B, Physical Biosciences Division<br>
Lawrence Berkeley National Laboratory<br>
Berkeley, CA 94720-8235<br>
Phone : 510-486-5709 Email : <a href="mailto:NWMoriarty@LBL.gov">NWMoriarty@LBL.gov</a><br>
Fax : 510-486-5909 Web : <a href="http://CCI.LBL.gov" target="_blank">CCI.LBL.gov</a><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Wed, May 27, 2015 at 8:28 AM, <<a href="mailto:markus.gerstel@diamond.ac.uk" target="_blank">markus.gerstel@diamond.ac.uk</a>> wrote:<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">We have recently seen intermittent build failures with the bootstrap base installation at Diamond. These all occurred during the compilation of python and were usually of the form<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;font-family:"Courier New";color:#333333"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;font-family:"Courier New";color:#333333">: gcc -pthread -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Wl,-rpath=\$ORIGIN/../lib Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o Parser/parsetok.o
Parser/bitset.o Parser/metagrammar.o Parser/firstsets.o Parser/grammar.o Parser/pgen.o Objects/obmalloc.o Python/mysnprintf.o Python/pyctype.o Parser/tokenizer_pgen.o Parser/printgrammar.o Parser/pgenmain.o -lpthread -ldl -lutil -o Parser/pgen</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;font-family:"Courier New";color:#333333">: Parser/node.o: file not recognized: File truncated</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;font-family:"Courier New";color:#333333">: collect2: ld returned 1 exit status</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;font-family:"Courier New";color:#333333">: make[1]: *** [Parser/pgen] Error 1</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;font-family:"Courier New";color:#333333">: make[1]: Leaving directory `/scratch/jenkins_slave/workspace/dials_bootstrap_platforms/compilationtarget/native/label/dials-ws154/build_dials/base_tmp/Python-2.7.8_cci'</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;font-family:"Courier New";color:#333333">: make: *** [Include/graminit.h] Error 2</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;font-family:"Courier New";color:#333333">: make: *** Waiting for unfinished jobs....</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;font-family:"Courier New";color:#333333">Traceback (most recent call last):</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;font-family:"Courier New";color:#333333"> File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 1194, in <module></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;font-family:"Courier New";color:#333333"> installer(args=sys.argv, log=sys.stdout)</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;font-family:"Courier New";color:#333333"> File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 184, in __init__</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;font-family:"Courier New";color:#333333"> self.build_dependencies(packages=packages)</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;font-family:"Courier New";color:#333333"> File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 571, in build_dependencies</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;font-family:"Courier New";color:#333333"> getattr(self, 'build_%s'%i)()</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;font-family:"Courier New";color:#333333"> File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 622, in build_python</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;font-family:"Courier New";color:#333333"> self.call('make -j %s install'%(self.nproc), log=log, cwd=python_dir)</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;font-family:"Courier New";color:#333333"> File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 264, in call</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;font-family:"Courier New";color:#333333"> return call(args, log=log, verbose=self.verbose, **kwargs)</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;font-family:"Courier New";color:#333333"> File "/scratch/jenkins_slave/workspace/dials_bootstrap_platforms/compilationtarget/native/label/dials-ws154/build_dials/modules/cctbx_project/libtbx/auto_build/installer_utils.py", line
81, in call</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;font-family:"Courier New";color:#333333"> raise RuntimeError("Call to '%s' failed with exit code %d" % (args, rc))</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;font-family:"Courier New";color:#333333">RuntimeError: Call to 'make -j 4 install' failed with exit code 2</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Other indicated errors are
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> /usr/bin/ld: final link failed: File truncated<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">or<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">/usr/bin/ld: can not read symbols: File truncated<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I suspect that these errors are manifestations of a race condition in the python build process, which the bootstrap script now by default runs with a
<i>-j 4</i> parallel make. <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">To test this we are now using
<i>--nproc=1</i> for the base installation step. We haven’t seen these build failures since.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I suggest we set the python compilation to never run with parallel make. Any other ideas?<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">-Markus<o:p></o:p></p>
</div>
<p><span style="color:#888888"> <o:p></o:p></span></p>
<p><span style="color:#888888">-- <o:p></o:p></span></p>
<p><span style="color:#888888">This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee
please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.<br>
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
<br>
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.<br>
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom<br>
<o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
cctbxbb mailing list<br>
<a href="mailto:cctbxbb@phenix-online.org">cctbxbb@phenix-online.org</a><br>
<a href="http://phenix-online.org/mailman/listinfo/cctbxbb" target="_blank">http://phenix-online.org/mailman/listinfo/cctbxbb</a><o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p align="justify"> </p>
<p align="justify">-- </p>
<p align="justify">This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.<br />Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. <br />Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.<br />Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom<br /> </p></body>
</html>