<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">This feature isn't fully complete yet. That's why the source installers do not have an option to use conda dependencies in the install script.<div><br></div><div> There were 2 cases where the packages in the conda-forge were changed and the specification lists had to be updated to new builds. That is the cause of the HTTP error when downloading the files. There was a fix for HDF5 on April 3 (<a href="https://github.com/cctbx/cctbx_project/commit/ab679156db1a27ec5002c872d3158f7e1d54b51f#diff-e3c3f3db586f95fc929aef0fe93923da">https://github.com/cctbx/cctbx_project/commit/ab679156db1a27ec5002c872d3158f7e1d54b51f#diff-e3c3f3db586f95fc929aef0fe93923da</a>) and April 8 for Windows. And then on March 22 there were fixes for the GCC libraries (<a href="https://github.com/cctbx/cctbx_project/commit/40fe517cfab03639248700ca28073442eb34e455#diff-e3c3f3db586f95fc929aef0fe93923da">https://github.com/cctbx/cctbx_project/commit/40fe517cfab03639248700ca28073442eb34e455#diff-e3c3f3db586f95fc929aef0fe93923da</a>) for Python 2. The same changes were applied to Python 3 on May 12 (<a href="https://github.com/cctbx/cctbx_project/commit/6af789668563d016b24f9b5749cacf992cc553f4#diff-e3c3f3db586f95fc929aef0fe93923da">https://github.com/cctbx/cctbx_project/commit/6af789668563d016b24f9b5749cacf992cc553f4#diff-e3c3f3db586f95fc929aef0fe93923da</a>). Other than that, all the other packages have been stable and accessible for months.</div><div><br></div><div>The solution is to just back up the packages and have the specification lists point to those locations instead. That would preserve the version and build of each dependency without worrying about changes in the conda-forge channel. I'll do this within the next 2 weeks.</div><div><div><br clear="all"><div><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><div>--</div><div><span style="font-size:12.8000001907349px">Billy K. Poon</span><br></div></div><div>Research Scientist, Molecular Biophysics and Integrated Bioimaging</div><div>Lawrence Berkeley National Laboratory</div><div>1 Cyclotron Road, M/S 33R0345</div><div>Berkeley, CA 94720</div><div>Tel: (510) 486-5709</div><div>Fax: (510) 486-5909</div><div>Web: <a href="https://phenix-online.org" target="_blank">https://phenix-online.org</a></div></div></div></div></div></div></div></div></div></div></div></div></div></div><br></div></div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 24, 2019 at 8:38 AM <a href="mailto:Graeme.Winter@Diamond.ac.uk">Graeme.Winter@Diamond.ac.uk</a> <<a href="mailto:Graeme.Winter@diamond.ac.uk">Graeme.Winter@diamond.ac.uk</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">Wait, this one confuses me<br>
<br>
I thought the whole point of the<br>
<br>
./libtbx/auto_build/conda_envs/cctbx_py27_osx-64.txt<br>
<br>
etc. which contains all this<br>
<br>
# This file may be used to create an environment using:<br>
# $ conda create --name <env> --file <this file><br>
# platform: osx-64<br>
@EXPLICIT<br>
<a href="https://repo.anaconda.com/pkgs/main/osx-64/libcxxabi-4.0.1-hcfea43d_1.tar.bz2#b06555a43b9c0857b87b342115bea761" rel="noreferrer" target="_blank">https://repo.anaconda.com/pkgs/main/osx-64/libcxxabi-4.0.1-hcfea43d_1.tar.bz2#b06555a43b9c0857b87b342115bea761</a><br>
<a href="https://conda.anaconda.org/conda-forge/osx-64/bzip2-1.0.6-h1de35cc_1002.tar.bz2#76f47e9f5f453036b7fe198af50c7b43" rel="noreferrer" target="_blank">https://conda.anaconda.org/conda-forge/osx-64/bzip2-1.0.6-h1de35cc_1002.tar.bz2#76f47e9f5f453036b7fe198af50c7b43</a><br>
<a href="https://conda.anaconda.org/conda-forge/osx-64/ca-certificates-2018.11.29-ha4d7672_0.tar.bz2#c695cfbfd5ee1406f7d89b9b7b6f8212" rel="noreferrer" target="_blank">https://conda.anaconda.org/conda-forge/osx-64/ca-certificates-2018.11.29-ha4d7672_0.tar.bz2#c695cfbfd5ee1406f7d89b9b7b6f8212</a><br>
<a href="https://conda.anaconda.org/conda-forge/osx-64/jpeg-9c-h1de35cc_1001.tar.bz2#bcc9abfebf1cc26568e1ec4502834512" rel="noreferrer" target="_blank">https://conda.anaconda.org/conda-forge/osx-64/jpeg-9c-h1de35cc_1001.tar.bz2#bcc9abfebf1cc26568e1ec4502834512</a><br>
<a href="https://repo.anaconda.com/pkgs/main/osx-64/libcxx-4.0.1-h579ed51_0.tar.bz2#291e61684d014641092020bdb1acb096" rel="noreferrer" target="_blank">https://repo.anaconda.com/pkgs/main/osx-64/libcxx-4.0.1-h579ed51_0.tar.bz2#291e61684d014641092020bdb1acb096</a><br>
<br>
etc.<br>
<br>
was to guarantee you could reproduce the precise binaries?<br>
<br>
otherwise you could just ask for a version range for packages (as I understand most condo things do)<br>
<br>
Or have I missed something?<br>
<br>
Thanks Graeme<br>
<br>
<br>
<br>
On 24 May 2019, at 16:12, Nicholas Devenish <<a href="mailto:ndevenish@gmail.com" target="_blank">ndevenish@gmail.com</a><mailto:<a href="mailto:ndevenish@gmail.com" target="_blank">ndevenish@gmail.com</a>>> wrote:<br>
<br>
(Forwarding this manually as apparently my Diamond address isn't on cctbxbb)<br>
<br>
Hi All,<br>
<br>
On 24 May 2019, at 14:12, David Waterman - UKRI STFC <<a href="mailto:david.waterman@stfc.ac.uk" target="_blank">david.waterman@stfc.ac.uk</a><mailto:<a href="mailto:david.waterman@stfc.ac.uk" target="_blank">david.waterman@stfc.ac.uk</a>><mailto:<a href="mailto:david.waterman@stfc.ac.uk" target="_blank">david.waterman@stfc.ac.uk</a><mailto:<a href="mailto:david.waterman@stfc.ac.uk" target="_blank">david.waterman@stfc.ac.uk</a>>>> wrote:<br>
In the process, I discovered that the source builds we point to from <a href="https://dials.github.io/installation.html" rel="noreferrer" target="_blank">https://dials.github.io/installation.html</a> don't work with --use-conda because the paths to various packages are out of date.<br>
<br>
This seems to be to be a fundamental problem with the conda implementation? Let’s try to reproduce an old build from, say, 1st march: the point in cctbx_project at which dials 1.14 split off. This is <a href="https://github.com/cctbx/cctbx_project/commit/35652e06564da7ff989500a6d929bc595f69040f" rel="noreferrer" target="_blank">https://github.com/cctbx/cctbx_project/commit/35652e06564da7ff989500a6d929bc595f69040f</a>.<br>
<br>
Try reproducing this point:<br>
git clone <a href="mailto:git@github.com" target="_blank">git@github.com</a><mailto:<a href="mailto:git@github.com" target="_blank">git@github.com</a>><mailto:<a href="mailto:git@github.com" target="_blank">git@github.com</a><mailto:<a href="mailto:git@github.com" target="_blank">git@github.com</a>>>:cctbx/cctbx_project.git modules/cctbx_project<br>
( cd modules/cctbx_project/ && git checkout 35652e06564da7ff989500a6d929bc595f69040f )<br>
ln -s modules/cctbx_project/libtbx/auto_build/bootstrap.py .<br>
./bootstrap.py base --use-conda<br>
<br>
—> many "An HTTP error occurred when trying to retrieve this URL.”<br>
<br>
This means that with the current conda implementation it’s impossible to guarantee reproducibility for any build beyond the past few days, without lots and lots of manual work reconstructing the versioning out of the file list dump.<br>
<br>
Because the version constraints aren’t kept with the repository, this means that a source distribution is not nearly enough to recreate any given release.<br>
<br>
This seems like a problem?<br>
<br>
Nick<br>
<br>
_______________________________________________<br>
cctbxbb mailing list<br>
<a href="mailto:cctbxbb@phenix-online.org" target="_blank">cctbxbb@phenix-online.org</a><mailto:<a href="mailto:cctbxbb@phenix-online.org" target="_blank">cctbxbb@phenix-online.org</a>><br>
<a href="http://phenix-online.org/mailman/listinfo/cctbxbb" rel="noreferrer" target="_blank">http://phenix-online.org/mailman/listinfo/cctbxbb</a><br>
<br>
<br>
-- <br>
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.<br>
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. <br>
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.<br>
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom<br>
<br>
_______________________________________________<br>
cctbxbb mailing list<br>
<a href="mailto:cctbxbb@phenix-online.org" target="_blank">cctbxbb@phenix-online.org</a><br>
<a href="http://phenix-online.org/mailman/listinfo/cctbxbb" rel="noreferrer" target="_blank">http://phenix-online.org/mailman/listinfo/cctbxbb</a><br>
</blockquote></div>