[cctbxbb] Fwd: Conda/DIALS 1.14.5 source build with --use-conda does not work

[Billy Poon]

This feature isn't fully complete yet. That's why the source installers do
not have an option to use conda dependencies in the install script.

 There were 2 cases where the packages in the conda-forge were changed and
the specification lists had to be updated to new builds. That is the cause
of the HTTP error when downloading the files. There was a fix for HDF5 on
April 3 (
https://github.com/cctbx/cctbx_project/commit/ab679156db1a27ec5002c872d3158f7e1d54b51f#diff-e3c3f3db586f95fc929aef0fe93923da)
and April 8 for Windows. And then on March 22 there were fixes for the GCC
libraries (
https://github.com/cctbx/cctbx_project/commit/40fe517cfab03639248700ca28073442eb34e455#diff-e3c3f3db586f95fc929aef0fe93923da)
for Python 2. The same changes were applied to Python 3 on May 12 (
https://github.com/cctbx/cctbx_project/commit/6af789668563d016b24f9b5749cacf992cc553f4#diff-e3c3f3db586f95fc929aef0fe93923da).
Other than that, all the other packages have been stable and accessible for
months.

The solution is to just back up the packages and have the specification
lists point to those locations instead. That would preserve the version and
build of each dependency without worrying about changes in the conda-forge
channel. I'll do this within the next 2 weeks.

--
Billy K. Poon
Research Scientist, Molecular Biophysics and Integrated Bioimaging
Lawrence Berkeley National Laboratory
1 Cyclotron Road, M/S 33R0345
Berkeley, CA 94720
Tel: (510) 486-5709
Fax: (510) 486-5909
Web: https://phenix-online.org


On Fri, May 24, 2019 at 8:38 AM Graeme.Winter at Diamond.ac.uk <
Graeme.Winter at diamond.ac.uk> wrote:

> Wait, this one confuses me
>
> I thought the whole point of the
>
> ./libtbx/auto_build/conda_envs/cctbx_py27_osx-64.txt
>
> etc. which contains all this
>
> # This file may be used to create an environment using:
> # $ conda create --name <env> --file <this file>
> # platform: osx-64
> @EXPLICIT
>
> https://repo.anaconda.com/pkgs/main/osx-64/libcxxabi-4.0.1-hcfea43d_1.tar.bz2#b06555a43b9c0857b87b342115bea761
>
> https://conda.anaconda.org/conda-forge/osx-64/bzip2-1.0.6-h1de35cc_1002.tar.bz2#76f47e9f5f453036b7fe198af50c7b43
>
> https://conda.anaconda.org/conda-forge/osx-64/ca-certificates-2018.11.29-ha4d7672_0.tar.bz2#c695cfbfd5ee1406f7d89b9b7b6f8212
>
> https://conda.anaconda.org/conda-forge/osx-64/jpeg-9c-h1de35cc_1001.tar.bz2#bcc9abfebf1cc26568e1ec4502834512
>
> https://repo.anaconda.com/pkgs/main/osx-64/libcxx-4.0.1-h579ed51_0.tar.bz2#291e61684d014641092020bdb1acb096
>
> etc.
>
> was to guarantee you could reproduce the precise binaries?
>
> otherwise you could just ask for a version range for packages (as I
> understand most condo things do)
>
> Or have I missed something?
>
> Thanks Graeme
>
>
>
> On 24 May 2019, at 16:12, Nicholas Devenish <ndevenish at gmail.com<mailto:
> ndevenish at gmail.com>> wrote:
>
> (Forwarding this manually as apparently my Diamond address isn't on
> cctbxbb)
>
> Hi All,
>
> On 24 May 2019, at 14:12, David Waterman - UKRI STFC <
> david.waterman at stfc.ac.uk<mailto:david.waterman at stfc.ac.uk><mailto:
> david.waterman at stfc.ac.uk<mailto:david.waterman at stfc.ac.uk>>> wrote:
> In the process, I discovered that the source builds we point to from
> https://dials.github.io/installation.html don't work with --use-conda
> because the paths to various packages are out of date.
>
> This seems to be to be a fundamental problem with the conda
> implementation? Let’s try to reproduce an old build from, say, 1st march:
> the point in cctbx_project at which dials 1.14 split off. This is
> https://github.com/cctbx/cctbx_project/commit/35652e06564da7ff989500a6d929bc595f69040f
> .
>
> Try reproducing this point:
> git clone git at github.com<mailto:git at github.com><mailto:git at github.com
> <mailto:git at github.com>>:cctbx/cctbx_project.git modules/cctbx_project
> ( cd modules/cctbx_project/ && git checkout
> 35652e06564da7ff989500a6d929bc595f69040f )
> ln -s modules/cctbx_project/libtbx/auto_build/bootstrap.py .
> ./bootstrap.py base --use-conda
>
> —> many "An HTTP error occurred when trying to retrieve this URL.”
>
> This means that with the current conda implementation it’s impossible to
> guarantee reproducibility for any build beyond the past few days, without
> lots and lots of manual work reconstructing the versioning out of the file
> list dump.
>
> Because the version constraints aren’t kept with the repository, this
> means that a source distribution is not nearly enough to recreate any given
> release.
>
> This seems like a problem?
>
> Nick
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
> --
> This e-mail and any attachments may contain confidential, copyright and or
> privileged material, and are for the use of the intended addressee only. If
> you are not the intended addressee or an authorised recipient of the
> addressee please notify us of receipt by returning the e-mail and do not
> use, copy, retain, distribute or disclose the information in or attached to
> the e-mail.
> Any opinions expressed within this e-mail are those of the individual and
> not necessarily of Diamond Light Source Ltd.
> Diamond Light Source Ltd. cannot guarantee that this e-mail or any
> attachments are free from viruses and we cannot accept liability for any
> damage which you may sustain as a result of software viruses which may be
> transmitted in or with the message.
> Diamond Light Source Limited (company no. 4375679). Registered in England
> and Wales with its registered office at Diamond House, Harwell Science and
> Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://phenix-online.org/pipermail/cctbxbb/attachments/20190527/3bbdf7bb/attachment.htm>