[cctbxbb] Conda

Mon Sep 24 08:22:11 PDT 2018

Thanks Billy, particularly for the detail in here.

Appreciated.

Cheers Graeme

On 24 Sep 2018, at 16:08, Billy Poon <BKPoon at lbl.gov<mailto:BKPoon at lbl.gov>> wrote:

Hi Graeme,

Here is a summary of the discussions across various email threads and meetings.

The plan is to get conda ready in October/November. With the delay from the Phenix release, there might be a delay for this work, but I'll have to see.

This will be a drop-in replacement for the "base" directory containing the dependencies. The usage of conda (or the current bootstrap.py/install_base_packages.py<http://bootstrap.py/install_base_packages.py> approach) will remain optional in the sense that anyone can always just supply the dependencies in any other way and CCTBX can work with those dependencies. This was discussed in the email thread with David Waterman about this change not affecting how CCP4 would build CCTBX. This is also important for developers that work on systems that have more specialized environments (e.g. large HPC systems).

There should be a list of dependency versions that basically define the API. An outdated list is at https://github.com/cctbx/conda_build/blob/master/cctbx_dependencies/meta.yaml. For example, we support Python 2.7, so any version of Python 2.7.x should be supported. This is so that CCTBX can be more friendly towards existing conda environments. There will be a separate list (location TBD, probably somewhere in libtbx/auto_build) that has an explicit set of versions. This would be for creating installers and clean developer environments. An example is attached that can be used to create an conda environment compatible with building CCTBX ("conda create -n <environment name> --file env.txt). The procrunner package is missing, but you can install it with pip once you're in the environment.

On linux, the conda compilers (https://conda.io/docs/user-guide/tasks/build-packages/compiler-tools.html) will be used so that we avoid compatibility issues with distribution-specific versions of compilers. On macOS, I've been testing clang, but it may be more trouble than it's worth since developers would need the macOS 10.9 SDK, which would need to be provided outside of conda (licensing issues). Windows compilers will follow the versions recommended by Python to ensure compatibility with pre-compiled dependencies (https://wiki.python.org/moin/WindowsCompilers).

As for dispatchers, I described how to run commands in issue #85 (https://github.com/cctbx/cctbx_project/issues/85). There is no need to be inside a conda environment to run commands, but there may be additional logic added to determine if the user is inside a conda environment.

Installers will probably use the conda constructor package (https://github.com/conda/constructor). It has the benefit of uniformly building the macOS, linux, and Windows installers. The command-line installer would behave like the miniconda installer. I have not tested the .pkg and .exe options for macOS and Windows installers, respectively, yet.

Some features that were brought up, but will not be part of this initial migration is,
1) "conda install cctbx" - Some directory rearrangement will probably be required and we would need to decide on what actually gets installed from the cctbx_project umbrella directory.
2) pip editable-mode - Markus brought this up at the GRC. There is a conda equivalent ("conda develop"), but since conda is not a requirement, this is probably not generally applicable. Also, CCTBX is used in non-Python programs as header files and linked as shared libraries, and it looks like this mode only really works with Python modules.

The current bootstrap.py approach will be kept since it provides a simple way for developers to build their projects. The "base" step will either be replaced by building the dependencies via conda instead of install_base_packages.py, or a different name can be used. Also, a step for updating dependencies could be added. The goal is for developers to not have to change there workflow if they are not interested in using conda explicitly.

--
Billy K. Poon
Research Scientist, Molecular Biophysics and Integrated Bioimaging
Lawrence Berkeley National Laboratory
1 Cyclotron Road, M/S 33R0345
Berkeley, CA 94720
Tel: (510) 486-5709
Fax: (510) 486-5909
Web: https://phenix-online.org<https://phenix-online.org/>

On Fri, Sep 21, 2018 at 3:06 PM Graeme.Winter at Diamond.ac.uk<mailto:Graeme.Winter at Diamond.ac.uk> <Graeme.Winter at diamond.ac.uk<mailto:Graeme.Winter at diamond.ac.uk>> wrote:
Hello again,

Trying to keep threads from getting warped :-)

We’re interested in what the plans and timescales are for the move to Conda for a base to cctbx-related projects, and in particular -

 - what is planned?
 - when i.e. timescales (rough)
 - will bootstrap go away?

I know from talking to Billy on Wednesday that this is back on the cards again now the PHENIX release is out, but I didn’t ask at the time fo the details… oops!

Many thanks Graeme
--
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

_______________________________________________
cctbxbb mailing list
cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
http://phenix-online.org/mailman/listinfo/cctbxbb
<env.txt>_______________________________________________
cctbxbb mailing list
cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
http://phenix-online.org/mailman/listinfo/cctbxbb