[cctbxbb] Conda

[Billy Poon]

Hi Graeme,

Here is a summary of the discussions across various email threads and
meetings.

The plan is to get conda ready in October/November. With the delay from the
Phenix release, there might be a delay for this work, but I'll have to see.

This will be a drop-in replacement for the "base" directory containing the
dependencies. The usage of conda (or the current
bootstrap.py/install_base_packages.py approach) will remain optional in the
sense that anyone can always just supply the dependencies in any other way
and CCTBX can work with those dependencies. This was discussed in the email
thread with David Waterman about this change not affecting how CCP4 would
build CCTBX. This is also important for developers that work on systems
that have more specialized environments (e.g. large HPC systems).

There should be a list of dependency versions that basically define the
API. An outdated list is at
https://github.com/cctbx/conda_build/blob/master/cctbx_dependencies/meta.yaml.
For
example, we support Python 2.7, so any version of Python 2.7.x should be
supported. This is so that CCTBX can be more friendly towards existing
conda environments. There will be a separate list (location TBD, probably
somewhere in libtbx/auto_build) that has an explicit set of versions. This
would be for creating installers and clean developer environments. An
example is attached that can be used to create an conda environment
compatible with building CCTBX ("conda create -n <environment name> --file
env.txt). The procrunner package is missing, but you can install it with
pip once you're in the environment.

On linux, the conda compilers (
https://conda.io/docs/user-guide/tasks/build-packages/compiler-tools.html)
will be used so that we avoid compatibility issues with
distribution-specific versions of compilers. On macOS, I've been testing
clang, but it may be more trouble than it's worth since developers would
need the macOS 10.9 SDK, which would need to be provided outside of conda
(licensing issues). Windows compilers will follow the versions recommended
by Python to ensure compatibility with pre-compiled dependencies (
https://wiki.python.org/moin/WindowsCompilers).

As for dispatchers, I described how to run commands in issue #85 (
https://github.com/cctbx/cctbx_project/issues/85). There is no need to be
inside a conda environment to run commands, but there may be additional
logic added to determine if the user is inside a conda environment.

Installers will probably use the conda constructor package (
https://github.com/conda/constructor). It has the benefit of uniformly
building the macOS, linux, and Windows installers. The command-line
installer would behave like the miniconda installer. I have not tested the
.pkg and .exe options for macOS and Windows installers, respectively, yet.

Some features that were brought up, but will not be part of this initial
migration is,
1) "conda install cctbx" - Some directory rearrangement will probably be
required and we would need to decide on what actually gets installed from
the cctbx_project umbrella directory.
2) pip editable-mode - Markus brought this up at the GRC. There is a conda
equivalent ("conda develop"), but since conda is not a requirement, this is
probably not generally applicable. Also, CCTBX is used in non-Python
programs as header files and linked as shared libraries, and it looks like
this mode only really works with Python modules.

The current bootstrap.py approach will be kept since it provides a simple
way for developers to build their projects. The "base" step will either be
replaced by building the dependencies via conda instead of
install_base_packages.py, or a different name can be used. Also, a step for
updating dependencies could be added. The goal is for developers to not
have to change there workflow if they are not interested in using conda
explicitly.

--
Billy K. Poon
Research Scientist, Molecular Biophysics and Integrated Bioimaging
Lawrence Berkeley National Laboratory
1 Cyclotron Road, M/S 33R0345
Berkeley, CA 94720
Tel: (510) 486-5709
Fax: (510) 486-5909
Web: https://phenix-online.org


On Fri, Sep 21, 2018 at 3:06 PM Graeme.Winter at Diamond.ac.uk <
Graeme.Winter at diamond.ac.uk> wrote:

> Hello again,
>
> Trying to keep threads from getting warped :-)
>
> We’re interested in what the plans and timescales are for the move to
> Conda for a base to cctbx-related projects, and in particular -
>
>  - what is planned?
>  - when i.e. timescales (rough)
>  - will bootstrap go away?
>
> I know from talking to Billy on Wednesday that this is back on the cards
> again now the PHENIX release is out, but I didn’t ask at the time fo the
> details… oops!
>
> Many thanks Graeme
> --
> This e-mail and any attachments may contain confidential, copyright and or
> privileged material, and are for the use of the intended addressee only. If
> you are not the intended addressee or an authorised recipient of the
> addressee please notify us of receipt by returning the e-mail and do not
> use, copy, retain, distribute or disclose the information in or attached to
> the e-mail.
> Any opinions expressed within this e-mail are those of the individual and
> not necessarily of Diamond Light Source Ltd.
> Diamond Light Source Ltd. cannot guarantee that this e-mail or any
> attachments are free from viruses and we cannot accept liability for any
> damage which you may sustain as a result of software viruses which may be
> transmitted in or with the message.
> Diamond Light Source Limited (company no. 4375679). Registered in England
> and Wales with its registered office at Diamond House, Harwell Science and
> Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://phenix-online.org/pipermail/cctbxbb/attachments/20180924/8c664b4c/attachment.htm>
-------------- next part --------------
# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: linux-64
alabaster=0.7.11=py_3
apipkg=1.5=py_0
asn1crypto=0.24.0=py27_3
atomicwrites=1.1.5=py27_0
attrs=18.1.0=py_1
babel=2.6.0=py_1
backports=1.0=py_2
backports.functools_lru_cache=1.5=py_1
backports.shutil_get_terminal_size=1.0.0=py_3
backports_abc=0.5=py_1
biopython=1.72=py27_0
blas=1.0=mkl
ca-certificates=2018.8.24=ha4d7672_0
certifi=2018.8.24=py27_1
cffi=1.11.5=py27h5e8e0c9_1
chardet=3.0.4=py27_3
cryptography=2.3.1=py27hdffb7b8_0
cryptography-vectors=2.3.1=py27_0
cycler=0.10.0=py_1
cython=0.28.5=py27hfc679d8_0
dbus=1.13.0=h3a4f0e9_0
decorator=4.3.0=py_0
docutils=0.14=py27_1
enum34=1.1.6=py27_1
execnet=1.5.0=py_0
expat=2.2.5=hfc679d8_1
fontconfig=2.12.1=4
freetype=2.7=1
funcsigs=1.0.2=py_2
functools32=3.2.3.2=py27_2
future=0.16.0=py27_2
futures=3.2.0=py27_0
gettext=0.19.8.1=0
glib=2.55.0=h464dc38_2
gst-plugins-base=1.12.5=hde13a9d_0
gstreamer=1.12.5=h61a6719_0
h5py=2.8.0=py27hb794570_1
hdf5=1.10.2=hc401514_1
icu=58.2=hfc679d8_0
idna=2.7=py27_2
imagesize=1.0.0=py_1
intel-openmp=2018.0.3=0
ipaddress=1.0.22=py_1
ipython=5.8.0=py27_0
ipython_genutils=0.2.0=py_1
jinja2=2.10=py_1
jpeg=9c=h470a237_0
junit-xml=1.8=py27_0
kiwisolver=1.0.1=py27h2d50403_2
libedit=3.1.20170329=h6b74fdf_2
libffi=3.2.1=hd88cf55_4
libgcc-ng=8.2.0=hdf63c60_1
libgfortran=3.0.0=1
libgfortran-ng=7.2.0=hdf63c60_3
libiconv=1.15=h470a237_2
libpng=1.6.35=ha92aebf_0
libstdcxx-ng=8.2.0=hdf63c60_1
libtiff=4.0.9=he6b73bb_1
libuuid=2.32.1=h470a237_0
libxcb=1.13=h470a237_2
libxml2=2.9.8=h422b904_3
libxslt=1.1.32=h88dbc4e_1
linecache2=1.0.0=py27_0
markupsafe=1.0=py27h470a237_1
matplotlib=2.0.2=py27_2
mkl=2018.0.3=1
mkl_fft=1.0.6=py27_0
mkl_random=1.0.1=py27_0
mock=2.0.0=py27_0
more-itertools=4.3.0=py27_0
ncurses=6.1=hf484d3e_0
numpy=1.13.3=py27hdbf6ddf_4
numpy-base=1.14.3=py27hdbf6ddf_0
numpydoc=0.8.0=py_1
olefile=0.45.1=py_1
openblas=0.2.20=8
openssl=1.0.2p=h470a237_0
orderedset=2.0=py27_0
packaging=17.1=py_0
pathlib2=2.3.2=py27_0
pbr=4.2.0=py_0
pcre=8.41=h470a237_2
pexpect=4.6.0=py27_0
pickleshare=0.7.4=py27_0
pillow=4.2.1=py27_0
pip=10.0.1=py27_0
pkg-config=0.29.2=h470a237_3
pluggy=0.6.0=py_0
prompt_toolkit=1.0.15=py_1
pthread-stubs=0.4=h470a237_1
ptyprocess=0.6.0=py27_0
py=1.6.0=py_0
pycparser=2.18=py_1
pygments=2.2.0=py_1
pyopenssl=18.0.0=py27_0
pyparsing=2.2.0=py_1
pyqt=5.6.0=py27h8210e8a_7
pyside2=5.6.0a1=py27h45ccf91_3
pysocks=1.6.8=py27_2
pytest=3.4.1=py27_0
pytest-forked=0.2=py27_0
pytest-xdist=1.23.0=0
python=2.7.11=0
python-dateutil=2.7.3=py_0
pytz=2018.5=py_0
qt=5.6.2=3
readline=6.2=2
reportlab=3.4.0=py27_0
requests=2.19.1=py27_1
scandir=1.9.0=py27h470a237_0
scikit-learn=0.19.1=py27hedc7406_0
scipy=1.1.0=py27hd20e5f9_0
setuptools=40.2.0=py27_0
simplegeneric=0.8.1=py_1
singledispatch=3.4.0.3=py27_0
sip=4.18.1=py27hfc679d8_0
six=1.11.0=py27_1
snowballstemmer=1.2.1=py_1
sphinx=1.7.5=py27_0
sphinxcontrib-websupport=1.0.1=py27_0
sqlite=3.24.0=h84994c4_0
subprocess32=3.5.2=py27_0
tabulate=0.8.2=py_0
tk=8.5.19=2
tornado=5.1=py27h470a237_1
traceback2=1.4.0=py27_0
traitlets=4.3.2=py27_0
typing=3.6.6=py27_0
unittest2=1.1.0=py_0
urllib3=1.23=py27_1
wcwidth=0.1.7=py_1
wheel=0.31.1=py27_0
wxpython=3.0.0.0=py27_2
xorg-libxau=1.0.8=h470a237_6
xorg-libxdmcp=1.1.2=h470a237_7
xz=5.2.4=h470a237_1
zlib=1.2.11=ha838bed_2