[cctbxbb] some thoughts on cctbx and pip

Billy Poon BKPoon at lbl.gov
Mon Dec 16 23:34:43 PST 2019


Hi Gergely,

Let me build the test package tomorrow. All the gritty details for building
with conda is being finalized and the official documentation will be
updated to describe the steps. It would be too confusing to keep changing
the documentation as the process evolves. You do have the general process,
though, which is summarized as follows.

1) Installing dependencies. The cctbx_dependencies metapackage was an
initial approach for managing the CCTBX dependencies, but after contacting
the conda-forge folks, they recommended using the --only-deps flag. So when
the CCTBX conda package is available, you'll be able to get a set of
dependencies with,

conda install -c conda-forge --only-deps cctbx

By default, the bootstrap.py file will automatically install a set of
dependencies in the "conda_base" directory (and a conda installation if one
is not found). It just uses standard conda environment files located in
libtbx/auto_build/conda_envs, so you do not need to install
cctbx_dependencies as a separate step. The environment files avoid channel
issues by explicitly defining the channel to pull the packages from and the
cctbx channel just stores copies of packages from conda-forge. There were
issues earlier where the conda-forge packages would sometimes be moved to a
different label. The --use-conda flag also accepts a path to $CONDA_PREFIX
if you want to use a specific environment for testing.

2) Building. The bootstrap.py file handles that with SCons.

3) Running. After building, there should be a setpaths.sh (and .csh) file
that adds build/bin to your path. The build/bin directory has our
dispatchers, which are just scripts that set up the environment variables
for you. This prevents other programs from loading our libraries, whose
versions may conflict. You should see that there is a "python" dispatcher,
which is a convenience for developers. Otherwise, you can use
libtbx.python, which will be able to import CCTBX modules.

The running part is where the conda package for CCTBX will be different
than this build. Since our Python files and extensions modules will be in
the "site-packages" directory for Python, the PYTHONPATH variable will not
be needed (and conda suggests that that variable not be set). The other
CCTBX libraries will be in $CONDA_PREFIX/lib, so LD_LIBRARY_PATH is not
needed. In an active environment, PATH will already be modified. And then
our LIBTBX_BUILD directory can be set to sys.prefix.

So with the conda package, you would only need to activate your conda
environment and CCTBX should integrate with other conda packages. What is
the conflict with hdf5? That's something that should be fixed. Thanks!

--
Billy K. Poon
Research Scientist, Molecular Biophysics and Integrated Bioimaging
Lawrence Berkeley National Laboratory
1 Cyclotron Road, M/S 33R0345
Berkeley, CA 94720
Tel: (510) 486-5709
Fax: (510) 486-5909
Web: https://phenix-online.org


On Mon, Dec 16, 2019 at 2:53 AM Gergely Katona <gkatona at gmail.com> wrote:

> Dear Billy,
>
> Thank you for this update and for your efforts! I found a solution and
> indeed most things already work in anaconda3. The steps I took (even
> if these instructions will have short expiry date):
>
> Modified .condarc with:
> channels:
>   - cctbx
>   - conda-forge
>   - defaults
>
> run
> conda conda install cctbx_dependencies python=3.6
>
> I expect this will work without python=3.6 in the near future.
>
> Then compiling cctbx with anaconda3 python went without any problem
> when using these flags:
> python bootstrap.py --use-conda --python3 --nproc=12
>
> Then I probably did the unorthodox thing and sourced these directories:
>
> setenv LIBTBX_BUILD /home/gergely/cctbx/build
> setenv PATH ${PATH}:/home/gergely/cctbx/build/bin
> setenv PYTHONPATH
>
> /home/gergely/cctbx/modules/cctbx_project:/home/gergely/cctbx/modules:/home/gergely/cctbx/modules/cctbx_project/boost_adaptbx:/home/gergely/cctbx/build/lib:/home/gergely/cctbx/conda_base/lib/python3.6/site-packages:$PYTHONPATH
> setenv LD_LIBRARY_PATH
>
> /home/gergely/cctbx/conda_base/lib:/home/gergely/cctbx/build/lib:$LD_LIBRARY_PATH
>
> There is probably a better way to put this into conda environment.
> With these steps I could run one of my scripts depending on cctbx. The
> only problem I found is that hdf5 library had conflict with another
> package in conda, but I does not find this as a showstopper.
>
> Best wishes,
>
> Gergely
>
>
>
>
>
>
> On Fri, Dec 13, 2019 at 7:23 PM Billy Poon <BKPoon at lbl.gov> wrote:
> >
> > Hi Gergely,
> >
> > It's still a work in progress. I'm sorting out some Windows issues right
> now. I can probably build a test package on a separate channel for people
> that want to test it (let's say next week?). I'll provide instructions, but
> basically, the test conda package will be in its own separate channel and
> the dependencies will be pulled from the conda-forge channel. I want most
> things to be working correctly on Python 2.7, 3.6, 3.7, and 3.8 on all 3
> platforms.
> >
> > Thanks!
> >
> > --
> > Billy K. Poon
> > Research Scientist, Molecular Biophysics and Integrated Bioimaging
> > Lawrence Berkeley National Laboratory
> > 1 Cyclotron Road, M/S 33R0345
> > Berkeley, CA 94720
> > Tel: (510) 486-5709
> > Fax: (510) 486-5909
> > Web: https://phenix-online.org
> >
> >
> > On Fri, Dec 13, 2019 at 5:44 AM Gergely Katona <gkatona at gmail.com>
> wrote:
> >>
> >> Dear Billy,
> >>
> >> This sounds very promising and exciting. I am not sure if cctbx is
> >> already functional as a conda package in anaconda3 (Linux) or this is
> >> still work in progress. My technical expertise does not allow me to
> >> tell the difference. What I tried:
> >>
> >> Fresh install of anaconda3. Adding - cctbx and - conda-forge to
> >> .condarc . Installing pyside2 with conda. Running conda install
> >> conda_dependencies .  I get a lot package version conflict, and I
> >> cannot import cctbx or iotbx to anaconda python. Am I following the
> >> right instructions? Or it is too early to expect that cctbx works when
> >> installed through conda?
> >>
> >> Best wishes,
> >>
> >> Gergely
> >>
> >> On Wed, Nov 27, 2019 at 3:56 PM Billy Poon <BKPoon at lbl.gov> wrote:
> >> >
> >> > Hi all,
> >> >
> >> > For a brief update, I have submitted a recipe for cctbxlite to
> conda-forge (https://github.com/conda-forge/staged-recipes/pull/10021)
> and support for Python 3.7 and 3.8 is being added (
> https://github.com/cctbx/cctbx_project/pull/409). With some fixes for
> Windows (https://github.com/cctbx/cctbx_project/pull/416), all platforms
> (macOS, linux, and Windows) can build for Python 2.7, 3.6, 3.7, and 3.8.
> Some additional changes will be needed to get Windows to work with Python 3
> and for tests to pass with Boost 1.70.0. That will enable the conda-forge
> recipe to build for all platforms and for all supported versions of Python.
> >> >
> >> > Currently, the conda-forge recipe will install into the conda python
> and cctbx imports can be done without sourcing the environment scripts. It
> looks like a lot of the environment variables being set in the dispatchers
> can be removed since the Python files and C++ extensions are in the right
> places. I'll update the libtbx_env file so that commands that load the
> environment can work correctly.
> >> >
> >> > --
> >> > Billy K. Poon
> >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging
> >> > Lawrence Berkeley National Laboratory
> >> > 1 Cyclotron Road, M/S 33R0345
> >> > Berkeley, CA 94720
> >> > Tel: (510) 486-5709
> >> > Fax: (510) 486-5909
> >> > Web: https://phenix-online.org
> >> >
> >> >
> >> > On Sun, Aug 25, 2019 at 2:33 PM Tristan Croll <tic20 at cam.ac.uk>
> wrote:
> >> >>
> >> >> Hi Luc,
> >> >>
> >> >> That sounds promising. From there, I’d need to work out how to make
> a fully-packaged installer (basically a modified wheel file) for the
> ChimeraX ToolShed - the aim is for the end user to not have to worry about
> any of this. That adds a couple of complications - e.g. $LIBTBX_BUILD would
> need to be set dynamically before first import - but doesn’t seem
> insurmountable.
> >> >>
> >> >> Thanks,
> >> >>
> >> >> Tristan
> >> >>
> >> >>
> >> >>
> >> >> Tristan Croll
> >> >> Research Fellow
> >> >> Cambridge Institute for Medical Research
> >> >> University of Cambridge CB2 0XY
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> > On 25 Aug 2019, at 18:31, Luc Bourhis <luc_j_bourhis at mac.com>
> wrote:
> >> >> >
> >> >> > Hi Tristan,
> >> >> >
> >> >> > cctbx could be built to use your ChimeraX python, now that cctbx
> is moving to Python 3. The option —with-python is there for that with the
> bootstrap script. The specific environment setup boil down to setting two
> environment variable LIBTBX_BUILD and either LD_LIBRARY_PATH on Linux, PATH
> on Win32, or DYLIB_LIBRARY_PATH on MacOS. If you work within a framework
> such as ChimeraX, that should not be difficult to ensure those two
> variables are set.
> >> >> >
> >> >> > Best wishes,
> >> >> >
> >> >> > Luc
> >> >> >
> >> >> >
> >> >> >> On 23 Aug 2019, at 19:02, Tristan Croll <tic20 at cam.ac.uk> wrote:
> >> >> >>
> >> >> >> To add my two cents on this: probably the second-most common
> question I've had about ISOLDE's implementation is, "why didn't you use
> CCTBX?". The honest answer to that is, "I didn't know how."
> >> >> >>
> >> >> >> Still don't, really - although the current developments are
> rather promising. The problem I've faced is that CCTBX was designed as its
> own self-contained Python (2.7, until very recently) environment, with its
> own interpreter and a lot of very specific environment setup. Meanwhile I'm
> developing ISOLDE in ChimeraX, which is *also* its own self-contained
> Python (3.7) environment. To plug one into the other in that form... well,
> I don't think I'm a good enough programmer to really know where to start.
> >> >> >>
> >> >> >> The move to Conda and a more modular CCTBX architecture should
> make a lot more possible in that direction. Pip would be even better for me
> personally (ChimeraX can install directly from the PyPI, but doesn't
> interact with Conda) - but I understand pretty well the substantial
> challenge that would amount to (not least being that the PyPI imposes a
> limit - around 100MB from memory? - on the size of an individual package).
> >> >> >>
> >> >> >> Best regards,
> >> >> >>
> >> >> >> Tristan
> >> >> >>
> >> >> >>> On 2019-08-23 09:28, Luc Bourhis wrote:
> >> >> >>> Hi Graeme,
> >> >> >>> Yes, I know. But “black" is a program doing a very particular
> task
> >> >> >>> (code formatting from the top of my head). Requiring to use a
> wrapper
> >> >> >>> for python itself is another level. But ok, I think I am
> mellowing to
> >> >> >>> the idea after all! Talking with people around me, and
> extrapolating,
> >> >> >>> I would bet that, right now, a great majority of people
> interested by
> >> >> >>> cctbx in pip have already used the cctbx, so they know about the
> >> >> >>> Python wrapper, and they would not be too sanguine about that. My
> >> >> >>> concern is for the future, when pip will be the first time some
> people
> >> >> >>> use cctbx. Big fat warning notices on PyPI page and a better
> error
> >> >> >>> message when cctbx fails because LIBTBX_BUILD is not set would be
> >> >> >>> needed but that could be all right.
> >> >> >>> If we do a pip installer, we should aim at a minimal install:
> cctbx,
> >> >> >>> iotbx and their dependencies, and that’s it.
> >> >> >>> Best wishes,
> >> >> >>> Luc
> >> >> >>>> On 23 Aug 2019, at 07:17, Graeme.Winter at Diamond.ac.uk <
> Graeme.Winter at diamond.ac.uk> wrote:
> >> >> >>>> Without discussing the merits of this or whether we _choose_ to
> make the move to supporting PIP, I am certain it would be _possible_ - many
> other packages make dispatcher scripts when you pip install them e.g.
> >> >> >>>> Silver-Surfer rescale_f2 :) $ which black; cat $(which black)
> >> >> >>>> /Library/Frameworks/Python.framework/Versions/3.6/bin/black
> >> >> >>>>
> #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6
> >> >> >>>> # -*- coding: utf-8 -*-
> >> >> >>>> import re
> >> >> >>>> import sys
> >> >> >>>> from black import main
> >> >> >>>> if __name__ == '__main__':
> >> >> >>>>  sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '',
> sys.argv[0])
> >> >> >>>>  sys.exit(main())
> >> >> >>>> So we _could_ work around the absence of LIBTBX_BUILD etc. in
> the system. Whether or not we elect to do the work is a different question,
> and it seems clear that here are very mixed opinions on this.
> >> >> >>>> Best wishes Graeme
> >> >> >>>> On 23 Aug 2019, at 01:21, Luc Bourhis <luc_j_bourhis at mac.com
> <mailto:luc_j_bourhis at mac.com>> wrote:
> >> >> >>>> Hi,
> >> >> >>>> Even if we managed to ship our the boost dynamic libraries with
> pip, it would still not be pip-like, as we would still need our python
> wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal pip packages work
> with the standard python exe. LD_LIBRARY_PATH, we could get around that by
> changing the way we compile, using -Wl,-R, which is the runtime equivalent
> of build time -L. That’s a significant change that would need to be tested.
> But there is no way around setting LIBTBX_BUILD right now. Leaving that to
> the user is horrible. Perhaps there is a way to hack libtbx/env_config.py
> so that we can hardwire LIBTBX_BUILD in there when pip installs?
> >> >> >>>> Best wishes,
> >> >> >>>> Luc
> >> >> >>>> On 16 Aug 2019, at 22:47, Luc Bourhis <luc_j_bourhis at mac.com
> <mailto:luc_j_bourhis at mac.com>> wrote:
> >> >> >>>> Hi,
> >> >> >>>> I did look into that many years ago, and even toyed with
> building a pip installer. What stopped me is the exact conclusion you
> reached too: the user would not have the pip experience he expects. You are
> right that it is a lot of effort but is it worth it? Considering that
> remark, I don’t think so. Now, Conda was created specifically to go beyond
> pip pure-python-only support. Since cctbx has garnered support for Conda,
> the best avenue imho is to go the extra length to have a package on
> Anaconda.org<http://anaconda.org/>, and then to advertise it hard to
> every potential user out there.
> >> >> >>>> Best wishes,
> >> >> >>>> Luc
> >> >> >>>> On 16 Aug 2019, at 21:45, Aaron Brewster <asbrewster at lbl.gov
> <mailto:asbrewster at lbl.gov>> wrote:
> >> >> >>>> Hi, to avoid clouding Dorothee's documentation email thread,
> which I think is a highly useful enterprise, here's some thoughts about
> putting cctbx into pip.  Pip doesn't install non-python dependencies well.
> I don't think boost is available as a package on pip (at least the package
> version we use).  wxPython4 isn't portable through pip (
> https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython-Phoenix_using_pip).
> MPI libraries are system dependent.  If cctbx were a pure python package,
> pip would be fine, but cctbx is not.
> >> >> >>>> All that said, we could build a manylinux1 version of cctbx and
> upload it to PyPi (I'm just learning about this).  For a pip package to be
> portable (which is a requirement for cctbx), it needs to conform to PEP513,
> the manylinux1 standard (https://www.python.org/dev/peps/pep-0513/).  For
> example, numpy is built according to this standard (see
> https://pypi.org/project/numpy/#files, where you'll see the manylinux1
> wheel).  Note, the manylinux1 standard is built with Centos 5.11 which we
> no longer support.
> >> >> >>>> There is also a manylinux2010 standard, which is based on
> Centos 6 (https://www.python.org/dev/peps/pep-0571/).  This is likely a
> more attainable target (note though by default C++11 is not supported on
> Centos 6).
> >> >> >>>> If we built a manylinuxX version of cctbx and uploaded it to
> PyPi, the user would need all the non-python dependencies.  There's no way
> to specify these in pip.  For example, cctbx requires boost 1.63 or
> better.  The user will need to have it in a place their python can find it,
> or we could package it ourselves and supply it, similar to how the pip h5py
> package now comes with an hd5f library, or how the pip numpy package
> includes an openblas library.  We'd have to do the same for any packages we
> depend on that aren't on pip using the manylinux standards, such as
> wxPython4.
> >> >> >>>> Further, we need to think about how dials and other cctbx-based
> packages interact.  If pip install cctbx is set up, how does pip install
> dials work, such that any dials shared libraries can find the cctbx
> libraries?  Can shared libraries from one pip package link against
> libraries in another pip package?  Would each package need to supply its
> own boost?  Possibly this is well understood in the pip field, but not by
> me :)
> >> >> >>>> Finally, there's the option of providing a source pip package.
> This would require the full compiler toolchain for any given platform
> (macOS, linux, windows).  These are likely available for developers, but
> not for general users.
> >> >> >>>> Anyway, these are some of the obstacles.  Not saying it isn't
> possible, it's just a lot of effort.
> >> >> >>>> Thanks,
> >> >> >>>> -Aaron
> >> >> >>>> _______________________________________________
> >> >> >>>> cctbxbb mailing list
> >> >> >>>> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> >> >>>> _______________________________________________
> >> >> >>>> cctbxbb mailing list
> >> >> >>>> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> >> >>>> _______________________________________________
> >> >> >>>> cctbxbb mailing list
> >> >> >>>> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> >> >>>> --
> >> >> >>>> This e-mail and any attachments may contain confidential,
> copyright and or privileged material, and are for the use of the intended
> addressee only. If you are not the intended addressee or an authorised
> recipient of the addressee please notify us of receipt by returning the
> e-mail and do not use, copy, retain, distribute or disclose the information
> in or attached to the e-mail.
> >> >> >>>> Any opinions expressed within this e-mail are those of the
> individual and not necessarily of Diamond Light Source Ltd.
> >> >> >>>> Diamond Light Source Ltd. cannot guarantee that this e-mail or
> any attachments are free from viruses and we cannot accept liability for
> any damage which you may sustain as a result of software viruses which may
> be transmitted in or with the message.
> >> >> >>>> Diamond Light Source Limited (company no. 4375679). Registered
> in England and Wales with its registered office at Diamond House, Harwell
> Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
> >> >> >>>> _______________________________________________
> >> >> >>>> cctbxbb mailing list
> >> >> >>>> cctbxbb at phenix-online.org
> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> >> >>> _______________________________________________
> >> >> >>> cctbxbb mailing list
> >> >> >>> cctbxbb at phenix-online.org
> >> >> >>> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> >> >>
> >> >> >>
> >> >> >> _______________________________________________
> >> >> >> cctbxbb mailing list
> >> >> >> cctbxbb at phenix-online.org
> >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> >> >
> >> >> >
> >> >> > _______________________________________________
> >> >> > cctbxbb mailing list
> >> >> > cctbxbb at phenix-online.org
> >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb
> >> >>
> >> >>
> >> >> _______________________________________________
> >> >> cctbxbb mailing list
> >> >> cctbxbb at phenix-online.org
> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> >
> >> > _______________________________________________
> >> > cctbxbb mailing list
> >> > cctbxbb at phenix-online.org
> >> > http://phenix-online.org/mailman/listinfo/cctbxbb
> >>
> >>
> >>
> >> --
> >> Gergely Katona, PhD
> >> Associate Professor
> >> Department of Chemistry and Molecular Biology, University of Gothenburg
> >> Box 462, 40530 Göteborg, Sweden
> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
> >> Web: http://katonalab.eu, Email: gergely.katona at gu.se
> >>
> >> _______________________________________________
> >> cctbxbb mailing list
> >> cctbxbb at phenix-online.org
> >> http://phenix-online.org/mailman/listinfo/cctbxbb
> >
> > _______________________________________________
> > cctbxbb mailing list
> > cctbxbb at phenix-online.org
> > http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
>
> --
> Gergely Katona, PhD
> Associate Professor
> Department of Chemistry and Molecular Biology, University of Gothenburg
> Box 462, 40530 Göteborg, Sweden
> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
> Web: http://katonalab.eu, Email: gergely.katona at gu.se
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://phenix-online.org/pipermail/cctbxbb/attachments/20191216/d409bdc6/attachment.htm>


More information about the cctbxbb mailing list