[cctbxbb] some thoughts on cctbx and pip

Tristan Croll tic20 at cam.ac.uk
Fri Aug 23 21:40:36 PDT 2019


Hi Billy,

Sorry, I wasn’t clear: ChimeraX can install packages *from* Pip, but isn’t installable that way itself. It’s built and distributed as a self-contained application. Would certainly make my life easier if they were to migrate to Conda, but I don’t see that happening in the near future.

Best regards,

Tristan

 
 

 

> On 24 Aug 2019, at 02:16, Billy Poon <BKPoon at lbl.gov> wrote:
> 
> Hi all,
> 
> Sorry for the delayed response, I was getting the message digests to test some bulletin board settings, so I wasn't getting the individual messages for replying.
> 
> For a brief update on conda,
> 
> 1) External Boost libraries can now be used to build on all platforms (linux, macOS, Windows) and I'm testing updated environments with Boost 1.68 and 1.70. This is for Python 2.7, 3.6, and some initial testing on 3.7. Once 3.7 is ready, Azure Pipelines will be updated to also build 3.7. I'll do the updated dependencies as a pull request since there will be changes that stop the downloading of modules/boost. Mixing Boost versions can cause seg faults or unexpected behavior. I also did some brief testing with 1.65 since that is the minimum version that the latest versions of CUDA require.
> 2) The next step, like Aaron mentioned, is to get our build system to put things in more standard locations (e.g. dispatchers/binaries in $PREFIX/bin, shared libraries in $PREFIX/lib, Python source/extension modules in $PREFIX/lib/python<version>/site-packages) as a "make install" command. conda on Windows has a similar hierarchy that's specific to conda, so for Windows, this will be conda-specific.
> 3) Update the conda_compiler branch (https://github.com/cctbx/cctbx_project/tree/conda_compiler). This lets the build process use the standard compilers in conda's build system.
> 
> Then a CCTBX conda package can be built and added to the conda-forge channel. The conda environments for building CCTBX are already based on conda-forge dependencies.
> 
> 4) With the ability to build a conda package, conda constructor (https://github.com/conda/constructor) can be used for building our installers. This would be the more official way of packaging CCTBX with conda dependencies as a stand-alone installer. One of the roadblocks to using conda constructor was supporting noarch packages. It looks like that has been fixed (https://www.anaconda.com/announcing-the-anaconda-2019-07-release/).  
> 
> So a goal for this work with conda is to make CCTBX-based programs installable as conda packages so that installing and running the software is consistent across platforms and operating system versions while also making it easier for users to add their tools to the software stack.
> 
> We do have a Phenix release coming up, so I won't be able to spend much effort on this until later in September.
> 
> Tristan,
> 
> If ChimeraX is installable via pip, it can be installed into a conda environment. And once 2) is complete, you would be able to access CCTBX from ChimeraX. Also, this would make installing ChimeraX pretty easy for Phenix. Interestingly, the latest open source PyMOL can be built with dependencies from Schrodinger's conda channel, so it should be possible to build a conda package for PyMOL that integrates with our conda environment.
> 
> --
> Billy K. Poon
> Research Scientist, Molecular Biophysics and Integrated Bioimaging
> Lawrence Berkeley National Laboratory
> 1 Cyclotron Road, M/S 33R0345
> Berkeley, CA 94720
> Tel: (510) 486-5709
> Fax: (510) 486-5909
> Web: https://phenix-online.org
> 
> 
>> On Fri, Aug 23, 2019 at 10:03 AM Tristan Croll <tic20 at cam.ac.uk> wrote:
>> To add my two cents on this: probably the second-most common question 
>> I've had about ISOLDE's implementation is, "why didn't you use CCTBX?". 
>> The honest answer to that is, "I didn't know how."
>> 
>> Still don't, really - although the current developments are rather 
>> promising. The problem I've faced is that CCTBX was designed as its own 
>> self-contained Python (2.7, until very recently) environment, with its 
>> own interpreter and a lot of very specific environment setup. Meanwhile 
>> I'm developing ISOLDE in ChimeraX, which is *also* its own 
>> self-contained Python (3.7) environment. To plug one into the other in 
>> that form... well, I don't think I'm a good enough programmer to really 
>> know where to start.
>> 
>> The move to Conda and a more modular CCTBX architecture should make a 
>> lot more possible in that direction. Pip would be even better for me 
>> personally (ChimeraX can install directly from the PyPI, but doesn't 
>> interact with Conda) - but I understand pretty well the substantial 
>> challenge that would amount to (not least being that the PyPI imposes a 
>> limit - around 100MB from memory? - on the size of an individual 
>> package).
>> 
>> Best regards,
>> 
>> Tristan
>> 
>> On 2019-08-23 09:28, Luc Bourhis wrote:
>> > Hi Graeme,
>> > 
>> > Yes, I know. But “black" is a program doing a very particular task
>> > (code formatting from the top of my head). Requiring to use a wrapper
>> > for python itself is another level. But ok, I think I am mellowing to
>> > the idea after all! Talking with people around me, and extrapolating,
>> > I would bet that, right now, a great majority of people interested by
>> > cctbx in pip have already used the cctbx, so they know about the
>> > Python wrapper, and they would not be too sanguine about that. My
>> > concern is for the future, when pip will be the first time some people
>> > use cctbx. Big fat warning notices on PyPI page and a better error
>> > message when cctbx fails because LIBTBX_BUILD is not set would be
>> > needed but that could be all right.
>> > 
>> > If we do a pip installer, we should aim at a minimal install: cctbx,
>> > iotbx and their dependencies, and that’s it.
>> > 
>> > Best wishes,
>> > 
>> > Luc
>> > 
>> > 
>> >> On 23 Aug 2019, at 07:17, Graeme.Winter at Diamond.ac.uk 
>> >> <Graeme.Winter at diamond.ac.uk> wrote:
>> >> 
>> >> Without discussing the merits of this or whether we _choose_ to make 
>> >> the move to supporting PIP, I am certain it would be _possible_ - many 
>> >> other packages make dispatcher scripts when you pip install them e.g.
>> >> 
>> >> Silver-Surfer rescale_f2 :) $ which black; cat $(which black)
>> >> /Library/Frameworks/Python.framework/Versions/3.6/bin/black
>> >> #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6
>> >> 
>> >> # -*- coding: utf-8 -*-
>> >> import re
>> >> import sys
>> >> 
>> >> from black import main
>> >> 
>> >> if __name__ == '__main__':
>> >>    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
>> >>    sys.exit(main())
>> >> 
>> >> So we _could_ work around the absence of LIBTBX_BUILD etc. in the 
>> >> system. Whether or not we elect to do the work is a different 
>> >> question, and it seems clear that here are very mixed opinions on 
>> >> this.
>> >> 
>> >> Best wishes Graeme
>> >> 
>> >> 
>> >> On 23 Aug 2019, at 01:21, Luc Bourhis 
>> >> <luc_j_bourhis at mac.com<mailto:luc_j_bourhis at mac.com>> wrote:
>> >> 
>> >> Hi,
>> >> 
>> >> Even if we managed to ship our the boost dynamic libraries with pip, 
>> >> it would still not be pip-like, as we would still need our python 
>> >> wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal pip packages 
>> >> work with the standard python exe. LD_LIBRARY_PATH, we could get 
>> >> around that by changing the way we compile, using -Wl,-R, which is the 
>> >> runtime equivalent of build time -L. That’s a significant change that 
>> >> would need to be tested. But there is no way around setting 
>> >> LIBTBX_BUILD right now. Leaving that to the user is horrible. Perhaps 
>> >> there is a way to hack libtbx/env_config.py so that we can hardwire 
>> >> LIBTBX_BUILD in there when pip installs?
>> >> 
>> >> Best wishes,
>> >> 
>> >> Luc
>> >> 
>> >> 
>> >> On 16 Aug 2019, at 22:47, Luc Bourhis 
>> >> <luc_j_bourhis at mac.com<mailto:luc_j_bourhis at mac.com>> wrote:
>> >> 
>> >> Hi,
>> >> 
>> >> I did look into that many years ago, and even toyed with building a 
>> >> pip installer. What stopped me is the exact conclusion you reached 
>> >> too: the user would not have the pip experience he expects. You are 
>> >> right that it is a lot of effort but is it worth it? Considering that 
>> >> remark, I don’t think so. Now, Conda was created specifically to go 
>> >> beyond pip pure-python-only support. Since cctbx has garnered support 
>> >> for Conda, the best avenue imho is to go the extra length to have a 
>> >> package on Anaconda.org<http://anaconda.org/>, and then to advertise 
>> >> it hard to every potential user out there.
>> >> 
>> >> Best wishes,
>> >> 
>> >> Luc
>> >> 
>> >> 
>> >> On 16 Aug 2019, at 21:45, Aaron Brewster 
>> >> <asbrewster at lbl.gov<mailto:asbrewster at lbl.gov>> wrote:
>> >> 
>> >> Hi, to avoid clouding Dorothee's documentation email thread, which I 
>> >> think is a highly useful enterprise, here's some thoughts about 
>> >> putting cctbx into pip.  Pip doesn't install non-python dependencies 
>> >> well.  I don't think boost is available as a package on pip (at least 
>> >> the package version we use).  wxPython4 isn't portable through pip 
>> >> (https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython-Phoenix_using_pip). 
>> >>  MPI libraries are system dependent.  If cctbx were a pure python 
>> >> package, pip would be fine, but cctbx is not.
>> >> 
>> >> All that said, we could build a manylinux1 version of cctbx and upload 
>> >> it to PyPi (I'm just learning about this).  For a pip package to be 
>> >> portable (which is a requirement for cctbx), it needs to conform to 
>> >> PEP513, the manylinux1 standard 
>> >> (https://www.python.org/dev/peps/pep-0513/).  For example, numpy is 
>> >> built according to this standard (see 
>> >> https://pypi.org/project/numpy/#files, where you'll see the manylinux1 
>> >> wheel).  Note, the manylinux1 standard is built with Centos 5.11 which 
>> >> we no longer support.
>> >> 
>> >> There is also a manylinux2010 standard, which is based on Centos 6 
>> >> (https://www.python.org/dev/peps/pep-0571/).  This is likely a more 
>> >> attainable target (note though by default C++11 is not supported on 
>> >> Centos 6).
>> >> 
>> >> If we built a manylinuxX version of cctbx and uploaded it to PyPi, the 
>> >> user would need all the non-python dependencies.  There's no way to 
>> >> specify these in pip.  For example, cctbx requires boost 1.63 or 
>> >> better.  The user will need to have it in a place their python can 
>> >> find it, or we could package it ourselves and supply it, similar to 
>> >> how the pip h5py package now comes with an hd5f library, or how the 
>> >> pip numpy package includes an openblas library.  We'd have to do the 
>> >> same for any packages we depend on that aren't on pip using the 
>> >> manylinux standards, such as wxPython4.
>> >> 
>> >> Further, we need to think about how dials and other cctbx-based 
>> >> packages interact.  If pip install cctbx is set up, how does pip 
>> >> install dials work, such that any dials shared libraries can find the 
>> >> cctbx libraries?  Can shared libraries from one pip package link 
>> >> against libraries in another pip package?  Would each package need to 
>> >> supply its own boost?  Possibly this is well understood in the pip 
>> >> field, but not by me :)
>> >> 
>> >> Finally, there's the option of providing a source pip package.  This 
>> >> would require the full compiler toolchain for any given platform 
>> >> (macOS, linux, windows).  These are likely available for developers, 
>> >> but not for general users.
>> >> 
>> >> Anyway, these are some of the obstacles.  Not saying it isn't 
>> >> possible, it's just a lot of effort.
>> >> 
>> >> Thanks,
>> >> -Aaron
>> >> 
>> >> _______________________________________________
>> >> cctbxbb mailing list
>> >> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
>> >> http://phenix-online.org/mailman/listinfo/cctbxbb
>> >> 
>> >> _______________________________________________
>> >> cctbxbb mailing list
>> >> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
>> >> http://phenix-online.org/mailman/listinfo/cctbxbb
>> >> 
>> >> _______________________________________________
>> >> cctbxbb mailing list
>> >> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
>> >> http://phenix-online.org/mailman/listinfo/cctbxbb
>> >> 
>> >> 
>> >> --
>> >> This e-mail and any attachments may contain confidential, copyright 
>> >> and or privileged material, and are for the use of the intended 
>> >> addressee only. If you are not the intended addressee or an authorised 
>> >> recipient of the addressee please notify us of receipt by returning 
>> >> the e-mail and do not use, copy, retain, distribute or disclose the 
>> >> information in or attached to the e-mail.
>> >> Any opinions expressed within this e-mail are those of the individual 
>> >> and not necessarily of Diamond Light Source Ltd.
>> >> Diamond Light Source Ltd. cannot guarantee that this e-mail or any 
>> >> attachments are free from viruses and we cannot accept liability for 
>> >> any damage which you may sustain as a result of software viruses which 
>> >> may be transmitted in or with the message.
>> >> Diamond Light Source Limited (company no. 4375679). Registered in 
>> >> England and Wales with its registered office at Diamond House, Harwell 
>> >> Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United 
>> >> Kingdom
>> >> 
>> >> _______________________________________________
>> >> cctbxbb mailing list
>> >> cctbxbb at phenix-online.org
>> >> http://phenix-online.org/mailman/listinfo/cctbxbb
>> > 
>> > 
>> > _______________________________________________
>> > cctbxbb mailing list
>> > cctbxbb at phenix-online.org
>> > http://phenix-online.org/mailman/listinfo/cctbxbb
>> 
>> 
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb at phenix-online.org
>> http://phenix-online.org/mailman/listinfo/cctbxbb
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://phenix-online.org/pipermail/cctbxbb/attachments/20190824/180bc8c1/attachment.htm>


More information about the cctbxbb mailing list