[cctbxbb] some thoughts on cctbx and pip

Fri Aug 23 10:02:06 PDT 2019

To add my two cents on this: probably the second-most common question 
I've had about ISOLDE's implementation is, "why didn't you use CCTBX?". 
The honest answer to that is, "I didn't know how."

Still don't, really - although the current developments are rather 
promising. The problem I've faced is that CCTBX was designed as its own 
self-contained Python (2.7, until very recently) environment, with its 
own interpreter and a lot of very specific environment setup. Meanwhile 
I'm developing ISOLDE in ChimeraX, which is *also* its own 
self-contained Python (3.7) environment. To plug one into the other in 
that form... well, I don't think I'm a good enough programmer to really 
know where to start.

The move to Conda and a more modular CCTBX architecture should make a 
lot more possible in that direction. Pip would be even better for me 
personally (ChimeraX can install directly from the PyPI, but doesn't 
interact with Conda) - but I understand pretty well the substantial 
challenge that would amount to (not least being that the PyPI imposes a 
limit - around 100MB from memory? - on the size of an individual 
package).

Best regards,

Tristan

On 2019-08-23 09:28, Luc Bourhis wrote:
> Hi Graeme,
> 
> Yes, I know. But “black" is a program doing a very particular task
> (code formatting from the top of my head). Requiring to use a wrapper
> for python itself is another level. But ok, I think I am mellowing to
> the idea after all! Talking with people around me, and extrapolating,
> I would bet that, right now, a great majority of people interested by
> cctbx in pip have already used the cctbx, so they know about the
> Python wrapper, and they would not be too sanguine about that. My
> concern is for the future, when pip will be the first time some people
> use cctbx. Big fat warning notices on PyPI page and a better error
> message when cctbx fails because LIBTBX_BUILD is not set would be
> needed but that could be all right.
> 
> If we do a pip installer, we should aim at a minimal install: cctbx,
> iotbx and their dependencies, and that’s it.
> 
> Best wishes,
> 
> Luc
> 
> 
>> On 23 Aug 2019, at 07:17, Graeme.Winter at Diamond.ac.uk 
>> <Graeme.Winter at diamond.ac.uk> wrote:
>> 
>> Without discussing the merits of this or whether we _choose_ to make 
>> the move to supporting PIP, I am certain it would be _possible_ - many 
>> other packages make dispatcher scripts when you pip install them e.g.
>> 
>> Silver-Surfer rescale_f2 :) $ which black; cat $(which black)
>> /Library/Frameworks/Python.framework/Versions/3.6/bin/black
>> #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6
>> 
>> # -*- coding: utf-8 -*-
>> import re
>> import sys
>> 
>> from black import main
>> 
>> if __name__ == '__main__':
>>    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
>>    sys.exit(main())
>> 
>> So we _could_ work around the absence of LIBTBX_BUILD etc. in the 
>> system. Whether or not we elect to do the work is a different 
>> question, and it seems clear that here are very mixed opinions on 
>> this.
>> 
>> Best wishes Graeme
>> 
>> 
>> On 23 Aug 2019, at 01:21, Luc Bourhis 
>> <luc_j_bourhis at mac.com<mailto:luc_j_bourhis at mac.com>> wrote:
>> 
>> Hi,
>> 
>> Even if we managed to ship our the boost dynamic libraries with pip, 
>> it would still not be pip-like, as we would still need our python 
>> wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal pip packages 
>> work with the standard python exe. LD_LIBRARY_PATH, we could get 
>> around that by changing the way we compile, using -Wl,-R, which is the 
>> runtime equivalent of build time -L. That’s a significant change that 
>> would need to be tested. But there is no way around setting 
>> LIBTBX_BUILD right now. Leaving that to the user is horrible. Perhaps 
>> there is a way to hack libtbx/env_config.py so that we can hardwire 
>> LIBTBX_BUILD in there when pip installs?
>> 
>> Best wishes,
>> 
>> Luc
>> 
>> 
>> On 16 Aug 2019, at 22:47, Luc Bourhis 
>> <luc_j_bourhis at mac.com<mailto:luc_j_bourhis at mac.com>> wrote:
>> 
>> Hi,
>> 
>> I did look into that many years ago, and even toyed with building a 
>> pip installer. What stopped me is the exact conclusion you reached 
>> too: the user would not have the pip experience he expects. You are 
>> right that it is a lot of effort but is it worth it? Considering that 
>> remark, I don’t think so. Now, Conda was created specifically to go 
>> beyond pip pure-python-only support. Since cctbx has garnered support 
>> for Conda, the best avenue imho is to go the extra length to have a 
>> package on Anaconda.org<http://anaconda.org/>, and then to advertise 
>> it hard to every potential user out there.
>> 
>> Best wishes,
>> 
>> Luc
>> 
>> 
>> On 16 Aug 2019, at 21:45, Aaron Brewster 
>> <asbrewster at lbl.gov<mailto:asbrewster at lbl.gov>> wrote:
>> 
>> Hi, to avoid clouding Dorothee's documentation email thread, which I 
>> think is a highly useful enterprise, here's some thoughts about 
>> putting cctbx into pip.  Pip doesn't install non-python dependencies 
>> well.  I don't think boost is available as a package on pip (at least 
>> the package version we use).  wxPython4 isn't portable through pip 
>> (https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython-Phoenix_using_pip). 
>>  MPI libraries are system dependent.  If cctbx were a pure python 
>> package, pip would be fine, but cctbx is not.
>> 
>> All that said, we could build a manylinux1 version of cctbx and upload 
>> it to PyPi (I'm just learning about this).  For a pip package to be 
>> portable (which is a requirement for cctbx), it needs to conform to 
>> PEP513, the manylinux1 standard 
>> (https://www.python.org/dev/peps/pep-0513/).  For example, numpy is 
>> built according to this standard (see 
>> https://pypi.org/project/numpy/#files, where you'll see the manylinux1 
>> wheel).  Note, the manylinux1 standard is built with Centos 5.11 which 
>> we no longer support.
>> 
>> There is also a manylinux2010 standard, which is based on Centos 6 
>> (https://www.python.org/dev/peps/pep-0571/).  This is likely a more 
>> attainable target (note though by default C++11 is not supported on 
>> Centos 6).
>> 
>> If we built a manylinuxX version of cctbx and uploaded it to PyPi, the 
>> user would need all the non-python dependencies.  There's no way to 
>> specify these in pip.  For example, cctbx requires boost 1.63 or 
>> better.  The user will need to have it in a place their python can 
>> find it, or we could package it ourselves and supply it, similar to 
>> how the pip h5py package now comes with an hd5f library, or how the 
>> pip numpy package includes an openblas library.  We'd have to do the 
>> same for any packages we depend on that aren't on pip using the 
>> manylinux standards, such as wxPython4.
>> 
>> Further, we need to think about how dials and other cctbx-based 
>> packages interact.  If pip install cctbx is set up, how does pip 
>> install dials work, such that any dials shared libraries can find the 
>> cctbx libraries?  Can shared libraries from one pip package link 
>> against libraries in another pip package?  Would each package need to 
>> supply its own boost?  Possibly this is well understood in the pip 
>> field, but not by me :)
>> 
>> Finally, there's the option of providing a source pip package.  This 
>> would require the full compiler toolchain for any given platform 
>> (macOS, linux, windows).  These are likely available for developers, 
>> but not for general users.
>> 
>> Anyway, these are some of the obstacles.  Not saying it isn't 
>> possible, it's just a lot of effort.
>> 
>> Thanks,
>> -Aaron
>> 
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
>> http://phenix-online.org/mailman/listinfo/cctbxbb
>> 
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
>> http://phenix-online.org/mailman/listinfo/cctbxbb
>> 
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
>> http://phenix-online.org/mailman/listinfo/cctbxbb
>> 
>> 
>> --
>> This e-mail and any attachments may contain confidential, copyright 
>> and or privileged material, and are for the use of the intended 
>> addressee only. If you are not the intended addressee or an authorised 
>> recipient of the addressee please notify us of receipt by returning 
>> the e-mail and do not use, copy, retain, distribute or disclose the 
>> information in or attached to the e-mail.
>> Any opinions expressed within this e-mail are those of the individual 
>> and not necessarily of Diamond Light Source Ltd.
>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any 
>> attachments are free from viruses and we cannot accept liability for 
>> any damage which you may sustain as a result of software viruses which 
>> may be transmitted in or with the message.
>> Diamond Light Source Limited (company no. 4375679). Registered in 
>> England and Wales with its registered office at Diamond House, Harwell 
>> Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United 
>> Kingdom
>> 
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb at phenix-online.org
>> http://phenix-online.org/mailman/listinfo/cctbxbb
> 
> 
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb