[cctbxbb] some thoughts on cctbx and pip

[Aaron Brewster]

Hi, to avoid clouding Dorothee's documentation email thread, which I think
is a highly useful enterprise, here's some thoughts about putting cctbx
into pip.  Pip doesn't install non-python dependencies well.  I don't think
boost is available as a package on pip (at least the package version we
use).  wxPython4 isn't portable through pip (
https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython-Phoenix_using_pip).
MPI libraries are system dependent.  If cctbx were a pure python package,
pip would be fine, but cctbx is not.

All that said, we could build a manylinux1 version of cctbx and upload it
to PyPi (I'm just learning about this).  For a pip package to be portable
(which is a requirement for cctbx), it needs to conform to PEP513, the
manylinux1 standard (https://www.python.org/dev/peps/pep-0513/).  For
example, numpy is built according to this standard (see
https://pypi.org/project/numpy/#files, where you'll see the manylinux1
wheel).  Note, the manylinux1 standard is built with Centos 5.11 which we
no longer support.

There is also a manylinux2010 standard, which is based on Centos 6 (
https://www.python.org/dev/peps/pep-0571/).  This is likely a more
attainable target (note though by default C++11 is not supported on Centos
6).

If we built a manylinuxX version of cctbx and uploaded it to PyPi, the user
would need all the non-python dependencies.  There's no way to specify
these in pip.  For example, cctbx requires boost 1.63 or better.  The user
will need to have it in a place their python can find it, or we could
package it ourselves and supply it, similar to how the pip h5py package now
comes with an hd5f library, or how the pip numpy package includes an
openblas library.  We'd have to do the same for any packages we depend on
that aren't on pip using the manylinux standards, such as wxPython4.

Further, we need to think about how dials and other cctbx-based packages
interact.  If pip install cctbx is set up, how does pip install dials work,
such that any dials shared libraries can find the cctbx libraries?  Can
shared libraries from one pip package link against libraries in another pip
package?  Would each package need to supply its own boost?  Possibly this
is well understood in the pip field, but not by me :)

Finally, there's the option of providing a source pip package.  This would
require the full compiler toolchain for any given platform (macOS, linux,
windows).  These are likely available for developers, but not for general
users.

Anyway, these are some of the obstacles.  Not saying it isn't possible,
it's just a lot of effort.

Thanks,
-Aaron
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://phenix-online.org/pipermail/cctbxbb/attachments/20190816/86af2bc1/attachment-0001.htm>