[cctbxbb] some thoughts on cctbx and pip

[Aaron Brewster]

Hi Luc, thanks.  I did recall someone working on this a while back.

For conda, there are a couple more things to finish and then we hope to
have cctbx available through conda.

1) There is work being done in a branch to make cctbx use boost in standard
locations (e.g. the system boost, or a conda boost).  That will allow us to
use install boost by conda and not build it ourselves.  (
https://github.com/cctbx/cctbx_project/tree/conda_boost).  Also, newer
versions of boost are being tested (up through 1.70).  (This will also
enable python 3.7 support.)
2) We need to work on getting a make install step in place so that we can
build the conda package and upload it.
3) We want to split the dependencies up by builder (cctbx, cctbx-lite,
dials, phenix, etc.) into meta packages and their associated manifests.  I
can talk more about this if there is interest.

Thanks,
-Aaron



On Fri, Aug 16, 2019 at 1:48 PM Luc Bourhis <luc_j_bourhis at mac.com> wrote:

> Hi,
>
> I did look into that many years ago, and even toyed with building a pip
> installer. What stopped me is the exact conclusion you reached too: the
> user would not have the pip experience he expects. You are right that it is
> a lot of effort but is it worth it? Considering that remark, I don’t think
> so. Now, Conda was created specifically to go beyond pip pure-python-only
> support. Since cctbx has garnered support for Conda, the best avenue imho
> is to go the extra length to have a package on Anaconda.org, and then to
> advertise it hard to every potential user out there.
>
> Best wishes,
>
> Luc
>
>
> On 16 Aug 2019, at 21:45, Aaron Brewster <asbrewster at lbl.gov> wrote:
>
> Hi, to avoid clouding Dorothee's documentation email thread, which I think
> is a highly useful enterprise, here's some thoughts about putting cctbx
> into pip.  Pip doesn't install non-python dependencies well.  I don't think
> boost is available as a package on pip (at least the package version we
> use).  wxPython4 isn't portable through pip (
> https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython-Phoenix_using_pip).
> MPI libraries are system dependent.  If cctbx were a pure python package,
> pip would be fine, but cctbx is not.
>
> All that said, we could build a manylinux1 version of cctbx and upload it
> to PyPi (I'm just learning about this).  For a pip package to be portable
> (which is a requirement for cctbx), it needs to conform to PEP513, the
> manylinux1 standard (https://www.python.org/dev/peps/pep-0513/).  For
> example, numpy is built according to this standard (see
> https://pypi.org/project/numpy/#files, where you'll see the manylinux1
> wheel).  Note, the manylinux1 standard is built with Centos 5.11 which we
> no longer support.
>
> There is also a manylinux2010 standard, which is based on Centos 6 (
> https://www.python.org/dev/peps/pep-0571/).  This is likely a more
> attainable target (note though by default C++11 is not supported on Centos
> 6).
>
> If we built a manylinuxX version of cctbx and uploaded it to PyPi, the
> user would need all the non-python dependencies.  There's no way to specify
> these in pip.  For example, cctbx requires boost 1.63 or better.  The user
> will need to have it in a place their python can find it, or we could
> package it ourselves and supply it, similar to how the pip h5py package now
> comes with an hd5f library, or how the pip numpy package includes an
> openblas library.  We'd have to do the same for any packages we depend on
> that aren't on pip using the manylinux standards, such as wxPython4.
>
> Further, we need to think about how dials and other cctbx-based packages
> interact.  If pip install cctbx is set up, how does pip install dials work,
> such that any dials shared libraries can find the cctbx libraries?  Can
> shared libraries from one pip package link against libraries in another pip
> package?  Would each package need to supply its own boost?  Possibly this
> is well understood in the pip field, but not by me :)
>
> Finally, there's the option of providing a source pip package.  This would
> require the full compiler toolchain for any given platform (macOS, linux,
> windows).  These are likely available for developers, but not for general
> users.
>
> Anyway, these are some of the obstacles.  Not saying it isn't possible,
> it's just a lot of effort.
>
> Thanks,
> -Aaron
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://phenix-online.org/pipermail/cctbxbb/attachments/20190816/894fdfbd/attachment.htm>