[cctbxbb] some thoughts on cctbx and pip

[Aaron Brewster]

(note, power at LBL is going down now, so save more thoughts on this until
Sunday afternoon, in case the messages are lost :)

On Fri, Aug 16, 2019 at 4:36 PM Aaron Brewster <asbrewster at lbl.gov> wrote:

> Hi Luc, thanks.  I did recall someone working on this a while back.
>
> For conda, there are a couple more things to finish and then we hope to
> have cctbx available through conda.
>
> 1) There is work being done in a branch to make cctbx use boost in
> standard locations (e.g. the system boost, or a conda boost).  That will
> allow us to use install boost by conda and not build it ourselves.  (
> https://github.com/cctbx/cctbx_project/tree/conda_boost).  Also, newer
> versions of boost are being tested (up through 1.70).  (This will also
> enable python 3.7 support.)
> 2) We need to work on getting a make install step in place so that we can
> build the conda package and upload it.
> 3) We want to split the dependencies up by builder (cctbx, cctbx-lite,
> dials, phenix, etc.) into meta packages and their associated manifests.  I
> can talk more about this if there is interest.
>
> Thanks,
> -Aaron
>
>
>
> On Fri, Aug 16, 2019 at 1:48 PM Luc Bourhis <luc_j_bourhis at mac.com> wrote:
>
>> Hi,
>>
>> I did look into that many years ago, and even toyed with building a pip
>> installer. What stopped me is the exact conclusion you reached too: the
>> user would not have the pip experience he expects. You are right that it is
>> a lot of effort but is it worth it? Considering that remark, I don’t think
>> so. Now, Conda was created specifically to go beyond pip pure-python-only
>> support. Since cctbx has garnered support for Conda, the best avenue imho
>> is to go the extra length to have a package on Anaconda.org, and then to
>> advertise it hard to every potential user out there.
>>
>> Best wishes,
>>
>> Luc
>>
>>
>> On 16 Aug 2019, at 21:45, Aaron Brewster <asbrewster at lbl.gov> wrote:
>>
>> Hi, to avoid clouding Dorothee's documentation email thread, which I
>> think is a highly useful enterprise, here's some thoughts about putting
>> cctbx into pip.  Pip doesn't install non-python dependencies well.  I don't
>> think boost is available as a package on pip (at least the package version
>> we use).  wxPython4 isn't portable through pip (
>> https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython-Phoenix_using_pip).
>> MPI libraries are system dependent.  If cctbx were a pure python package,
>> pip would be fine, but cctbx is not.
>>
>> All that said, we could build a manylinux1 version of cctbx and upload it
>> to PyPi (I'm just learning about this).  For a pip package to be portable
>> (which is a requirement for cctbx), it needs to conform to PEP513, the
>> manylinux1 standard (https://www.python.org/dev/peps/pep-0513/).  For
>> example, numpy is built according to this standard (see
>> https://pypi.org/project/numpy/#files, where you'll see the manylinux1
>> wheel).  Note, the manylinux1 standard is built with Centos 5.11 which we
>> no longer support.
>>
>> There is also a manylinux2010 standard, which is based on Centos 6 (
>> https://www.python.org/dev/peps/pep-0571/).  This is likely a more
>> attainable target (note though by default C++11 is not supported on Centos
>> 6).
>>
>> If we built a manylinuxX version of cctbx and uploaded it to PyPi, the
>> user would need all the non-python dependencies.  There's no way to specify
>> these in pip.  For example, cctbx requires boost 1.63 or better.  The user
>> will need to have it in a place their python can find it, or we could
>> package it ourselves and supply it, similar to how the pip h5py package now
>> comes with an hd5f library, or how the pip numpy package includes an
>> openblas library.  We'd have to do the same for any packages we depend on
>> that aren't on pip using the manylinux standards, such as wxPython4.
>>
>> Further, we need to think about how dials and other cctbx-based packages
>> interact.  If pip install cctbx is set up, how does pip install dials work,
>> such that any dials shared libraries can find the cctbx libraries?  Can
>> shared libraries from one pip package link against libraries in another pip
>> package?  Would each package need to supply its own boost?  Possibly this
>> is well understood in the pip field, but not by me :)
>>
>> Finally, there's the option of providing a source pip package.  This
>> would require the full compiler toolchain for any given platform (macOS,
>> linux, windows).  These are likely available for developers, but not for
>> general users.
>>
>> Anyway, these are some of the obstacles.  Not saying it isn't possible,
>> it's just a lot of effort.
>>
>> Thanks,
>> -Aaron
>>
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb at phenix-online.org
>> http://phenix-online.org/mailman/listinfo/cctbxbb
>>
>>
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb at phenix-online.org
>> http://phenix-online.org/mailman/listinfo/cctbxbb
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://phenix-online.org/pipermail/cctbxbb/attachments/20190816/f8db5700/attachment-0001.htm>