[cctbxbb] some thoughts on cctbx and pip
Aaron Brewster
asbrewster at lbl.gov
Fri Aug 16 16:38:25 PDT 2019
(note, power at LBL is going down now, so save more thoughts on this until
Sunday afternoon, in case the messages are lost :)
On Fri, Aug 16, 2019 at 4:36 PM Aaron Brewster <asbrewster at lbl.gov> wrote:
> Hi Luc, thanks. I did recall someone working on this a while back.
>
> For conda, there are a couple more things to finish and then we hope to
> have cctbx available through conda.
>
> 1) There is work being done in a branch to make cctbx use boost in
> standard locations (e.g. the system boost, or a conda boost). That will
> allow us to use install boost by conda and not build it ourselves. (
> https://github.com/cctbx/cctbx_project/tree/conda_boost). Also, newer
> versions of boost are being tested (up through 1.70). (This will also
> enable python 3.7 support.)
> 2) We need to work on getting a make install step in place so that we can
> build the conda package and upload it.
> 3) We want to split the dependencies up by builder (cctbx, cctbx-lite,
> dials, phenix, etc.) into meta packages and their associated manifests. I
> can talk more about this if there is interest.
>
> Thanks,
> -Aaron
>
>
>
> On Fri, Aug 16, 2019 at 1:48 PM Luc Bourhis <luc_j_bourhis at mac.com> wrote:
>
>> Hi,
>>
>> I did look into that many years ago, and even toyed with building a pip
>> installer. What stopped me is the exact conclusion you reached too: the
>> user would not have the pip experience he expects. You are right that it is
>> a lot of effort but is it worth it? Considering that remark, I don’t think
>> so. Now, Conda was created specifically to go beyond pip pure-python-only
>> support. Since cctbx has garnered support for Conda, the best avenue imho
>> is to go the extra length to have a package on Anaconda.org, and then to
>> advertise it hard to every potential user out there.
>>
>> Best wishes,
>>
>> Luc
>>
>>
>> On 16 Aug 2019, at 21:45, Aaron Brewster <asbrewster at lbl.gov> wrote:
>>
>> Hi, to avoid clouding Dorothee's documentation email thread, which I
>> think is a highly useful enterprise, here's some thoughts about putting
>> cctbx into pip. Pip doesn't install non-python dependencies well. I don't
>> think boost is available as a package on pip (at least the package version
>> we use). wxPython4 isn't portable through pip (
>> https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython-Phoenix_using_pip).
>> MPI libraries are system dependent. If cctbx were a pure python package,
>> pip would be fine, but cctbx is not.
>>
>> All that said, we could build a manylinux1 version of cctbx and upload it
>> to PyPi (I'm just learning about this). For a pip package to be portable
>> (which is a requirement for cctbx), it needs to conform to PEP513, the
>> manylinux1 standard (https://www.python.org/dev/peps/pep-0513/). For
>> example, numpy is built according to this standard (see
>> https://pypi.org/project/numpy/#files, where you'll see the manylinux1
>> wheel). Note, the manylinux1 standard is built with Centos 5.11 which we
>> no longer support.
>>
>> There is also a manylinux2010 standard, which is based on Centos 6 (
>> https://www.python.org/dev/peps/pep-0571/). This is likely a more
>> attainable target (note though by default C++11 is not supported on Centos
>> 6).
>>
>> If we built a manylinuxX version of cctbx and uploaded it to PyPi, the
>> user would need all the non-python dependencies. There's no way to specify
>> these in pip. For example, cctbx requires boost 1.63 or better. The user
>> will need to have it in a place their python can find it, or we could
>> package it ourselves and supply it, similar to how the pip h5py package now
>> comes with an hd5f library, or how the pip numpy package includes an
>> openblas library. We'd have to do the same for any packages we depend on
>> that aren't on pip using the manylinux standards, such as wxPython4.
>>
>> Further, we need to think about how dials and other cctbx-based packages
>> interact. If pip install cctbx is set up, how does pip install dials work,
>> such that any dials shared libraries can find the cctbx libraries? Can
>> shared libraries from one pip package link against libraries in another pip
>> package? Would each package need to supply its own boost? Possibly this
>> is well understood in the pip field, but not by me :)
>>
>> Finally, there's the option of providing a source pip package. This
>> would require the full compiler toolchain for any given platform (macOS,
>> linux, windows). These are likely available for developers, but not for
>> general users.
>>
>> Anyway, these are some of the obstacles. Not saying it isn't possible,
>> it's just a lot of effort.
>>
>> Thanks,
>> -Aaron
>>
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb at phenix-online.org
>> http://phenix-online.org/mailman/listinfo/cctbxbb
>>
>>
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb at phenix-online.org
>> http://phenix-online.org/mailman/listinfo/cctbxbb
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://phenix-online.org/pipermail/cctbxbb/attachments/20190816/f8db5700/attachment-0001.htm>
More information about the cctbxbb
mailing list