[cctbxbb] some thoughts on cctbx and pip

[Dr Robert Oeffner]

Agree. I talked with a small molecules software developer here at the ECM32 yesterday. He would love to be able to do “pip install cctbx” when he develops his program, provided it doesn’t take up much file space.

Rob 

Sent from my Windows 10 phone
-- 
Robert Oeffner, Ph.D.
Research Associate, The Read Group
Department of Haematology,
Cambridge Institute for Medical Research
University of Cambridge
Cambridge Biomedical Campus
Wellcome Trust/MRC Building
Hills Road
Cambridge CB2 0XY

www.cimr.cam.ac.uk/investigators/read/index.html
tel: +44(0)1223 763234

From: Derek Mendez
Sent: 20 August 2019 18:06
To: cctbx mailing list
Subject: Re: [cctbxbb] some thoughts on cctbx and pip

I think its worth getting a cctbx-light pip build..  I think modules like cctbx miller, sgtbx are extremely  useful. Also simtbx.nanoBragg. 

-Derek

On Sun, Aug 18, 2019 at 10:48 PM Graeme.Winter at Diamond.ac.uk <Graeme.Winter at diamond.ac.uk> wrote:
Hi Aaron

Re: talk about if interest

I think it would be very useful to have a roadmap of where this is going, the intentions and how we expect it to play with other cctbx-dependent projects - have I missed this? Having it somewhere e.g. on the wiki would be a big help

Thanks Graeme

On 17 Aug 2019, at 00:36, Aaron Brewster <asbrewster at lbl.gov<mailto:asbrewster at lbl.gov>> wrote:

Hi Luc, thanks.  I did recall someone working on this a while back.

For conda, there are a couple more things to finish and then we hope to have cctbx available through conda.

1) There is work being done in a branch to make cctbx use boost in standard locations (e.g. the system boost, or a conda boost).  That will allow us to use install boost by conda and not build it ourselves.  (https://github.com/cctbx/cctbx_project/tree/conda_boost).  Also, newer versions of boost are being tested (up through 1.70).  (This will also enable python 3.7 support.)
2) We need to work on getting a make install step in place so that we can build the conda package and upload it.
3) We want to split the dependencies up by builder (cctbx, cctbx-lite, dials, phenix, etc.) into meta packages and their associated manifests.  I can talk more about this if there is interest.

Thanks,
-Aaron



On Fri, Aug 16, 2019 at 1:48 PM Luc Bourhis <luc_j_bourhis at mac.com<mailto:luc_j_bourhis at mac.com>> wrote:
Hi,

I did look into that many years ago, and even toyed with building a pip installer. What stopped me is the exact conclusion you reached too: the user would not have the pip experience he expects. You are right that it is a lot of effort but is it worth it? Considering that remark, I don’t think so. Now, Conda was created specifically to go beyond pip pure-python-only support. Since cctbx has garnered support for Conda, the best avenue imho is to go the extra length to have a package on Anaconda.org<http://anaconda.org/>, and then to advertise it hard to every potential user out there.

Best wishes,

Luc


On 16 Aug 2019, at 21:45, Aaron Brewster <asbrewster at lbl.gov<mailto:asbrewster at lbl.gov>> wrote:

Hi, to avoid clouding Dorothee's documentation email thread, which I think is a highly useful enterprise, here's some thoughts about putting cctbx into pip.  Pip doesn't install non-python dependencies well.  I don't think boost is available as a package on pip (at least the package version we use).  wxPython4 isn't portable through pip (https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython-Phoenix_using_pip).  MPI libraries are system dependent.  If cctbx were a pure python package, pip would be fine, but cctbx is not.

All that said, we could build a manylinux1 version of cctbx and upload it to PyPi (I'm just learning about this).  For a pip package to be portable (which is a requirement for cctbx), it needs to conform to PEP513, the manylinux1 standard (https://www.python.org/dev/peps/pep-0513/).  For example, numpy is built according to this standard (see https://pypi.org/project/numpy/#files, where you'll see the manylinux1 wheel).  Note, the manylinux1 standard is built with Centos 5.11 which we no longer support.

There is also a manylinux2010 standard, which is based on Centos 6 (https://www.python.org/dev/peps/pep-0571/).  This is likely a more attainable target (note though by default C++11 is not supported on Centos 6).

If we built a manylinuxX version of cctbx and uploaded it to PyPi, the user would need all the non-python dependencies.  There's no way to specify these in pip.  For example, cctbx requires boost 1.63 or better.  The user will need to have it in a place their python can find it, or we could package it ourselves and supply it, similar to how the pip h5py package now comes with an hd5f library, or how the pip numpy package includes an openblas library.  We'd have to do the same for any packages we depend on that aren't on pip using the manylinux standards, such as wxPython4.

Further, we need to think about how dials and other cctbx-based packages interact.  If pip install cctbx is set up, how does pip install dials work, such that any dials shared libraries can find the cctbx libraries?  Can shared libraries from one pip package link against libraries in another pip package?  Would each package need to supply its own boost?  Possibly this is well understood in the pip field, but not by me :)

Finally, there's the option of providing a source pip package.  This would require the full compiler toolchain for any given platform (macOS, linux, windows).  These are likely available for developers, but not for general users.

Anyway, these are some of the obstacles.  Not saying it isn't possible, it's just a lot of effort.

Thanks,
-Aaron

_______________________________________________
cctbxbb mailing list
cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________
cctbxbb mailing list
cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
http://phenix-online.org/mailman/listinfo/cctbxbb
_______________________________________________
cctbxbb mailing list
cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
http://phenix-online.org/mailman/listinfo/cctbxbb


-- 
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. 
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

_______________________________________________
cctbxbb mailing list
cctbxbb at phenix-online.org
http://phenix-online.org/mailman/listinfo/cctbxbb

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://phenix-online.org/pipermail/cctbxbb/attachments/20190822/16fbfbde/attachment.htm>