[cctbxbb] some thoughts on cctbx and pip

Nigel Moriarty nwmoriarty at lbl.gov
Thu Aug 22 16:07:44 PDT 2019 

-1

Cheers

Nigel

---
Nigel W. Moriarty
Building 33R0349, Molecular Biophysics and Integrated Bioimaging
Lawrence Berkeley National Laboratory
Berkeley, CA 94720-8235
Phone : 510-486-5709     Email : NWMoriarty at LBL.gov
Fax   : 510-486-5909       Web  : CCI.LBL.gov


On Thu, Aug 22, 2019 at 1:48 AM Dr Robert Oeffner <rdo20 at cam.ac.uk> wrote:

> Agree. I talked with a small molecules software developer here at the
> ECM32 yesterday. He would love to be able to do “pip install cctbx” when he
> develops his program, provided it doesn’t take up much file space.
>
>
>
> Rob
>
>
>
> Sent from my Windows 10 phone
> --
> Robert Oeffner, Ph.D.
> Research Associate, The Read Group
> Department of Haematology,
> Cambridge Institute for Medical Research
> University of Cambridge
> Cambridge Biomedical Campus
> Wellcome Trust/MRC Building
> Hills Road
> Cambridge CB2 0XY
>
> www.cimr.cam.ac.uk/investigators/read/index.html
> tel: +44(0)1223 763234
>
>
>
> *From: *Derek Mendez <dermen at lbl.gov>
> *Sent: *20 August 2019 18:06
> *To: *cctbx mailing list <cctbxbb at phenix-online.org>
> *Subject: *Re: [cctbxbb] some thoughts on cctbx and pip
>
>
>
> I think its worth getting a cctbx-light pip build..  I think modules like
> cctbx miller, sgtbx are extremely  useful. Also simtbx.nanoBragg.
>
>
>
> -Derek
>
>
>
> On Sun, Aug 18, 2019 at 10:48 PM Graeme.Winter at Diamond.ac.uk <
> Graeme.Winter at diamond.ac.uk> wrote:
>
> Hi Aaron
>
> Re: talk about if interest
>
> I think it would be very useful to have a roadmap of where this is going,
> the intentions and how we expect it to play with other cctbx-dependent
> projects - have I missed this? Having it somewhere e.g. on the wiki would
> be a big help
>
> Thanks Graeme
>
> On 17 Aug 2019, at 00:36, Aaron Brewster <asbrewster at lbl.gov<mailto:
> asbrewster at lbl.gov>> wrote:
>
> Hi Luc, thanks.  I did recall someone working on this a while back.
>
> For conda, there are a couple more things to finish and then we hope to
> have cctbx available through conda.
>
> 1) There is work being done in a branch to make cctbx use boost in
> standard locations (e.g. the system boost, or a conda boost).  That will
> allow us to use install boost by conda and not build it ourselves.  (
> https://github.com/cctbx/cctbx_project/tree/conda_boost).  Also, newer
> versions of boost are being tested (up through 1.70).  (This will also
> enable python 3.7 support.)
> 2) We need to work on getting a make install step in place so that we can
> build the conda package and upload it.
> 3) We want to split the dependencies up by builder (cctbx, cctbx-lite,
> dials, phenix, etc.) into meta packages and their associated manifests.  I
> can talk more about this if there is interest.
>
> Thanks,
> -Aaron
>
>
>
> On Fri, Aug 16, 2019 at 1:48 PM Luc Bourhis <luc_j_bourhis at mac.com<mailto:
> luc_j_bourhis at mac.com>> wrote:
> Hi,
>
> I did look into that many years ago, and even toyed with building a pip
> installer. What stopped me is the exact conclusion you reached too: the
> user would not have the pip experience he expects. You are right that it is
> a lot of effort but is it worth it? Considering that remark, I don’t think
> so. Now, Conda was created specifically to go beyond pip pure-python-only
> support. Since cctbx has garnered support for Conda, the best avenue imho
> is to go the extra length to have a package on Anaconda.org<
> http://anaconda.org/>, and then to advertise it hard to every potential
> user out there.
>
> Best wishes,
>
> Luc
>
>
> On 16 Aug 2019, at 21:45, Aaron Brewster <asbrewster at lbl.gov<mailto:
> asbrewster at lbl.gov>> wrote:
>
> Hi, to avoid clouding Dorothee's documentation email thread, which I think
> is a highly useful enterprise, here's some thoughts about putting cctbx
> into pip.  Pip doesn't install non-python dependencies well.  I don't think
> boost is available as a package on pip (at least the package version we
> use).  wxPython4 isn't portable through pip (
> https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython-Phoenix_using_pip).
> MPI libraries are system dependent.  If cctbx were a pure python package,
> pip would be fine, but cctbx is not.
>
> All that said, we could build a manylinux1 version of cctbx and upload it
> to PyPi (I'm just learning about this).  For a pip package to be portable
> (which is a requirement for cctbx), it needs to conform to PEP513, the
> manylinux1 standard (https://www.python.org/dev/peps/pep-0513/).  For
> example, numpy is built according to this standard (see
> https://pypi.org/project/numpy/#files, where you'll see the manylinux1
> wheel).  Note, the manylinux1 standard is built with Centos 5.11 which we
> no longer support.
>
> There is also a manylinux2010 standard, which is based on Centos 6 (
> https://www.python.org/dev/peps/pep-0571/).  This is likely a more
> attainable target (note though by default C++11 is not supported on Centos
> 6).
>
> If we built a manylinuxX version of cctbx and uploaded it to PyPi, the
> user would need all the non-python dependencies.  There's no way to specify
> these in pip.  For example, cctbx requires boost 1.63 or better.  The user
> will need to have it in a place their python can find it, or we could
> package it ourselves and supply it, similar to how the pip h5py package now
> comes with an hd5f library, or how the pip numpy package includes an
> openblas library.  We'd have to do the same for any packages we depend on
> that aren't on pip using the manylinux standards, such as wxPython4.
>
> Further, we need to think about how dials and other cctbx-based packages
> interact.  If pip install cctbx is set up, how does pip install dials work,
> such that any dials shared libraries can find the cctbx libraries?  Can
> shared libraries from one pip package link against libraries in another pip
> package?  Would each package need to supply its own boost?  Possibly this
> is well understood in the pip field, but not by me :)
>
> Finally, there's the option of providing a source pip package.  This would
> require the full compiler toolchain for any given platform (macOS, linux,
> windows).  These are likely available for developers, but not for general
> users.
>
> Anyway, these are some of the obstacles.  Not saying it isn't possible,
> it's just a lot of effort.
>
> Thanks,
> -Aaron
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
> http://phenix-online.org/mailman/listinfo/cctbxbb
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
> --
> This e-mail and any attachments may contain confidential, copyright and or
> privileged material, and are for the use of the intended addressee only. If
> you are not the intended addressee or an authorised recipient of the
> addressee please notify us of receipt by returning the e-mail and do not
> use, copy, retain, distribute or disclose the information in or attached to
> the e-mail.
> Any opinions expressed within this e-mail are those of the individual and
> not necessarily of Diamond Light Source Ltd.
> Diamond Light Source Ltd. cannot guarantee that this e-mail or any
> attachments are free from viruses and we cannot accept liability for any
> damage which you may sustain as a result of software viruses which may be
> transmitted in or with the message.
> Diamond Light Source Limited (company no. 4375679). Registered in England
> and Wales with its registered office at Diamond House, Harwell Science and
> Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://phenix-online.org/pipermail/cctbxbb/attachments/20190822/c3a6aad1/attachment.htm>

More information about the cctbxbb mailing list