[cctbxbb] cctbx.python readline issue

wtempel wtempel at gmail.com
Sat Aug 24 19:21:10 PDT 2019


Thank you Billy and Blaine,
just when I was about to try Billy's suggested solution, I saw Blaine's
post. Because I had to pip install some other packages (like pyyaml)
anyway, I also pip-installed readline. Problem solved.
Best regards.
Wolfram

On Sat, Aug 24, 2019 at 3:47 PM Mooers, Blaine H.M. (HSC) <
Blaine-Mooers at ouhsc.edu> wrote:

> In response to Wolfram Tempel's trouble with backspace in interactive
> sessions,
> I found the same problem with backspace in interactive sessions in
> cctbx.python version 1723 on Mac OS 10.14.6.
> I solved the problem by installing readline with pip using the command
> below (I did not try using conda) and sourcing the again
> $HOME/local/cctbx-dev-1723/cctbx_env.sh.
>
> $HOME/local/cctbx-dev-1723/build/bin/libtbx.pip install --upgrade readline
>
> Best regards,
>
> Blaine
>
> Blaine Mooers, Ph.D.
> Associate Professor
> Department of Biochemistry and Molecular Biology
> College of Medicine
> University of Oklahoma Health Sciences Center
> S.L. Young Biomedical Research Center (BRC) Rm. 466
> 975 NE 10th Street, BRC 466
> Oklahoma City, OK 73104-5419
>
> ________________________________________
> From: cctbxbb-bounces at phenix-online.org [cctbxbb-bounces at phenix-online.org]
> on behalf of cctbxbb-request at phenix-online.org [
> cctbxbb-request at phenix-online.org]
> Sent: Saturday, August 24, 2019 2:00 PM
> To: cctbxbb at phenix-online.org
> Subject: [EXTERNAL] cctbxbb Digest, Vol 125, Issue 22
>
> Send cctbxbb mailing list submissions to
>         cctbxbb at phenix-online.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mailman_listinfo_cctbxbb&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=W95a_DpmVSjvqMSzHgvTQ4gWgnlGBm0aOrAOebMg2Lo&e=
> or, via email, send a message with subject or body 'help' to
>         cctbxbb-request at phenix-online.org
>
> You can reach the person managing the list at
>         cctbxbb-owner at phenix-online.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of cctbxbb digest..."
>
>
> Today's Topics:
>
>    1. interactive cctbx.python backspace behavior (on linux) (wtempel)
>    2. Re: some thoughts on cctbx and pip (Billy Poon)
>    3. Re: interactive cctbx.python backspace behavior (on linux)
>       (Billy Poon)
>    4. Re: interactive cctbx.python backspace behavior (on linux)
>       (Billy Poon)
>    5. Re: some thoughts on cctbx and pip (Tristan Croll)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 23 Aug 2019 15:54:09 -0400
> From: wtempel <wtempel at gmail.com>
> To: cctbxbb at phenix-online.org
> Subject: [cctbxbb] interactive cctbx.python backspace behavior (on
>         linux)
> Message-ID:
>         <
> CAGjFzmBj3m_Uez1Siz6HPrjxAdOJqSczO-udnJo7C2W5Od4Q8A at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi,
> Currently, cctbx.python behaves like described in this stackoverflow post:
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_893053_seeing-2Descape-2Dcharacters-2Dwhen-2Dpressing-2Dthe-2Darrow-2Dkeys-2Din-2Dpython-2Dshell&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=Q7VWA0EW6aVF6MGxZEXhez1kRpB0sqtTuvu5F51DxqI&e=
> .
> I am missing the convenience of backspace delete, which is available on
> /usr/bin/python interactive sessions. Could that feature be added during
> compilation of the pre-built cctbx distributions?
> Regards.
> Wolfram Tempel
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_pipermail_cctbxbb_attachments_20190823_46bd7e22_attachment.html&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=7cEr0UkQT56UP88kjBwf4McQwJRpYscwKqKQQ-r2mvo&e=
> >
>
> ------------------------------
>
> Message: 2
> Date: Fri, 23 Aug 2019 18:16:27 -0700
> From: Billy Poon <BKPoon at lbl.gov>
> To: cctbx mailing list <cctbxbb at phenix-online.org>
> Subject: Re: [cctbxbb] some thoughts on cctbx and pip
> Message-ID:
>         <
> CAHmYUca4xdx723Lx-MoKLbipvW3Ohgo9B614AjAPsCt8FGbnfg at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi all,
>
> Sorry for the delayed response, I was getting the message digests to test
> some bulletin board settings, so I wasn't getting the individual messages
> for replying.
>
> For a brief update on conda,
>
> 1) External Boost libraries can now be used to build on all platforms
> (linux, macOS, Windows) and I'm testing updated environments with Boost
> 1.68 and 1.70. This is for Python 2.7, 3.6, and some initial testing on
> 3.7. Once 3.7 is ready, Azure Pipelines will be updated to also build 3.7.
> I'll do the updated dependencies as a pull request since there will be
> changes that stop the downloading of modules/boost. Mixing Boost versions
> can cause seg faults or unexpected behavior. I also did some brief testing
> with 1.65 since that is the minimum version that the latest versions of
> CUDA require.
> 2) The next step, like Aaron mentioned, is to get our build system to put
> things in more standard locations (e.g. dispatchers/binaries in
> $PREFIX/bin, shared libraries in $PREFIX/lib, Python source/extension
> modules in $PREFIX/lib/python<version>/site-packages) as a "make install"
> command. conda on Windows has a similar hierarchy that's specific to conda,
> so for Windows, this will be conda-specific.
> 3) Update the conda_compiler branch (
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_cctbx_cctbx-5Fproject_tree_conda-5Fcompiler&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=pZLc23N2JvLGC3xcVjPt7-hTkrGBn7YKnKyAcudUNpo&e=
> ). This lets the
> build process use the standard compilers in conda's build system.
>
> Then a CCTBX conda package can be built and added to the conda-forge
> channel. The conda environments for building CCTBX are already based on
> conda-forge dependencies.
>
> 4) With the ability to build a conda package, conda constructor (
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_conda_constructor&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=mNtfU28QvccVCRK7pVUY96JCWFONmw1-Db6zZAmph_o&e=
> ) can be used for building our
> installers. This would be the more official way of packaging CCTBX with
> conda dependencies as a stand-alone installer. One of the roadblocks to
> using conda constructor was supporting noarch packages. It looks like that
> has been fixed (
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.anaconda.com_announcing-2Dthe-2Danaconda-2D2019-2D07-2Drelease_&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=9dHst_9qvOudkeSlaJVQvYLS_XATq38_Y60BX4jwcGg&e=
> ).
>
> So a goal for this work with conda is to make CCTBX-based programs
> installable as conda packages so that installing and running the software
> is consistent across platforms and operating system versions while also
> making it easier for users to add their tools to the software stack.
>
> We do have a Phenix release coming up, so I won't be able to spend much
> effort on this until later in September.
>
> Tristan,
>
> If ChimeraX is installable via pip, it can be installed into a conda
> environment. And once 2) is complete, you would be able to access CCTBX
> from ChimeraX. Also, this would make installing ChimeraX pretty easy for
> Phenix. Interestingly, the latest open source PyMOL can be built with
> dependencies from Schrodinger's conda channel, so it should be possible to
> build a conda package for PyMOL that integrates with our conda environment.
>
> --
> Billy K. Poon
> Research Scientist, Molecular Biophysics and Integrated Bioimaging
> Lawrence Berkeley National Laboratory
> 1 Cyclotron Road, M/S 33R0345
> Berkeley, CA 94720
> Tel: (510) 486-5709
> Fax: (510) 486-5909
> Web:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__phenix-2Donline.org&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=4z9DrBeWDv4KyPNyfSgrmvwWzefUCK69jc4d5Yyps7w&e=
>
>
> On Fri, Aug 23, 2019 at 10:03 AM Tristan Croll <tic20 at cam.ac.uk> wrote:
>
> > To add my two cents on this: probably the second-most common question
> > I've had about ISOLDE's implementation is, "why didn't you use CCTBX?".
> > The honest answer to that is, "I didn't know how."
> >
> > Still don't, really - although the current developments are rather
> > promising. The problem I've faced is that CCTBX was designed as its own
> > self-contained Python (2.7, until very recently) environment, with its
> > own interpreter and a lot of very specific environment setup. Meanwhile
> > I'm developing ISOLDE in ChimeraX, which is *also* its own
> > self-contained Python (3.7) environment. To plug one into the other in
> > that form... well, I don't think I'm a good enough programmer to really
> > know where to start.
> >
> > The move to Conda and a more modular CCTBX architecture should make a
> > lot more possible in that direction. Pip would be even better for me
> > personally (ChimeraX can install directly from the PyPI, but doesn't
> > interact with Conda) - but I understand pretty well the substantial
> > challenge that would amount to (not least being that the PyPI imposes a
> > limit - around 100MB from memory? - on the size of an individual
> > package).
> >
> > Best regards,
> >
> > Tristan
> >
> > On 2019-08-23 09:28, Luc Bourhis wrote:
> > > Hi Graeme,
> > >
> > > Yes, I know. But ?black" is a program doing a very particular task
> > > (code formatting from the top of my head). Requiring to use a wrapper
> > > for python itself is another level. But ok, I think I am mellowing to
> > > the idea after all! Talking with people around me, and extrapolating,
> > > I would bet that, right now, a great majority of people interested by
> > > cctbx in pip have already used the cctbx, so they know about the
> > > Python wrapper, and they would not be too sanguine about that. My
> > > concern is for the future, when pip will be the first time some people
> > > use cctbx. Big fat warning notices on PyPI page and a better error
> > > message when cctbx fails because LIBTBX_BUILD is not set would be
> > > needed but that could be all right.
> > >
> > > If we do a pip installer, we should aim at a minimal install: cctbx,
> > > iotbx and their dependencies, and that?s it.
> > >
> > > Best wishes,
> > >
> > > Luc
> > >
> > >
> > >> On 23 Aug 2019, at 07:17, Graeme.Winter at Diamond.ac.uk
> > >> <Graeme.Winter at diamond.ac.uk> wrote:
> > >>
> > >> Without discussing the merits of this or whether we _choose_ to make
> > >> the move to supporting PIP, I am certain it would be _possible_ - many
> > >> other packages make dispatcher scripts when you pip install them e.g.
> > >>
> > >> Silver-Surfer rescale_f2 :) $ which black; cat $(which black)
> > >> /Library/Frameworks/Python.framework/Versions/3.6/bin/black
> > >> #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6
> > >>
> > >> # -*- coding: utf-8 -*-
> > >> import re
> > >> import sys
> > >>
> > >> from black import main
> > >>
> > >> if __name__ == '__main__':
> > >>    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
> > >>    sys.exit(main())
> > >>
> > >> So we _could_ work around the absence of LIBTBX_BUILD etc. in the
> > >> system. Whether or not we elect to do the work is a different
> > >> question, and it seems clear that here are very mixed opinions on
> > >> this.
> > >>
> > >> Best wishes Graeme
> > >>
> > >>
> > >> On 23 Aug 2019, at 01:21, Luc Bourhis
> > >> <luc_j_bourhis at mac.com<mailto:luc_j_bourhis at mac.com>> wrote:
> > >>
> > >> Hi,
> > >>
> > >> Even if we managed to ship our the boost dynamic libraries with pip,
> > >> it would still not be pip-like, as we would still need our python
> > >> wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal pip packages
> > >> work with the standard python exe. LD_LIBRARY_PATH, we could get
> > >> around that by changing the way we compile, using -Wl,-R, which is the
> > >> runtime equivalent of build time -L. That?s a significant change that
> > >> would need to be tested. But there is no way around setting
> > >> LIBTBX_BUILD right now. Leaving that to the user is horrible. Perhaps
> > >> there is a way to hack libtbx/
> https://urldefense.proofpoint.com/v2/url?u=http-3A__env-5Fconfig.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=5tRgn0yDOi7u9JsSVXIMM6D5NzBFMtAC8OqFFrm-0qM&e=
> so that we can hardwire
> > >> LIBTBX_BUILD in there when pip installs?
> > >>
> > >> Best wishes,
> > >>
> > >> Luc
> > >>
> > >>
> > >> On 16 Aug 2019, at 22:47, Luc Bourhis
> > >> <luc_j_bourhis at mac.com<mailto:luc_j_bourhis at mac.com>> wrote:
> > >>
> > >> Hi,
> > >>
> > >> I did look into that many years ago, and even toyed with building a
> > >> pip installer. What stopped me is the exact conclusion you reached
> > >> too: the user would not have the pip experience he expects. You are
> > >> right that it is a lot of effort but is it worth it? Considering that
> > >> remark, I don?t think so. Now, Conda was created specifically to go
> > >> beyond pip pure-python-only support. Since cctbx has garnered support
> > >> for Conda, the best avenue imho is to go the extra length to have a
> > >> package on
> https://urldefense.proofpoint.com/v2/url?u=http-3A__Anaconda.org&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=LTXosS3SYkN2lzASpSd1A47FoDWWfKyKkJVkQknHbFI&e=
> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__anaconda.org_&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=xEUexfDy2KPfHxGictcK-Rnxag5jmm5RjzAKoJRKMFQ&e=
> >, and then to advertise
> > >> it hard to every potential user out there.
> > >>
> > >> Best wishes,
> > >>
> > >> Luc
> > >>
> > >>
> > >> On 16 Aug 2019, at 21:45, Aaron Brewster
> > >> <asbrewster at lbl.gov<mailto:asbrewster at lbl.gov>> wrote:
> > >>
> > >> Hi, to avoid clouding Dorothee's documentation email thread, which I
> > >> think is a highly useful enterprise, here's some thoughts about
> > >> putting cctbx into pip.  Pip doesn't install non-python dependencies
> > >> well.  I don't think boost is available as a package on pip (at least
> > >> the package version we use).  wxPython4 isn't portable through pip
> > >> (
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.wxpython.org_How-2520to-2520install-2520wxPython-23Installing-5FwxPython-2DPhoenix-5Fusing-5Fpip&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=3S4Ul0MLjWsEg9-Paov_bPhLYc9TAkgSSTZqHkyuqo8&e=
> ).
> >
> > >>  MPI libraries are system dependent.  If cctbx were a pure python
> > >> package, pip would be fine, but cctbx is not.
> > >>
> > >> All that said, we could build a manylinux1 version of cctbx and upload
> > >> it to PyPi (I'm just learning about this).  For a pip package to be
> > >> portable (which is a requirement for cctbx), it needs to conform to
> > >> PEP513, the manylinux1 standard
> > >> (
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.python.org_dev_peps_pep-2D0513_&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=HwLo0JfnP5hTlTNAyDpSgaD0IhQVgEhmEZzBa12e78o&e=
> ).  For example, numpy is
> > >> built according to this standard (see
> > >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__pypi.org_project_numpy_-23files&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=5aDA0n5NQ0qoMHMXpj3PBRPfB9GGnhlrZheurjjBmE4&e=
> , where you'll see the manylinux1
> > >> wheel).  Note, the manylinux1 standard is built with Centos 5.11 which
> > >> we no longer support.
> > >>
> > >> There is also a manylinux2010 standard, which is based on Centos 6
> > >> (
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.python.org_dev_peps_pep-2D0571_&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=lJpz6hVmDG53lwLTol5NkNDKUEwVeldbQytTkIBzJu0&e=
> ).  This is likely a more
> > >> attainable target (note though by default C++11 is not supported on
> > >> Centos 6).
> > >>
> > >> If we built a manylinuxX version of cctbx and uploaded it to PyPi, the
> > >> user would need all the non-python dependencies.  There's no way to
> > >> specify these in pip.  For example, cctbx requires boost 1.63 or
> > >> better.  The user will need to have it in a place their python can
> > >> find it, or we could package it ourselves and supply it, similar to
> > >> how the pip h5py package now comes with an hd5f library, or how the
> > >> pip numpy package includes an openblas library.  We'd have to do the
> > >> same for any packages we depend on that aren't on pip using the
> > >> manylinux standards, such as wxPython4.
> > >>
> > >> Further, we need to think about how dials and other cctbx-based
> > >> packages interact.  If pip install cctbx is set up, how does pip
> > >> install dials work, such that any dials shared libraries can find the
> > >> cctbx libraries?  Can shared libraries from one pip package link
> > >> against libraries in another pip package?  Would each package need to
> > >> supply its own boost?  Possibly this is well understood in the pip
> > >> field, but not by me :)
> > >>
> > >> Finally, there's the option of providing a source pip package.  This
> > >> would require the full compiler toolchain for any given platform
> > >> (macOS, linux, windows).  These are likely available for developers,
> > >> but not for general users.
> > >>
> > >> Anyway, these are some of the obstacles.  Not saying it isn't
> > >> possible, it's just a lot of effort.
> > >>
> > >> Thanks,
> > >> -Aaron
> > >>
> > >> _______________________________________________
> > >> cctbxbb mailing list
> > >> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
> > >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mailman_listinfo_cctbxbb&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=W95a_DpmVSjvqMSzHgvTQ4gWgnlGBm0aOrAOebMg2Lo&e=
> > >>
> > >> _______________________________________________
> > >> cctbxbb mailing list
> > >> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
> > >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mailman_listinfo_cctbxbb&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=W95a_DpmVSjvqMSzHgvTQ4gWgnlGBm0aOrAOebMg2Lo&e=
> > >>
> > >> _______________________________________________
> > >> cctbxbb mailing list
> > >> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
> > >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mailman_listinfo_cctbxbb&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=W95a_DpmVSjvqMSzHgvTQ4gWgnlGBm0aOrAOebMg2Lo&e=
> > >>
> > >>
> > >> --
> > >> This e-mail and any attachments may contain confidential, copyright
> > >> and or privileged material, and are for the use of the intended
> > >> addressee only. If you are not the intended addressee or an authorised
> > >> recipient of the addressee please notify us of receipt by returning
> > >> the e-mail and do not use, copy, retain, distribute or disclose the
> > >> information in or attached to the e-mail.
> > >> Any opinions expressed within this e-mail are those of the individual
> > >> and not necessarily of Diamond Light Source Ltd.
> > >> Diamond Light Source Ltd. cannot guarantee that this e-mail or any
> > >> attachments are free from viruses and we cannot accept liability for
> > >> any damage which you may sustain as a result of software viruses which
> > >> may be transmitted in or with the message.
> > >> Diamond Light Source Limited (company no. 4375679). Registered in
> > >> England and Wales with its registered office at Diamond House, Harwell
> > >> Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United
> > >> Kingdom
> > >>
> > >> _______________________________________________
> > >> cctbxbb mailing list
> > >> cctbxbb at phenix-online.org
> > >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mailman_listinfo_cctbxbb&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=W95a_DpmVSjvqMSzHgvTQ4gWgnlGBm0aOrAOebMg2Lo&e=
> > >
> > >
> > > _______________________________________________
> > > cctbxbb mailing list
> > > cctbxbb at phenix-online.org
> > >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mailman_listinfo_cctbxbb&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=W95a_DpmVSjvqMSzHgvTQ4gWgnlGBm0aOrAOebMg2Lo&e=
> >
> >
> > _______________________________________________
> > cctbxbb mailing list
> > cctbxbb at phenix-online.org
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mailman_listinfo_cctbxbb&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=W95a_DpmVSjvqMSzHgvTQ4gWgnlGBm0aOrAOebMg2Lo&e=
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_pipermail_cctbxbb_attachments_20190823_f9e70f74_attachment-2D0001.htm&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=4E4kw3FmxR0v3-xqf8pMDH5RStsuxQYKsBFjWG2zBe0&e=
> >
>
> ------------------------------
>
> Message: 3
> Date: Fri, 23 Aug 2019 18:21:09 -0700
> From: Billy Poon <BKPoon at lbl.gov>
> To: cctbx mailing list <cctbxbb at phenix-online.org>
> Subject: Re: [cctbxbb] interactive cctbx.python backspace behavior (on
>         linux)
> Message-ID:
>         <
> CAHmYUcZrQhU3g_rDoHnS5gAERQxDnmZDdYwJHUyqzORMcQcP_w at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi Wolfram,
>
> Sorry for the issue, are you using a recent build of CCTBX? Can you try
> these steps?
>
> 1) cd <CCTBX directory>
> 2) rm -fr conda_base
> 3) python modules/cctbx_project/libtbx/auto_build/
> https://urldefense.proofpoint.com/v2/url?u=http-3A__bootstrap.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=EHieOoJlubwjTowwli2WmEdfxbWc4lfgK2UWBgjbwUQ&e=
> base
> --builder=phenix --use-conda
>
> It basically recreates the dependencies directory. It's using conda, so it
> should take less than 5 minutes. If you are using conda, you will need
> conda version 4.4 or greater.
>
> Let me know if that doesn't solve the problem. Thanks!
>
> --
> Billy K. Poon
> Research Scientist, Molecular Biophysics and Integrated Bioimaging
> Lawrence Berkeley National Laboratory
> 1 Cyclotron Road, M/S 33R0345
> Berkeley, CA 94720
> Tel: (510) 486-5709
> Fax: (510) 486-5909
> Web:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__phenix-2Donline.org&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=4z9DrBeWDv4KyPNyfSgrmvwWzefUCK69jc4d5Yyps7w&e=
>
>
> On Fri, Aug 23, 2019 at 12:55 PM wtempel <wtempel at gmail.com> wrote:
>
> > Hi,
> > Currently, cctbx.python behaves like described in this stackoverflow
> post:
> >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_893053_seeing-2Descape-2Dcharacters-2Dwhen-2Dpressing-2Dthe-2Darrow-2Dkeys-2Din-2Dpython-2Dshell&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=Q7VWA0EW6aVF6MGxZEXhez1kRpB0sqtTuvu5F51DxqI&e=
> .
> > I am missing the convenience of backspace delete, which is available on
> > /usr/bin/python interactive sessions. Could that feature be added during
> > compilation of the pre-built cctbx distributions?
> > Regards.
> > Wolfram Tempel
> > _______________________________________________
> > cctbxbb mailing list
> > cctbxbb at phenix-online.org
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mailman_listinfo_cctbxbb&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=W95a_DpmVSjvqMSzHgvTQ4gWgnlGBm0aOrAOebMg2Lo&e=
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_pipermail_cctbxbb_attachments_20190823_9a60b561_attachment-2D0001.htm&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=dXlDOw3xhBrjJmm_Qm17RG5HVmjJKimTmOkBuT5S8tk&e=
> >
>
> ------------------------------
>
> Message: 4
> Date: Fri, 23 Aug 2019 18:22:01 -0700
> From: Billy Poon <BKPoon at lbl.gov>
> To: cctbx mailing list <cctbxbb at phenix-online.org>
> Subject: Re: [cctbxbb] interactive cctbx.python backspace behavior (on
>         linux)
> Message-ID:
>         <CAHmYUcaSDZHv7JerH98bMfcXwfsmwpsZ6zao-J1WdKNvy=
> GUmQ at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi Wolfram
>
> Sorry, step 3 should be,
>
> 3) python modules/cctbx_project/libtbx/auto_build/
> https://urldefense.proofpoint.com/v2/url?u=http-3A__bootstrap.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=EHieOoJlubwjTowwli2WmEdfxbWc4lfgK2UWBgjbwUQ&e=
> base
> --use-conda
>
> You do not need the --builder argument.
>
> --
> Billy K. Poon
> Research Scientist, Molecular Biophysics and Integrated Bioimaging
> Lawrence Berkeley National Laboratory
> 1 Cyclotron Road, M/S 33R0345
> Berkeley, CA 94720
> Tel: (510) 486-5709
> Fax: (510) 486-5909
> Web:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__phenix-2Donline.org&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=4z9DrBeWDv4KyPNyfSgrmvwWzefUCK69jc4d5Yyps7w&e=
>
>
> On Fri, Aug 23, 2019 at 6:21 PM Billy Poon <BKPoon at lbl.gov> wrote:
>
> > Hi Wolfram,
> >
> > Sorry for the issue, are you using a recent build of CCTBX? Can you try
> > these steps?
> >
> > 1) cd <CCTBX directory>
> > 2) rm -fr conda_base
> > 3) python modules/cctbx_project/libtbx/auto_build/
> https://urldefense.proofpoint.com/v2/url?u=http-3A__bootstrap.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=EHieOoJlubwjTowwli2WmEdfxbWc4lfgK2UWBgjbwUQ&e=
> base
> > --builder=phenix --use-conda
> >
> > It basically recreates the dependencies directory. It's using conda, so
> it
> > should take less than 5 minutes. If you are using conda, you will need
> > conda version 4.4 or greater.
> >
> > Let me know if that doesn't solve the problem. Thanks!
> >
> > --
> > Billy K. Poon
> > Research Scientist, Molecular Biophysics and Integrated Bioimaging
> > Lawrence Berkeley National Laboratory
> > 1 Cyclotron Road, M/S 33R0345
> > Berkeley, CA 94720
> > Tel: (510) 486-5709
> > Fax: (510) 486-5909
> > Web:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__phenix-2Donline.org&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=4z9DrBeWDv4KyPNyfSgrmvwWzefUCK69jc4d5Yyps7w&e=
> >
> >
> > On Fri, Aug 23, 2019 at 12:55 PM wtempel <wtempel at gmail.com> wrote:
> >
> >> Hi,
> >> Currently, cctbx.python behaves like described in this stackoverflow
> post:
> >>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_893053_seeing-2Descape-2Dcharacters-2Dwhen-2Dpressing-2Dthe-2Darrow-2Dkeys-2Din-2Dpython-2Dshell&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=Q7VWA0EW6aVF6MGxZEXhez1kRpB0sqtTuvu5F51DxqI&e=
> .
> >> I am missing the convenience of backspace delete, which is available on
> >> /usr/bin/python interactive sessions. Could that feature be added during
> >> compilation of the pre-built cctbx distributions?
> >> Regards.
> >> Wolfram Tempel
> >> _______________________________________________
> >> cctbxbb mailing list
> >> cctbxbb at phenix-online.org
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mailman_listinfo_cctbxbb&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=W95a_DpmVSjvqMSzHgvTQ4gWgnlGBm0aOrAOebMg2Lo&e=
> >>
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_pipermail_cctbxbb_attachments_20190823_41d5ae8d_attachment-2D0001.htm&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=_III0sQBEn0hbJHyV_oiAkMI3z-cH_HhHAPoa8mWWMQ&e=
> >
>
> ------------------------------
>
> Message: 5
> Date: Sat, 24 Aug 2019 05:40:36 +0100
> From: Tristan Croll <tic20 at cam.ac.uk>
> To: cctbx mailing list <cctbxbb at phenix-online.org>
> Subject: Re: [cctbxbb] some thoughts on cctbx and pip
> Message-ID: <EB7E47C0-AE1F-4953-8B99-F3558EB98889 at cam.ac.uk>
> Content-Type: text/plain; charset="utf-8"
>
> Hi Billy,
>
> Sorry, I wasn?t clear: ChimeraX can install packages *from* Pip, but isn?t
> installable that way itself. It?s built and distributed as a self-contained
> application. Would certainly make my life easier if they were to migrate to
> Conda, but I don?t see that happening in the near future.
>
> Best regards,
>
> Tristan
>
>
>
>
>
>
> > On 24 Aug 2019, at 02:16, Billy Poon <BKPoon at lbl.gov> wrote:
> >
> > Hi all,
> >
> > Sorry for the delayed response, I was getting the message digests to
> test some bulletin board settings, so I wasn't getting the individual
> messages for replying.
> >
> > For a brief update on conda,
> >
> > 1) External Boost libraries can now be used to build on all platforms
> (linux, macOS, Windows) and I'm testing updated environments with Boost
> 1.68 and 1.70. This is for Python 2.7, 3.6, and some initial testing on
> 3.7. Once 3.7 is ready, Azure Pipelines will be updated to also build 3.7.
> I'll do the updated dependencies as a pull request since there will be
> changes that stop the downloading of modules/boost. Mixing Boost versions
> can cause seg faults or unexpected behavior. I also did some brief testing
> with 1.65 since that is the minimum version that the latest versions of
> CUDA require.
> > 2) The next step, like Aaron mentioned, is to get our build system to
> put things in more standard locations (e.g. dispatchers/binaries in
> $PREFIX/bin, shared libraries in $PREFIX/lib, Python source/extension
> modules in $PREFIX/lib/python<version>/site-packages) as a "make install"
> command. conda on Windows has a similar hierarchy that's specific to conda,
> so for Windows, this will be conda-specific.
> > 3) Update the conda_compiler branch (
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_cctbx_cctbx-5Fproject_tree_conda-5Fcompiler&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=pZLc23N2JvLGC3xcVjPt7-hTkrGBn7YKnKyAcudUNpo&e=
> ). This lets the build process use the standard compilers in conda's build
> system.
> >
> > Then a CCTBX conda package can be built and added to the conda-forge
> channel. The conda environments for building CCTBX are already based on
> conda-forge dependencies.
> >
> > 4) With the ability to build a conda package, conda constructor (
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_conda_constructor&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=mNtfU28QvccVCRK7pVUY96JCWFONmw1-Db6zZAmph_o&e=
> ) can be used for building our installers. This would be the more official
> way of packaging CCTBX with conda dependencies as a stand-alone installer.
> One of the roadblocks to using conda constructor was supporting noarch
> packages. It looks like that has been fixed (
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.anaconda.com_announcing-2Dthe-2Danaconda-2D2019-2D07-2Drelease_&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=9dHst_9qvOudkeSlaJVQvYLS_XATq38_Y60BX4jwcGg&e=
> ).
> >
> > So a goal for this work with conda is to make CCTBX-based programs
> installable as conda packages so that installing and running the software
> is consistent across platforms and operating system versions while also
> making it easier for users to add their tools to the software stack.
> >
> > We do have a Phenix release coming up, so I won't be able to spend much
> effort on this until later in September.
> >
> > Tristan,
> >
> > If ChimeraX is installable via pip, it can be installed into a conda
> environment. And once 2) is complete, you would be able to access CCTBX
> from ChimeraX. Also, this would make installing ChimeraX pretty easy for
> Phenix. Interestingly, the latest open source PyMOL can be built with
> dependencies from Schrodinger's conda channel, so it should be possible to
> build a conda package for PyMOL that integrates with our conda environment.
> >
> > --
> > Billy K. Poon
> > Research Scientist, Molecular Biophysics and Integrated Bioimaging
> > Lawrence Berkeley National Laboratory
> > 1 Cyclotron Road, M/S 33R0345
> > Berkeley, CA 94720
> > Tel: (510) 486-5709
> > Fax: (510) 486-5909
> > Web:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__phenix-2Donline.org&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=4z9DrBeWDv4KyPNyfSgrmvwWzefUCK69jc4d5Yyps7w&e=
> >
> >
> >> On Fri, Aug 23, 2019 at 10:03 AM Tristan Croll <tic20 at cam.ac.uk> wrote:
> >> To add my two cents on this: probably the second-most common question
> >> I've had about ISOLDE's implementation is, "why didn't you use CCTBX?".
> >> The honest answer to that is, "I didn't know how."
> >>
> >> Still don't, really - although the current developments are rather
> >> promising. The problem I've faced is that CCTBX was designed as its own
> >> self-contained Python (2.7, until very recently) environment, with its
> >> own interpreter and a lot of very specific environment setup. Meanwhile
> >> I'm developing ISOLDE in ChimeraX, which is *also* its own
> >> self-contained Python (3.7) environment. To plug one into the other in
> >> that form... well, I don't think I'm a good enough programmer to really
> >> know where to start.
> >>
> >> The move to Conda and a more modular CCTBX architecture should make a
> >> lot more possible in that direction. Pip would be even better for me
> >> personally (ChimeraX can install directly from the PyPI, but doesn't
> >> interact with Conda) - but I understand pretty well the substantial
> >> challenge that would amount to (not least being that the PyPI imposes a
> >> limit - around 100MB from memory? - on the size of an individual
> >> package).
> >>
> >> Best regards,
> >>
> >> Tristan
> >>
> >> On 2019-08-23 09:28, Luc Bourhis wrote:
> >> > Hi Graeme,
> >> >
> >> > Yes, I know. But ?black" is a program doing a very particular task
> >> > (code formatting from the top of my head). Requiring to use a wrapper
> >> > for python itself is another level. But ok, I think I am mellowing to
> >> > the idea after all! Talking with people around me, and extrapolating,
> >> > I would bet that, right now, a great majority of people interested by
> >> > cctbx in pip have already used the cctbx, so they know about the
> >> > Python wrapper, and they would not be too sanguine about that. My
> >> > concern is for the future, when pip will be the first time some people
> >> > use cctbx. Big fat warning notices on PyPI page and a better error
> >> > message when cctbx fails because LIBTBX_BUILD is not set would be
> >> > needed but that could be all right.
> >> >
> >> > If we do a pip installer, we should aim at a minimal install: cctbx,
> >> > iotbx and their dependencies, and that?s it.
> >> >
> >> > Best wishes,
> >> >
> >> > Luc
> >> >
> >> >
> >> >> On 23 Aug 2019, at 07:17, Graeme.Winter at Diamond.ac.uk
> >> >> <Graeme.Winter at diamond.ac.uk> wrote:
> >> >>
> >> >> Without discussing the merits of this or whether we _choose_ to make
> >> >> the move to supporting PIP, I am certain it would be _possible_ -
> many
> >> >> other packages make dispatcher scripts when you pip install them e.g.
> >> >>
> >> >> Silver-Surfer rescale_f2 :) $ which black; cat $(which black)
> >> >> /Library/Frameworks/Python.framework/Versions/3.6/bin/black
> >> >> #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6
> >> >>
> >> >> # -*- coding: utf-8 -*-
> >> >> import re
> >> >> import sys
> >> >>
> >> >> from black import main
> >> >>
> >> >> if __name__ == '__main__':
> >> >>    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
> >> >>    sys.exit(main())
> >> >>
> >> >> So we _could_ work around the absence of LIBTBX_BUILD etc. in the
> >> >> system. Whether or not we elect to do the work is a different
> >> >> question, and it seems clear that here are very mixed opinions on
> >> >> this.
> >> >>
> >> >> Best wishes Graeme
> >> >>
> >> >>
> >> >> On 23 Aug 2019, at 01:21, Luc Bourhis
> >> >> <luc_j_bourhis at mac.com<mailto:luc_j_bourhis at mac.com>> wrote:
> >> >>
> >> >> Hi,
> >> >>
> >> >> Even if we managed to ship our the boost dynamic libraries with pip,
> >> >> it would still not be pip-like, as we would still need our python
> >> >> wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal pip packages
> >> >> work with the standard python exe. LD_LIBRARY_PATH, we could get
> >> >> around that by changing the way we compile, using -Wl,-R, which is
> the
> >> >> runtime equivalent of build time -L. That?s a significant change that
> >> >> would need to be tested. But there is no way around setting
> >> >> LIBTBX_BUILD right now. Leaving that to the user is horrible. Perhaps
> >> >> there is a way to hack libtbx/
> https://urldefense.proofpoint.com/v2/url?u=http-3A__env-5Fconfig.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=5tRgn0yDOi7u9JsSVXIMM6D5NzBFMtAC8OqFFrm-0qM&e=
> so that we can hardwire
> >> >> LIBTBX_BUILD in there when pip installs?
> >> >>
> >> >> Best wishes,
> >> >>
> >> >> Luc
> >> >>
> >> >>
> >> >> On 16 Aug 2019, at 22:47, Luc Bourhis
> >> >> <luc_j_bourhis at mac.com<mailto:luc_j_bourhis at mac.com>> wrote:
> >> >>
> >> >> Hi,
> >> >>
> >> >> I did look into that many years ago, and even toyed with building a
> >> >> pip installer. What stopped me is the exact conclusion you reached
> >> >> too: the user would not have the pip experience he expects. You are
> >> >> right that it is a lot of effort but is it worth it? Considering that
> >> >> remark, I don?t think so. Now, Conda was created specifically to go
> >> >> beyond pip pure-python-only support. Since cctbx has garnered support
> >> >> for Conda, the best avenue imho is to go the extra length to have a
> >> >> package on
> https://urldefense.proofpoint.com/v2/url?u=http-3A__Anaconda.org&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=LTXosS3SYkN2lzASpSd1A47FoDWWfKyKkJVkQknHbFI&e=
> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__anaconda.org_&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=xEUexfDy2KPfHxGictcK-Rnxag5jmm5RjzAKoJRKMFQ&e=
> >, and then to advertise
> >> >> it hard to every potential user out there.
> >> >>
> >> >> Best wishes,
> >> >>
> >> >> Luc
> >> >>
> >> >>
> >> >> On 16 Aug 2019, at 21:45, Aaron Brewster
> >> >> <asbrewster at lbl.gov<mailto:asbrewster at lbl.gov>> wrote:
> >> >>
> >> >> Hi, to avoid clouding Dorothee's documentation email thread, which I
> >> >> think is a highly useful enterprise, here's some thoughts about
> >> >> putting cctbx into pip.  Pip doesn't install non-python dependencies
> >> >> well.  I don't think boost is available as a package on pip (at least
> >> >> the package version we use).  wxPython4 isn't portable through pip
> >> >> (
> https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.wxpython.org_How-2520to-2520install-2520wxPython-23Installing-5FwxPython-2DPhoenix-5Fusing-5Fpip&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=3S4Ul0MLjWsEg9-Paov_bPhLYc9TAkgSSTZqHkyuqo8&e=
> ).
> >> >>  MPI libraries are system dependent.  If cctbx were a pure python
> >> >> package, pip would be fine, but cctbx is not.
> >> >>
> >> >> All that said, we could build a manylinux1 version of cctbx and
> upload
> >> >> it to PyPi (I'm just learning about this).  For a pip package to be
> >> >> portable (which is a requirement for cctbx), it needs to conform to
> >> >> PEP513, the manylinux1 standard
> >> >> (
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.python.org_dev_peps_pep-2D0513_&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=HwLo0JfnP5hTlTNAyDpSgaD0IhQVgEhmEZzBa12e78o&e=
> ).  For example, numpy is
> >> >> built according to this standard (see
> >> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__pypi.org_project_numpy_-23files&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=5aDA0n5NQ0qoMHMXpj3PBRPfB9GGnhlrZheurjjBmE4&e=
> , where you'll see the manylinux1
> >> >> wheel).  Note, the manylinux1 standard is built with Centos 5.11
> which
> >> >> we no longer support.
> >> >>
> >> >> There is also a manylinux2010 standard, which is based on Centos 6
> >> >> (
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.python.org_dev_peps_pep-2D0571_&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=lJpz6hVmDG53lwLTol5NkNDKUEwVeldbQytTkIBzJu0&e=
> ).  This is likely a more
> >> >> attainable target (note though by default C++11 is not supported on
> >> >> Centos 6).
> >> >>
> >> >> If we built a manylinuxX version of cctbx and uploaded it to PyPi,
> the
> >> >> user would need all the non-python dependencies.  There's no way to
> >> >> specify these in pip.  For example, cctbx requires boost 1.63 or
> >> >> better.  The user will need to have it in a place their python can
> >> >> find it, or we could package it ourselves and supply it, similar to
> >> >> how the pip h5py package now comes with an hd5f library, or how the
> >> >> pip numpy package includes an openblas library.  We'd have to do the
> >> >> same for any packages we depend on that aren't on pip using the
> >> >> manylinux standards, such as wxPython4.
> >> >>
> >> >> Further, we need to think about how dials and other cctbx-based
> >> >> packages interact.  If pip install cctbx is set up, how does pip
> >> >> install dials work, such that any dials shared libraries can find the
> >> >> cctbx libraries?  Can shared libraries from one pip package link
> >> >> against libraries in another pip package?  Would each package need to
> >> >> supply its own boost?  Possibly this is well understood in the pip
> >> >> field, but not by me :)
> >> >>
> >> >> Finally, there's the option of providing a source pip package.  This
> >> >> would require the full compiler toolchain for any given platform
> >> >> (macOS, linux, windows).  These are likely available for developers,
> >> >> but not for general users.
> >> >>
> >> >> Anyway, these are some of the obstacles.  Not saying it isn't
> >> >> possible, it's just a lot of effort.
> >> >>
> >> >> Thanks,
> >> >> -Aaron
> >> >>
> >> >> _______________________________________________
> >> >> cctbxbb mailing list
> >> >> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
> >> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mailman_listinfo_cctbxbb&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=W95a_DpmVSjvqMSzHgvTQ4gWgnlGBm0aOrAOebMg2Lo&e=
> >> >>
> >> >> _______________________________________________
> >> >> cctbxbb mailing list
> >> >> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
> >> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mailman_listinfo_cctbxbb&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=W95a_DpmVSjvqMSzHgvTQ4gWgnlGBm0aOrAOebMg2Lo&e=
> >> >>
> >> >> _______________________________________________
> >> >> cctbxbb mailing list
> >> >> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
> >> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mailman_listinfo_cctbxbb&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=W95a_DpmVSjvqMSzHgvTQ4gWgnlGBm0aOrAOebMg2Lo&e=
> >> >>
> >> >>
> >> >> --
> >> >> This e-mail and any attachments may contain confidential, copyright
> >> >> and or privileged material, and are for the use of the intended
> >> >> addressee only. If you are not the intended addressee or an
> authorised
> >> >> recipient of the addressee please notify us of receipt by returning
> >> >> the e-mail and do not use, copy, retain, distribute or disclose the
> >> >> information in or attached to the e-mail.
> >> >> Any opinions expressed within this e-mail are those of the individual
> >> >> and not necessarily of Diamond Light Source Ltd.
> >> >> Diamond Light Source Ltd. cannot guarantee that this e-mail or any
> >> >> attachments are free from viruses and we cannot accept liability for
> >> >> any damage which you may sustain as a result of software viruses
> which
> >> >> may be transmitted in or with the message.
> >> >> Diamond Light Source Limited (company no. 4375679). Registered in
> >> >> England and Wales with its registered office at Diamond House,
> Harwell
> >> >> Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United
> >> >> Kingdom
> >> >>
> >> >> _______________________________________________
> >> >> cctbxbb mailing list
> >> >> cctbxbb at phenix-online.org
> >> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mailman_listinfo_cctbxbb&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=W95a_DpmVSjvqMSzHgvTQ4gWgnlGBm0aOrAOebMg2Lo&e=
> >> >
> >> >
> >> > _______________________________________________
> >> > cctbxbb mailing list
> >> > cctbxbb at phenix-online.org
> >> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mailman_listinfo_cctbxbb&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=W95a_DpmVSjvqMSzHgvTQ4gWgnlGBm0aOrAOebMg2Lo&e=
> >>
> >>
> >> _______________________________________________
> >> cctbxbb mailing list
> >> cctbxbb at phenix-online.org
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mailman_listinfo_cctbxbb&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=W95a_DpmVSjvqMSzHgvTQ4gWgnlGBm0aOrAOebMg2Lo&e=
> > _______________________________________________
> > cctbxbb mailing list
> > cctbxbb at phenix-online.org
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mailman_listinfo_cctbxbb&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=W95a_DpmVSjvqMSzHgvTQ4gWgnlGBm0aOrAOebMg2Lo&e=
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_pipermail_cctbxbb_attachments_20190824_180bc8c1_attachment-2D0001.htm&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=UwdEyNX7J-SfW8-kxGopqMgiuaRIA26YeMW2oej4rWE&e=
> >
>
> ------------------------------
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org
>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mailman_listinfo_cctbxbb&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=YAow0nkXkRwLy6SM_tAub2NSjj6LBr14R8z8HmBv6vI&s=W95a_DpmVSjvqMSzHgvTQ4gWgnlGBm0aOrAOebMg2Lo&e=
>
>
> End of cctbxbb Digest, Vol 125, Issue 22
> ****************************************
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://phenix-online.org/pipermail/cctbxbb/attachments/20190824/1c17f1a4/attachment.htm>


More information about the cctbxbb mailing list