[cctbxbb] Tidy up CCTBX

Fri Jan 26 17:05:20 PST 2018

Hi Richard,

Making dxtbx an optional dependency (as Nick just did, see his commit)
doesn't help us.  If iotbx is configured, then dxtbx will be compiled since
dxtbx is always available in cctbx_project.  dxtbx would only truly be
optional if it were moved out of cctbx_project, which we aren't willing to
do.

Again all of our distributions (phenix, labelit, dials) include cbflib and
hdf5.  If we made these packages optional in dxtbx, then any application
who wants to use cctbx without those packages would be able to.  Those
applications won't be reading image data anyway (otherwise they would solve
the hdf5/cbflib problem themselves).

-Aaron

On Fri, Jan 26, 2018 at 3:08 PM, <richard.gildea at diamond.ac.uk> wrote:

> Hi Aaron,
>
> As far as I understand it the issue is not being able to use iotbx to read
> structure model/reflection files without an enforced dependency on
> hdf5/cbflib via dxtbx which are not required in order to do such
> operations. Only a relatively small part of iotbx (i.e. iotbx.detectors)
> seems to be dependent on dxtbx, so perhaps the quickest and easiest
> solution would be to make iotbx optionally dependent on dxtbx, and disable
> the iotbx.detectors module if hdf5 or cbflib aren't available. In contrast,
> making dxtbx optionally dependent on hdf5 and cbflib sounds like it would
> be more involved, and the functionality of dxtbx would be dramatically
> reduced without the ability to read hdf5/cbf files (probably the majority
> of diffraction images currently/going forward).
>
> Cheers,
>
> Richard
>
> Dr Richard Gildea
> Data Analysis Scientist
> Tel: +441235 77 8078
>
> Diamond Light Source Ltd.
> Diamond House
> Harwell Science & Innovation Campus
> Didcot
> Oxfordshire
> OX11 0DE
> ________________________________
> From: cctbxbb-bounces at phenix-online.org [cctbxbb-bounces at phenix-online.org]
> on behalf of Aaron Brewster [asbrewster at lbl.gov]
> Sent: 26 January 2018 19:52
> To: cctbx mailing list
> Cc: rjr27 at cam.ac.uk
> Subject: Re: [cctbxbb] Tidy up CCTBX
>
> @Graeme, replying to your points
>
> " - I am not sure that these address the biggest problem i.e. iotbx
> depends_on dxtbx"
>
> Luc's original problem with iotbx depending on dxtbx was bringing in the
> hdf5 dependency.  ISo making hdf5 optional for dxtbx would be helpful.  The
> secondary complaint of a cyclical dependency of iotbx on dxtbx and dxtbx on
> iotbx is valid.  Merging iotbx and dxtbx has been discussed previously and
> is probably the right solution, but that would take a bit of doing...
>
> " - Will we end up getting bug reports where DIALS programs can’t read
> e.g. Eiger, CBF images?"
>
> This depends on distribution.  We are distributing builds of phenix,
> labelit and dials for linux and mac created by bootstrap that include hdf5
> and cbflib, so there are no issues there.  I don't know how or if Windows
> builds will be affected as I thought they were distributing HDF5 and cbflib
> too.  I believe this change only opens up possibilities for easier
> distribution for applications outside of phenix, labelit and dials.
> Please, if Rob or Luc want to correct me, feel free :)
>
> -Aaron
>
> On Fri, Jan 26, 2018 at 4:00 AM, <Graeme.Winter at diamond.ac.uk<mailto:
> Graeme.Winter at diamond.ac.uk>> wrote:
> This sounds like “pip install libtbx” “pip install scitbx”
>
> Since both of these are relatively static & self contained, would make
> sense
>
> ISTR there was a suggestion earlier in the thread to this effect, which
> perhaps got lost…
>
> Best wishes Graeme
>
> > On 25 Jan 2018, at 20:41, Tristan Croll <tic20 at cam.ac.uk<mailto:tic20@
> cam.ac.uk>> wrote:
> >
> > My 2 cents: when talking about creating PyPI CCTBX modules it may be
> worthwhile to start by thinking smaller. For instance, while planning out
> how to interface my (largely NumPy-based by necessity) application to
> PHENIX, it would be an enormous help if I could, say, 'pip install
> flex_array' and 'pip install phil'.
> >
> > On 2018-01-25 19:20, Aaron Brewster wrote:
> >> I've created the issues:
> >> https://github.com/cctbx/cctbx_project/issues/125 [6]
> >> https://github.com/cctbx/cctbx_project/issues/126 [7]
> >> -Aaron
> >> On Tue, Jan 23, 2018 at 10:50 AM, Paul Adams <pdadams at lbl.gov<mailto:
> pdadams at lbl.gov>> wrote:
> >>> Hi Aaron,
> >>> thanks for the email. I think your proposal to remove the compile
> >>> time dependencies by moving code to extension modules is great. I
> >>> think this would be greatly appreciated by Rob and others.
> >>> Fable, contrary to recent rumors, is actually often used by
> >>> others outside of the cctbx community. While it is true that we
> >>> wrote it for mainly one task, it turns out there are lots of people
> >>> with old f77 code they’d like to translate. Given that we don’t
> >>> seem to have any dependency or support issues with the code I’d
> >>> recommend it stays - making and supporting a separate package would
> >>> be too much.
> >>> Cheers,
> >>> Paul
> >>>> On Jan 23, 2018, at 9:53 AM, Aaron Brewster <asbrewster at lbl.gov
> <mailto:asbrewster at lbl.gov>>
> >>> wrote:
> >>>> Hi folks,
> >>>> There are lots of threads in this email. Pulling a couple of
> >>> them...
> >>>> • The LBL team runs the libtbx/phil/tst_interface.py test
> >>> nightly in buildbot as part of testing phenix, so the test is
> >>> exercised (and it passed last night). Whether it should be moved to
> >>> phenix_regression is another issue (I don't think it's necessary,
> >>> the has_module call seems clear to me).
> >>>> • I think there are many projects that shouldn't be moved
> >>> out of cctbx_project. IOTA, xfel, prime, simtbx, and dxtbx are all
> >>> under active development and are useful to many crystallographers.
> >>> cctbx is a toolbox after all. It just has a lot of tools.
> >>>> • That said, It may be worth having a discussion about
> >>> fable, and maybe one or two of the others. That can be done in
> >>> separate threads for each project?
> >>>> • I would like to break compile-time dependencies on
> >>> external packages in dxtbx as identified by Rob. If the C++ code in
> >>> dxtbx/iotbx that imports cbf.h and hdf5.h is moved to separate
> >>> boost-python extension modules (some of this is done already), then
> >>> the SConscript can be made smarter and compile only the libraries it
> >>> can find headers for. I'd be in favor of making two issues to track
> >>> this work, one for cbflib and one for hdf5. That would be a step in
> >>> the direction restoring the 'standaloneness' of the cctbx package.
> >>>> -Aaron
> >>>> On Tue, Jan 23, 2018 at 3:16 AM, <markus.gerstel at diamond.ac.uk<
> mailto:markus.gerstel at diamond.ac.uk>>
> >>> wrote:
> >>>> Hi Marcin
> >>>> Hmm, so in other words "his free / open source software cannot be
> >>> fully tested without agreeing to this license"?
> >>>> No, still doesn't make sense to me. It only raises more
> >>> questions. Such as: does Travis have a phenix license?
> >>>> Currently we don't run these tests at all - although we *should*
> >>> run all tests on libtbx - because we don't have phenix installed
> >>> during cctbx tests.
> >>>> Either the required test data should fully go into libtbx or the
> >>> test should fully go into phenix. Distributing a test in two parts
> >>> is no good to anyone.
> >>>> -Markus
> >>>> ________________________________________
> >>>> From: cctbxbb-bounces at phenix-online.org<mailto:cctbxbb-bounces@
> phenix-online.org>
> >>> [cctbxbb-bounces at phenix-online.org<mailto:cctbxbb-
> bounces at phenix-online.org>] on behalf of Marcin Wojdyr
> >>> [wojdyr at gmail.com<mailto:wojdyr at gmail.com>]
> >>>> Sent: Tuesday, January 23, 2018 10:55
> >>>> To: cctbx mailing list
> >>>> Cc: Randy Read
> >>>> Subject: Re: [cctbxbb] Tidy up CCTBX
> >>>> Hi Graeme,
> >>>> > Turns out that the assert that libtbx has no dependencies is
> >>> not strictly true:
> >>>> >
> >>>> > if (not libtbx.env.has_module(name="phenix")):
> >>>> > print "phenix module not available: skipping advanced
> >>> tests"
> >>>> > return
> >>>> This is optional dependency (cctbx optionally uses Phenix for
> >>> testing
> >>>> in various places) and doesn't cause problems.
> >>>> If you'd like to do disentangling, dxtbx-iotbx would be a good
> >>>> starting point as it really causes problems.
> >>>> _______________________________________________
> >>>> cctbxbb mailing list
> >>>> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb [1]
> >>>> --
> >>>> This e-mail and any attachments may contain confidential,
> >>> copyright and or privileged material, and are for the use of the
> >>> intended addressee only. If you are not the intended addressee or an
> >>> authorised recipient of the addressee please notify us of receipt by
> >>> returning the e-mail and do not use, copy, retain, distribute or
> >>> disclose the information in or attached to the e-mail.
> >>>> Any opinions expressed within this e-mail are those of the
> >>> individual and not necessarily of Diamond Light Source Ltd.
> >>>> Diamond Light Source Ltd. cannot guarantee that this e-mail or
> >>> any attachments are free from viruses and we cannot accept liability
> >>> for any damage which you may sustain as a result of software viruses
> >>> which may be transmitted in or with the message.
> >>>> Diamond Light Source Limited (company no. 4375679). Registered in
> >>> England and Wales with its registered office at Diamond House,
> >>> Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11
> >>> 0DE, United Kingdom
> >>>> _______________________________________________
> >>>> cctbxbb mailing list
> >>>> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb [1]
> >>>> _______________________________________________
> >>>> cctbxbb mailing list
> >>>> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb [1]
> >>> --
> >>> Paul Adams
> >>> Division Director, Molecular Biophysics & Integrated Bioimaging,
> >>> Lawrence Berkeley Lab
> >>> Division Deputy for Biosciences, Advanced Light Source, Lawrence
> >>> Berkeley Lab
> >>> Adjunct Professor, Department of Bioengineering, U.C. Berkeley
> >>> Vice President for Technology, the Joint BioEnergy Institute
> >>> Laboratory Research Manager, ENIGMA Science Focus Area
> >>> Building 33, Room 347
> >>> Building 978, Room 4126
> >>> Tel: 1-510-486-4225<tel:1-510-486-4225> [2], Fax: 1-510-486-5909<tel:
> 1-510-486-5909> [3]
> >>> http://cci.lbl.gov/paul [4]
> >>> Lawrence Berkeley Laboratory
> >>> 1 Cyclotron Road
> >>> BLDG 33R0345
> >>> Berkeley, CA 94720, USA.
> >>> Executive Assistant: Louise Benvenue [ LBenvenue at lbl.gov<mailto:LBenv
> enue at lbl.gov> ][
> >>> 1-510-495-2506<tel:1-510-495-2506> [5] ]
> >>> --
> >>> _______________________________________________
> >>> cctbxbb mailing list
> >>> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
> >>> http://phenix-online.org/mailman/listinfo/cctbxbb [1]
> >> Links:
> >> ------
> >> [1] http://phenix-online.org/mailman/listinfo/cctbxbb
> >> [2] tel:1-510-486-4225<tel:1-510-486-4225>
> >> [3] tel:1-510-486-5909<tel:1-510-486-5909>
> >> [4] http://cci.lbl.gov/paul
> >> [5] tel:%5B%201-510-495-2506
> >> [6] https://github.com/cctbx/cctbx_project/issues/125
> >> [7] https://github.com/cctbx/cctbx_project/issues/126
> >> _______________________________________________
> >> cctbxbb mailing list
> >> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
> >> http://phenix-online.org/mailman/listinfo/cctbxbb
> >
> >
> > _______________________________________________
> > cctbxbb mailing list
> > cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
> > http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
> --
> This e-mail and any attachments may contain confidential, copyright and or
> privileged material, and are for the use of the intended addressee only. If
> you are not the intended addressee or an authorised recipient of the
> addressee please notify us of receipt by returning the e-mail and do not
> use, copy, retain, distribute or disclose the information in or attached to
> the e-mail.
> Any opinions expressed within this e-mail are those of the individual and
> not necessarily of Diamond Light Source Ltd.
> Diamond Light Source Ltd. cannot guarantee that this e-mail or any
> attachments are free from viruses and we cannot accept liability for any
> damage which you may sustain as a result of software viruses which may be
> transmitted in or with the message.
> Diamond Light Source Limited (company no. 4375679). Registered in England
> and Wales with its registered office at Diamond House, Harwell Science and
> Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://phenix-online.org/pipermail/cctbxbb/attachments/20180126/f0bc0c0c/attachment-0001.htm>