Hi James,

fix-division ============ For all .py files in the present directory, it will add the “from future” to any files that need it, recursively. The new line will go after any “#!” line or module level docstrings and before any other line. Variable whitespace is handled gracefully. This allows any document generators to recognize the module-level docstrings.

libtbx.add_from_future_import_division path/to/some/directory does already do that. That's the script I wrote in order to mass insert "from __future__ import division" everywhere in the cctbx a few years back. Best wishes, Luc

On Wed, Jan 8, 2014 at 4:00 PM, James Stroud wrote:

...

/usr/bin/python /path/to/libtbx/configure.py --old_division cctbx

which will give you cleaner dispatchers. I'm pretty sure the rest of CCTBX should still behave sensibly, since we're pretty militant about including the __future__ statement in these files. Is that sufficiently usable for the short term?

That works fine for a developer, but it doesn’t help much with users, who shouldn’t be expected to create a custom build of CCTBX.

True - but our assumption (possibly incorrect) has been that there aren't many people downloading binary builds of CCTBX and then combining them with other Python modules. The majority of CCTBX "users", as far as I know, are either people actively developing code based on or within CCTBX, or the end users of Phenix, CCP4, etc. Attached is a tar file (fixit.tar) of three python scripts that should

solve this problem easily:

All of the code in CCTBX is already patched (I just fixed the one or two recent additions that weren't). The problem is all of the code in Phenix, which we would like to ensure has consistent behavior - this was the primary reason for adding "-Qnew". The maintenance overhead is already enormous and I don't have either the time or authority to micromanage my collaborators. (Nor does anyone else, for that matter.) In this case I think the current behavior is risky enough to justify a one-time intervention, but I'm not going to do anything without first discussing it with the Powers That Be. -Nat

On 9 Jan 2014, at 02:00, James Stroud wrote:

The way to promote widespread use is to make CCTBX importable just like any other package, like matplotlib, scipy, numpy, etc. I think CCTBX is the only major package I know of that insulates itself like this and requires its own “dispatcher”.

The dispatcher's main job is to set PYTHONPATH, DYLD_LIBRARY_PATH on MacOS X, LD_LIBRARY_PATH on Linux and PATH on Windows, as well as LIBTBX_BUILD. Thus it can be argued that it is a by-product of the way the cctbx installs itself on a system. By that I mean that if we installed it with distutils like your typical Python package, then /usr/bin/python would be able to run any cctbx-based script because the cctbx shared libraries and Python modules would be where /usr/bin/python could find them, and we would just need to decide on where to put libtbx_env inside site_packages. For historical reasons, this is not the path that was taken. It can certainly be rectified but as Nat said, Phenix takes precedence here, since the cctbx effort ultimately relies on the funding of Phenix. As long as the Phenix people aren't comfortable with removing -Qnew or with making the installation rely on distutils, then so be it. Luc

Hi All, On the distutils thing: I agree completely with Nat that in terms of developing cctbx it would be a monumental pain to have to be running python setup install all the while... however for someone who wants to *use* cctbx in the same way as they use matplotlib and so on (there are a lot of people like this here at Diamond - physical scientists mostly) it would be a great help to be able to install cctbx in the way that they are used to. So what would be really cool would be to be able to use distutils methods to *install* cctbx in the usual pythonic way, but to keep up with the current mechanism for *working on* cctbx - though the Scons build mechanism may not help this... Could it be possible to run the two side-by-side? I have encountered people for whom this is a blocker for example they want to just add cctbx to their enthought python installation or something... (agreed though about the Phenix caveats: this is not useful for Phenix and for the large part that is where the $$ come from for cctbx so ...) I'll ask around here to see if anyone is interested in looking into this... Thanks & cheerio, Graeme From: [email protected] [mailto:[email protected]] On Behalf Of Nathaniel Echols Sent: 09 January 2014 01:21 To: cctbx mailing list Subject: Re: [cctbxbb] cctbx.python unnecessarily unfriendly On Wed, Jan 8, 2014 at 5:05 PM, Luc Bourhis mailto:[email protected]> wrote: The dispatcher's main job is to set PYTHONPATH, DYLD_LIBRARY_PATH on MacOS X, LD_LIBRARY_PATH on Linux and PATH on Windows, as well as LIBTBX_BUILD. Thus it can be argued that it is a by-product of the way the cctbx installs itself on a system. Actually, a major justification for the dispatchers was to avoid the "pollution" of the environment in ways that might interfere with other packages. (This is why the default behavior is also to completely wipe any existing values of LD_LIBRARY_PATH, etc.) Again, this mainly makes deployment of Phenix easier. By that I mean that if we installed it with distutils like your typical Python package, then /usr/bin/python would be able to run any cctbx-based script because the cctbx shared libraries and Python modules would be where /usr/bin/python could find them, and we would just need to decide on where to put libtbx_env inside site_packages. The reason why we don't do this is simply to speed development - there's no distinction between the working code (i.e. SVN checkout) and runtime code. Otherwise we might have to run "python setup.py install" every time we want to test even the most minor change. There are almost certainly other ways to avoid this requirement, but that was the logic at the time. For historical reasons, this is not the path that was taken. It can certainly be rectified but as Nat said, Phenix takes precedence here, since the cctbx effort ultimately relies on the funding of Phenix. As long as the Phenix people aren't comfortable with removing -Qnew or with making the installation rely on distutils, then so be it. It's not a matter of comfort: the limitation is what we have time for (or money, which is basically the same thing). Removing -Qnew is probably easy enough to be justified if it makes CCTBX as a whole more usable. I don't think anyone would argue against converting to use distutils on technical grounds; we're simply too busy. I can't speak for the other developers, but if someone else outside our group were willing to do this we would probably not complain and might even encourage it, as long as it didn't interrupt our day jobs. -Nat -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

Le 09/01/2014 09:51, Luc Bourhis a écrit :

[...] And you know what: the Debian people were working on this about one year ago, as part of creating a proper deb package for the cctbx. They proposed a series of patch to achieve that: if you browse the history of this mailing list, you will find the discussions we had. The Debian developer were Radostan Riedel, Baptiste Carvello and Frédéric-Emmanuel Picca. The last of the tree works at Soleil and his concerns were very similar to those you expressed. He even invited me to visit him, which I did. Unfortunately, neither I nor Nat have followed this project.

Hi Luc, hi all, the project got stuck due to lack of time. The last commits were made in February last year. Speaking for myself, sticking around on the mailing lists is as much time I can put in it these days (sorry about that). Cheers, Baptiste

Hi Marcin, from what I remember, distutils was not the difficult problem. The setup.py was mostly a thin wrapper around Scons. As you say, it's just another build system. The main reason we use it is, that the Debian packaging tools interface well with it. The difficulty was in making sure that cctbx would find its different bits (python modules, shared libraries, runtime and test data) in the different places where Debian traditionally install them (instead of the source + build directories). We tried to make this work even when using the system python. This involved some delicate surgery in the libtbx environment file. For this task, a python-written build system is a big plus, because it enables us to import the libtbx modules. Cheers, Baptiste Le 09/01/2014 20:04, Marcin Wojdyr a écrit :

In CCP4 we have the same problem that Linux distros have with cctbx. We're using an ad-hoc shell script that copies files around as a substitute for "make install" (or python setup.py install / scons install) functionality.

Scons like any build system supports installation. From what I just googled the functions for this are named Install() and InstallAs(). Maybe it would be easier to use them instead of distutils?

Marcin

On 9 January 2014 08:51, Luc Bourhis wrote:

...

Hi Graeme,

So what would be really cool would be to be able to use distutils methods to *install* cctbx in the usual pythonic way, but to keep up with the current mechanism for *working on* cctbx

Actually, thinking about it a bit more, it seems that installing the cctbx with distutils is orthogonal with the way we do it now. I mean, we could keep using the Python wrapper to make the build + source (non-)installation work but then have a script that would create a distutils-based distro. And you know what: the Debian people were working on this about one year ago, as part of creating a proper deb package for the cctbx. They proposed a series of patch to achieve that: if you browse the history of this mailing list, you will find the discussions we had. The Debian developer were Radostan Riedel, Baptiste Carvello and Frédéric-Emmanuel Picca. The last of the tree works at Soleil and his concerns were very similar to those you expressed. He even invited me to visit him, which I did. Unfortunately, neither I nor Nat have followed this project.

Best wishes,

Luc

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb