[cctbxbb] Debian Package

Luc Bourhis luc_j_bourhis at mac.com
Wed Aug 8 18:35:11 PDT 2012


Hi gentlemen,

it turns out that I am based in the South of Paris and therefore Frédéric-Emmanuel and I agreed to meet sometimes later this month.

Nevertheless, if you allow me, I will answer some important remarks, responding to all your emails together, in a bit of a random fashion I am afraid.

First, facilitating the use of cctbx at the ESRF and Soleil is definitively a further motivation on our side to make this Debian packaging work.

Radostan Riedel wrote:
> OK to make that clear a little bit. A few patches are really only for packaging
> they can't and shouldn't go back upstream.

So I had missed something indeed! This is my first experience of a Linux package in the making, you see. Unfortunately, the both of Baptiste and Frederic-Emmanuel answered my criticisms thinking they were aimed at your design of the cctbx Debian package. On the contrary, they had to be understood in the context of a installation of the cctbx by hand or using the package that CCI provides. I am afraid we talked past each other here!

Nevertheless, my thanks to Frederic-Emmanuel and Baptiste to elaborate on those patches that were only meant for packaging and that I criticised most: it is still very useful for us to understand your rationales. Some specifics:

PICCA Frédéric-Emmanuel wrote:
> So yes for now apt-get install python-cctbx pull also openGL libraries
> (<30 Mo on my computer). The room on a server is no more a problem
> nowaday. we can find harddisk of 1To for less than 250euros. Indeed we
> should also split python-cctbx under finer grain packages, but is it
> worth the effort ? This can indeed be discussed.

Disk-space is not the problem I had in mind. Some people in charge of a web server are pretty paranoid about reliability and security and thus they fiercely try to minimise the number of packages installed. You may wish to consider that aspect.

I have a few more remarks about your design choices for the packaging from my point of view as a cctbx developer but I would like to concentrate on upstream issues in the rest of this email.

>> Note that you use a couple of new methods, e.g. env_etc.check_syslib, that
>> none of the patches define as far as I can tell.
> 
> let's see with Radostan

Could you provide us with the code for those, Radostan?

> this is part of the questions we would like to ask you.  We want to
> use the default Debian python interpreter, so we need to change all
> #!/usr/bin/env xxxx.python with #!/usr/bin/env python in all your
> files.  If I remember correctly this job is also done by distutils
> [11] is declare as a script.

First, let me shed some light on those xxx.python dispatchers. They are all identical to each others and to the "python" dispatcher (save for the irrelevant variable LIBTBX_DISPATCHER_NAME). Actually, you should know that the "python" dispatcher is only created in what we call a development environment, that is when the sources are version controlled (the configuration scripts searches for proofs that CVS, subversion or git is used). The idea is that we don't want to override python, except for developers that are supposed to know what they are doing. This is why all the demos of the cctbx you can find out there always use libtbx.python or cctbx.python.

Then, it seems to me that every single Python script in the cctbx featuring '#!/usr/bin/env python' or '#!/usr/bin/env xxx.python' is for internal use only. Conversely, every single script of any use for the users of your Debian packages are generated by the configure step and live in <build directory>/bin. As a result it seems to me that you made the right choice after all: we should change all those #!... strings to #!/usr/bin/env python as it will work for you and for us.

> we do not know now how to deal with the mutiple scripts that you are
> provinding in your bin directory. we call them dispatcher scripts
> (look at the wiki [10]).my concerne is with third party softwares
> which are relying on them in the PATH. Policy of Debian says that for
> each binary in /usr/bin we should provide a man pages so in our case
> this should be an issue...


I guess this time you were writing about those scripts that do not wrap around python, e.g. iotbx.show_distances. For the issue at hand, there are 3 categories:
i. those that are only useful for cctbx developers;
ii. those that are useful to cctbx users and that are not documented;
iii. those that are useful to cctbx users and that support the --help command line option.

For the category iii, it should be easy to generate a man page from the output of xxx --help.
Then we should document the category ii.
Finally, we should give you a comprehensive list of the category i, so that you can ignore them. 

>> 0013-fix-to-support-LDFLAGS-in-use_enviroment_flags: not sure
>> This seems done in orthodox a manner. However, this has the potential
>> of wrecking havoc to Phenix on some machines where LDFLAGS is set in
>> fancy ways.
> 
> This is also true with the other flag also, why do you treat LDFLAGS
> differently than others ?

In my experience, it is very unlikely a non-techie user would have permanently set CXXFLAGS or CCFLAGS but he may have set LDFLAGS in his .bashrc so as to help a program he installed to find the right shlibs. If that non-techie user then installs Phenix from source, esoteric bug reports may follow. But I may just be paranoid indeed. 

> In that case as explained by Radostan, Debian need to tune the build
> process by providing their own build flags. The trade-off would be to
> add a config flag that should allow or not to use the LDFLAGS
> --use-also-LDFLAGS

Indeed.

> what is your opinion ?

Nat? What do you think of this issue?

>> 0015-fix-cif-parser-to-work-with-antlr3c-3.2: for Richard's eyes
>> Richard (Gildea) is the expert when it comes to ANTLR
> 
> this should be problematic as Debian provide only 3.2 for now.  We
> shoul dask for the packaging of 3.4, but as you told us you also
> pacthed it. Did you forwarded you changed to the antlr3c upstream ?

It has definitively not been forwarded upstream because it is just a quick and dirty hack that completely goes against the original design. I honestly think that we will leave that hack as it is as any alternative would be too time-consuming. Especially in the light of ANTLR 4 coming, that will allegedly come with a proper C++ runtime (as opposed to the C runtime Richard's work is currently based on) that should not be affected by the memory allocation bloat solved by Richard's hack.

> If not our last solution is to compile it statically for now.


I reckon you should just do that indeed, as that hack is very important to make the CIF parser usable on large structures.

Baptiste Carvello wrote:
> The patch 0016-adapt-test_utils-in-libtbx-for-setup_py-test [...]
> 
> It may be, though, that this can be achieved more cleanly by
> appropriately reconfiguring the pickled Environment object.

I would indeed think so but really it is up to you whether you want to maintain such a patch or not.

> Radostan told me that this patch no longer applies on the newer nightly
> builds of cctbx, but I didn't look into this problem yet.

Most likely because of the dramatic change to the test framework since you forked. To make it use multi-processor machines more efficiently for the record. Which illustrates the issue of maintaining such a patch, although to be fair, this infrastructure code does not change very often.

Now about the support of distools. Even though as we both agree this code is totally orthogonal to our existing base code, this is no small patch and it will go upstream and we will therefore need to maintain it. Hence my scrutiny. So thanks for your detailed explanations that will save me some time reading your code.

> The "build_py" class is an experiment at adding "from __future__ import
> division" lines to python files, in case it would be needed to use them
> without the "-Qnew" option of python. Indeed, in Debian, cctbx will have
> to be compatible with the system python, without needing a global option.

I see.

> Whether this "build_py" class is needed actually depends on you: if the
> cctbx community is committed to keeping cctbx working also without the
> "-Qnew" option, we don't need it, which is much better for us.

This is actually the other way around: we wanted the cctbx to always be run with -Qnew and we actually had to fix the code in quite a few places to make all tests pass with -Qnew. Having those Python dispatchers in the first place, the least intrusive change was definitively to add -Qnew. The alternative, adding "from __future__ import division" to every single Python module did not appeal to us. Thus that build_py class is definitively necessary.

As I wrote above, the patch is not small and I definitively need to spend more time looking into it. And I will therefore come back to you with more questions.

Radostan Riedel wrote:
>> 0011-fix-missing-python-lib-during-linking: needs tidying up
>> Why don't you append to env_etc.libs_python instead of created the string env_etc.py_lib? We try to use list as much as possible in the SConscript.
> This way every extension would be additionally linked by python libs as I
> understand boost_adaptbx/SConscript right. This is normally not the right way.
You are right indeed. 

> But in gcc4.7 libsctbx_boost_python should be linked by python lib otherwise I'm
> getting undefined references.

Could you elaborate? As that sounds like a bug we should fix.

> To sum up the patches that we would like to give back:
> 
> remove-hardcoded-libtbx_build-env
> options-for-system-libs-installtarget-and-prefix
> adding-shlib-versioning
> adding-setup_py

Ok, thanks for the clarification. We will concentrate our scrutiny on those then.

Best wishes,

Luc Bourhis





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://phenix-online.org/pipermail/cctbxbb/attachments/20120809/1dbe79b1/attachment.htm>


More information about the cctbxbb mailing list