[cctbxbb] Misuse of libtbx/configure.py

[Graeme.Winter@Diamond.ac.uk]

Hi Rob

As Markus mentioned, this was fairly extensively discussed before on the ticket #151 which I have now reopened

The situation we have is this - 

 - some projects build on CCTBX but are not distributed - we have a bunch of in-house stuff we use for automation, adding dependencies for these to bootstrap would quite reasonably meet resistance as they are not strictly needed for Phenix or DIALS

 - with hdf5 for dxtbx the conclusion was to sometimes release a broken cctbx without the dependency, which I continue to think is not a great solution.

 - sometimes we have new dependencies - this ordered is a prime example - and in that case we can add them to bootstrap *and also* pip install them if you have not recently run bootstrap base. So, the base install should have the dependency in already. The problem here is that pip tends to assume you have a conventional build environment. If you don’t update base from time to time, things will either be broken or we need to add dependencies. 

 - some third party projects build on cctbx but use dependencies which are completely outside of scope - not needed for Phenix or DIALS - and this mechanism allows them to be fetched without disrupting anyone’s workflow

I’d like a clear agreement on how things should work here - with the preparations to move to Python 3 there will be a lot of dependency changes (wx etc.) and having an agreed way to work will be critical. 

As you can see from the transactions in #151 - this was discussed with Nick & Co at LBL and agreed as a way forward. 

Cheers Graeme


> On 4 Apr 2018, at 17:40, Robert Oeffner <rdo20 at cam.ac.uk> wrote:
> 
> Hi,
> 
> I'm not sure when this has been put into configure.py as I have had taken my eyes of the ball lately regarding Windows builds. But it seems that during bootstrap build the configure.py step is now doing more than just configuring modules. On my PC the excerpt below from stdout shows an error during the configure.py step
> 
> Processing: "c:\busers\oeffner\nightlybuild\x32\current\modules\dials\libtbx_refresh.py"
> requires mock>=2.0, has 2.0.0
> requirement orderedset is not currently met, package not installed
> attempting install of orderedset...
> Collecting orderedset
>   Using cached orderedset-2.0.1.tar.gz
> Installing collected packages: orderedset
>   Running setup.py install for orderedset: started
>     Running setup.py install for orderedset: finished with status 'error'
>     Complete output from command C:\Busers\oeffner\NightlyBuild\x32\Current\base\bin\python\python.exe -u -c "import setuptools, tokenize;__file__='c:\\users\\oeffner\\appdata\\local\\temp\\pip-build-smhifw\\orderedset\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record c:\users\oeffner\appdata\local\temp\pip-0seb8q-record\install-record.txt --single-version-externally-managed --compile:
>     running install
>     running build
>     running build_py
>     creating build
>     creating build\lib.win32-2.7
>     creating build\lib.win32-2.7\orderedset
>     copying lib\orderedset\__init__.py -> build\lib.win32-2.7\orderedset
>     running build_ext
>     building 'orderedset._orderedset' extension
>     error: Microsoft Visual C++ 9.0 is required (Unable to find vcvarsall.bat). Get it from http://aka.ms/vcpython27
> 
> ----------------------------------------
> Command "C:\Busers\oeffner\NightlyBuild\x32\Current\base\bin\python\python.exe -u -c "import setuptools, tokenize;__file__='c:\\users\\oeffner\\appdata\\local\\temp\\pip-build-smhifw\\orderedset\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record c:\users\oeffner\appdata\local\temp\pip-0seb8q-record\install-record.txt --single-version-externally-managed --compile" failed with error code 1 in c:\users\oeffner\appdata\local\temp\pip-build-smhifw\orderedset\
> You are using pip version 9.0.1, however version 9.0.3 is available.
> You should consider upgrading via the 'python -m pip install --upgrade pip' command.
> install failed. please check manually
> 
> My first concern with this approach is that the bootstrap process doesn't stop. It carries on and eventually builds the Phenix software suite. This may or may not be a broken build as it is not clear whether the above failure to build orderedset was a showstopper for something.
> 
> Secondly, I thought that modules would not be installed anywhere else than when building base components or when downloading sources. For a small project it is a nice and sneaky solution to have certain build steps second guessing what dependencies are missing and then installing them "on the fly", if not covertly. But for a large project such as CCTBX with several teams using that as a common framework this becomes messy I would argue.
> 
> A quick and dirty fix for the above error might be just to put vcvarsall.bat in the PATH environment. But that is not a solution since bootstrap.py has been fashioned to use different versions of Visual C++, some of which are not invoked through a vcvarsall.bat file.
> 
> I appreciate the need to install additional component modules for python. But would it not be more tidy to install these during the base step of the bootstrap process? Is there any reason why this cannot happen?
> 
> 
> Regards,
> 
> Rob
> 
> -- 
> Robert Oeffner, Ph.D.
> Research Associate, The Read Group
> Department of Haematology,
> Cambridge Institute for Medical Research
> University of Cambridge
> Cambridge Biomedical Campus
> Wellcome Trust/MRC Building
> Hills Road
> Cambridge CB2 0XY
> 
> www.cimr.cam.ac.uk/investigators/read/index.html
> tel: +44(0)1223 763234
> 
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb


-- 
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. 
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom