[cctbxbb] Misuse of libtbx/configure.py

Fri Apr 6 07:23:17 PDT 2018

Hi Rob,

This appears to be a problem with the way we bootstrap windows builds.

Firstly, to add to the reasons for installing dependencies in configure, we
discovered that some sites were not using base to install - at all - so
never got updated packages. Moving to configure was a compromise - and
arguably this is the best place in our ecosystem anyway, because
a) until you configure you don't know what modules you will have (and thus
don't know the external package dependencies)
b) you use configure to add new modules - so their dependencies will be
picked up at the correct time.

Except... Windows seems to have been missed with this.

The Windows bootstrap doesn't build python in bootstrap but instead
downloads a prebuilt bundle of Python 2.7.12 from LBL - we moved off this
version a while ago, so presumably many parts of dials have been broken for
you for a while. Only now we explicitly check for missing package
dependencies in configure is the problem noticed.

So, the new refresh mechanism just showed an old problem rather than
causing a new one.

I have no idea whose responsibility this package is, so we'll investigate.

Nick

On Fri, Apr 6, 2018 at 8:45 AM, Graeme.Winter at Diamond.ac.uk <
Graeme.Winter at diamond.ac.uk> wrote:

> Hi Rob
>
> As Markus mentioned, this was fairly extensively discussed before on the
> ticket #151 which I have now reopened
>
> The situation we have is this -
>
>  - some projects build on CCTBX but are not distributed - we have a bunch
> of in-house stuff we use for automation, adding dependencies for these to
> bootstrap would quite reasonably meet resistance as they are not strictly
> needed for Phenix or DIALS
>
>  - with hdf5 for dxtbx the conclusion was to sometimes release a broken
> cctbx without the dependency, which I continue to think is not a great
> solution.
>
>  - sometimes we have new dependencies - this ordered is a prime example -
> and in that case we can add them to bootstrap *and also* pip install them
> if you have not recently run bootstrap base. So, the base install should
> have the dependency in already. The problem here is that pip tends to
> assume you have a conventional build environment. If you don’t update base
> from time to time, things will either be broken or we need to add
> dependencies.
>
>  - some third party projects build on cctbx but use dependencies which are
> completely outside of scope - not needed for Phenix or DIALS - and this
> mechanism allows them to be fetched without disrupting anyone’s workflow
>
> I’d like a clear agreement on how things should work here - with the
> preparations to move to Python 3 there will be a lot of dependency changes
> (wx etc.) and having an agreed way to work will be critical.
>
> As you can see from the transactions in #151 - this was discussed with
> Nick & Co at LBL and agreed as a way forward.
>
> Cheers Graeme
>
>
> > On 4 Apr 2018, at 17:40, Robert Oeffner <rdo20 at cam.ac.uk> wrote:
> >
> > Hi,
> >
> > I'm not sure when this has been put into configure.py as I have had
> taken my eyes of the ball lately regarding Windows builds. But it seems
> that during bootstrap build the configure.py step is now doing more than
> just configuring modules. On my PC the excerpt below from stdout shows an
> error during the configure.py step
> >
> > Processing: "c:\busers\oeffner\nightlybuild\x32\current\
> modules\dials\libtbx_refresh.py"
> > requires mock>=2.0, has 2.0.0
> > requirement orderedset is not currently met, package not installed
> > attempting install of orderedset...
> > Collecting orderedset
> >   Using cached orderedset-2.0.1.tar.gz
> > Installing collected packages: orderedset
> >   Running setup.py install for orderedset: started
> >     Running setup.py install for orderedset: finished with status 'error'
> >     Complete output from command C:\Busers\oeffner\
> NightlyBuild\x32\Current\base\bin\python\python.exe -u -c "import
> setuptools, tokenize;__file__='c:\\users\\oeffner\\appdata\\local\\temp\
> \pip-build-smhifw\\orderedset\\setup.py';f=getattr(tokenize, 'open',
> open)(__file__);code=f.read().replace('\r\n',
> '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record
> c:\users\oeffner\appdata\local\temp\pip-0seb8q-record\install-record.txt
> --single-version-externally-managed --compile:
> >     running install
> >     running build
> >     running build_py
> >     creating build
> >     creating build\lib.win32-2.7
> >     creating build\lib.win32-2.7\orderedset
> >     copying lib\orderedset\__init__.py -> build\lib.win32-2.7\orderedset
> >     running build_ext
> >     building 'orderedset._orderedset' extension
> >     error: Microsoft Visual C++ 9.0 is required (Unable to find
> vcvarsall.bat). Get it from http://aka.ms/vcpython27
> >
> > ----------------------------------------
> > Command "C:\Busers\oeffner\NightlyBuild\x32\Current\base\bin\python\python.exe
> -u -c "import setuptools, tokenize;__file__='c:\\users\\
> oeffner\\appdata\\local\\temp\\pip-build-smhifw\\orderedset\\setup.py';f=getattr(tokenize,
> 'open', open)(__file__);code=f.read().replace('\r\n',
> '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record
> c:\users\oeffner\appdata\local\temp\pip-0seb8q-record\install-record.txt
> --single-version-externally-managed --compile" failed with error code 1
> in c:\users\oeffner\appdata\local\temp\pip-build-smhifw\orderedset\
> > You are using pip version 9.0.1, however version 9.0.3 is available.
> > You should consider upgrading via the 'python -m pip install --upgrade
> pip' command.
> > install failed. please check manually
> >
> > My first concern with this approach is that the bootstrap process
> doesn't stop. It carries on and eventually builds the Phenix software
> suite. This may or may not be a broken build as it is not clear whether the
> above failure to build orderedset was a showstopper for something.
> >
> > Secondly, I thought that modules would not be installed anywhere else
> than when building base components or when downloading sources. For a small
> project it is a nice and sneaky solution to have certain build steps second
> guessing what dependencies are missing and then installing them "on the
> fly", if not covertly. But for a large project such as CCTBX with several
> teams using that as a common framework this becomes messy I would argue.
> >
> > A quick and dirty fix for the above error might be just to put
> vcvarsall.bat in the PATH environment. But that is not a solution since
> bootstrap.py has been fashioned to use different versions of Visual C++,
> some of which are not invoked through a vcvarsall.bat file.
> >
> > I appreciate the need to install additional component modules for
> python. But would it not be more tidy to install these during the base step
> of the bootstrap process? Is there any reason why this cannot happen?
> >
> >
> > Regards,
> >
> > Rob
> >
> > --
> > Robert Oeffner, Ph.D.
> > Research Associate, The Read Group
> > Department of Haematology,
> > Cambridge Institute for Medical Research
> > University of Cambridge
> > Cambridge Biomedical Campus
> > Wellcome Trust/MRC Building
> > Hills Road
> > Cambridge CB2 0XY
> >
> > www.cimr.cam.ac.uk/investigators/read/index.html
> > tel: +44(0)1223 763234
> >
> > _______________________________________________
> > cctbxbb mailing list
> > cctbxbb at phenix-online.org
> > http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
> --
> This e-mail and any attachments may contain confidential, copyright and or
> privileged material, and are for the use of the intended addressee only. If
> you are not the intended addressee or an authorised recipient of the
> addressee please notify us of receipt by returning the e-mail and do not
> use, copy, retain, distribute or disclose the information in or attached to
> the e-mail.
> Any opinions expressed within this e-mail are those of the individual and
> not necessarily of Diamond Light Source Ltd.
> Diamond Light Source Ltd. cannot guarantee that this e-mail or any
> attachments are free from viruses and we cannot accept liability for any
> damage which you may sustain as a result of software viruses which may be
> transmitted in or with the message.
> Diamond Light Source Limited (company no. 4375679). Registered in England
> and Wales with its registered office at Diamond House, Harwell Science and
> Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://phenix-online.org/pipermail/cctbxbb/attachments/20180406/ae42268b/attachment-0001.htm>