[cctbxbb] Misuse of libtbx/configure.py
Viktor Bengtsson
viktor.bengtsson at mmk.su.se
Mon Apr 9 01:02:54 PDT 2018
Dear all,
In my group we are mostly using a Windows environment and recently had reason to set up Dials on an instrument computer. Thus I ended up compiling Dials from scratch on Windows, using the old Windows C++ compiler for Python 2.7 (https://www.microsoft.com/en-us/download/details.aspx?id=44266) and the bootstrap. However, we did notice some issues. If someone starts making a fix I would be happy if the other issues were also fixed.
1. On computers with newer C++ compilers it was not possible to compile.
2. I tried following the instructions at https://dials.github.io/documentation/installation_developer.html#installing-dials-from-svn-on-windows. These are outdated. The correct way would be to make sure that the correct cl.exe file is available, use Python 2.7 and run bootstrap.py.
The one thing I can see missing for Dials on Windows right now is an easy way to redistribute it to other computers once compiled.
Regarding the comment about shipping Dials with Phenix: I am personally using Ubuntu on Windows subsystem for Linux, but ran into some issues with the Phenix version I installed. Is there any way as a non CCI-member to compile Phenix on my computer, and would it be possible to compile without duplicating the common parts? When I tried bootstrap.py --builder=Phenix there was some issue about not having a CCI account.
Best regards,
Viktor
-----Original Message-----
From: cctbxbb-bounces at phenix-online.org <cctbxbb-bounces at phenix-online.org> On Behalf Of Robert Oeffner
Sent: den 6 april 2018 18:09
To: cctbx mailing list <cctbxbb at phenix-online.org>; Nicholas Devenish <ndevenish at gmail.com>
Subject: Re: [cctbxbb] Misuse of libtbx/configure.py
Hi Nick, Graeme and Markus,
I appreciate your replies and your explanations as I didn't follow the discussions on #151.
Regarding the broken build on my PC it appears that Windows builds at CCI are working fine since they have the traditional developer environment with VS2008 installed which succeeds compiling the missing orderedset python module. On my machine the compiler is installed as part of a different package and is not invoked with a vcvarsall.bat file. Hence this step fails.
The python package downloaded during bootstrap builds on Windows was created by me and passed on to Billy who put it on the CCI website. The package could be souped up to also contain the additional modules (like
orderedset) that are being checked for and installed during the configure.py step and I'll get around to do so in the near future.
However, seeing that Dials officially doesn't yet support Windows I guess it is not a high priority, in particular since Dials builds still work fine at CCI.
Just to recap some background info: A few years ago when CCI began using Buildbot for building Linux and MacOS versions of Phenix there was a need to also provide a Windows build. I assisted in doing so using Nat Echol's shortcut that on Windows virtually all 3rd party modules for python can be bundled into a souped up python installation (see https://github.com/cctbx/cctbx_project/wiki/Base-components-for-Windows-build
). This makes the base step of bootstrap very painless. However, communication between teams is necessary as to ensure that when additional modules become a requirement these are bundled into the python installation package for Windows.
Exactly who is responsible for this is not totally clear. Phenix and CCTBX are built for Windows, MacOS and Linux whereas Dials only support MacOS and Linux. Having assisted in providing a Windows build of Phenix and CCTBX it becomes grey area when Dials is also shipped with Phenix.
Broken Dials builds within Phenix on Windows is not flagged as showstoppers by anyone and bugs are only discovered by chance as I don't think anyone is glancing over the regression test log files of Dials builds on Windows. Correct me if I'm wrong.
There may be better ways than the current one to incorporate Windows builds. In any case once we move to Conda builds this all may become a non-issue.
Regards,
Rob
On 06/04/2018 15:23, Nicholas Devenish wrote:
> Hi Rob,
>
> This appears to be a problem with the way we bootstrap windows builds.
>
> Firstly, to add to the reasons for installing dependencies in
> configure, we discovered that some sites were not using base to
> install - at all - so never got updated packages. Moving to configure
> was a compromise - and arguably this is the best place in our
> ecosystem anyway, because
> a) until you configure you don't know what modules you will have (and
> thus don't know the external package dependencies)
> b) you use configure to add new modules - so their dependencies will
> be picked up at the correct time.
>
> Except... Windows seems to have been missed with this.
>
> The Windows bootstrap doesn't build python in bootstrap but instead
> downloads a prebuilt bundle of Python 2.7.12 from LBL - we moved off
> this version a while ago, so presumably many parts of dials have been
> broken for you for a while. Only now we explicitly check for missing
> package dependencies in configure is the problem noticed.
>
> So, the new refresh mechanism just showed an old problem rather than
> causing a new one.
>
> I have no idea whose responsibility this package is, so we'll investigate.
>
> Nick
>
> On Fri, Apr 6, 2018 at 8:45 AM, Graeme.Winter at Diamond.ac.uk
> <mailto:Graeme.Winter at Diamond.ac.uk> <Graeme.Winter at diamond.ac.uk
> <mailto:Graeme.Winter at diamond.ac.uk>> wrote:
>
> Hi Rob
>
> As Markus mentioned, this was fairly extensively discussed before
> on the ticket #151 which I have now reopened
>
> The situation we have is this -
>
> - some projects build on CCTBX but are not distributed - we have
> a bunch of in-house stuff we use for automation, adding
> dependencies for these to bootstrap would quite reasonably meet
> resistance as they are not strictly needed for Phenix or DIALS
>
> - with hdf5 for dxtbx the conclusion was to sometimes release a
> broken cctbx without the dependency, which I continue to think is
> not a great solution.
>
> - sometimes we have new dependencies - this ordered is a prime
> example - and in that case we can add them to bootstrap *and also*
> pip install them if you have not recently run bootstrap base. So,
> the base install should have the dependency in already. The
> problem here is that pip tends to assume you have a conventional
> build environment. If you don't update base from time to time,
> things will either be broken or we need to add dependencies.
>
> - some third party projects build on cctbx but use dependencies
> which are completely outside of scope - not needed for Phenix or
> DIALS - and this mechanism allows them to be fetched without
> disrupting anyone's workflow
>
> I'd like a clear agreement on how things should work here - with
> the preparations to move to Python 3 there will be a lot of
> dependency changes (wx etc.) and having an agreed way to work will
> be critical.
>
> As you can see from the transactions in #151 - this was discussed
> with Nick & Co at LBL and agreed as a way forward.
>
> Cheers Graeme
>
>
> > On 4 Apr 2018, at 17:40, Robert Oeffner <rdo20 at cam.ac.uk
> <mailto:rdo20 at cam.ac.uk>> wrote:
> >
> > Hi,
> >
> > I'm not sure when this has been put into configure.py as I have
> had taken my eyes of the ball lately regarding Windows builds. But
> it seems that during bootstrap build the configure.py step is now
> doing more than just configuring modules. On my PC the excerpt
> below from stdout shows an error during the configure.py step
> >
> > Processing:
> "c:\busers\oeffner\nightlybuild\x32\current\modules\dials\libtbx_refresh.py"
> > requires mock>=2.0, has 2.0.0
> > requirement orderedset is not currently met, package not installed
> > attempting install of orderedset...
> > Collecting orderedset
> > Using cached orderedset-2.0.1.tar.gz
> > Installing collected packages: orderedset
> > Running setup.py install for orderedset: started
> > Running setup.py install for orderedset: finished with
> status 'error'
> > Complete output from command
> C:\Busers\oeffner\NightlyBuild\x32\Current\base\bin\python\python.exe
> -u -c "import setuptools,
> tokenize;__file__='c:\\users\\oeffner\\appdata\\local\\temp\\pip-build-smhifw\\orderedset\\setup.py';f=getattr(tokenize,
> 'open', open)(__file__);code=f.read().replace('\r\n',
> '\n');f.close();exec(compile(code, __file__, 'exec'))" install
> --record
> c:\users\oeffner\appdata\local\temp\pip-0seb8q-record\install-record.txt
> --single-version-externally-managed --compile:
> > running install
> > running build
> > running build_py
> > creating build
> > creating build\lib.win32-2.7
> > creating build\lib.win32-2.7\orderedset
> > copying lib\orderedset\__init__.py ->
> build\lib.win32-2.7\orderedset
> > running build_ext
> > building 'orderedset._orderedset' extension
> > error: Microsoft Visual C++ 9.0 is required (Unable to find
> vcvarsall.bat). Get it from http://aka.ms/vcpython27
> >
> > ----------------------------------------
> > Command
> "C:\Busers\oeffner\NightlyBuild\x32\Current\base\bin\python\python.exe
> -u -c "import setuptools,
> tokenize;__file__='c:\\users\\oeffner\\appdata\\local\\temp\\pip-build-smhifw\\orderedset\\setup.py';f=getattr(tokenize,
> 'open', open)(__file__);code=f.read().replace('\r\n',
> '\n');f.close();exec(compile(code, __file__, 'exec'))" install
> --record
> c:\users\oeffner\appdata\local\temp\pip-0seb8q-record\install-record.txt
> --single-version-externally-managed --compile" failed with error
> code 1 in
> c:\users\oeffner\appdata\local\temp\pip-build-smhifw\orderedset\
> > You are using pip version 9.0.1, however version 9.0.3 is available.
> > You should consider upgrading via the 'python -m pip install
> --upgrade pip' command.
> > install failed. please check manually
> >
> > My first concern with this approach is that the bootstrap
> process doesn't stop. It carries on and eventually builds the
> Phenix software suite. This may or may not be a broken build as it
> is not clear whether the above failure to build orderedset was a
> showstopper for something.
> >
> > Secondly, I thought that modules would not be installed anywhere
> else than when building base components or when downloading
> sources. For a small project it is a nice and sneaky solution to
> have certain build steps second guessing what dependencies are
> missing and then installing them "on the fly", if not covertly.
> But for a large project such as CCTBX with several teams using
> that as a common framework this becomes messy I would argue.
> >
> > A quick and dirty fix for the above error might be just to put
> vcvarsall.bat in the PATH environment. But that is not a solution
> since bootstrap.py has been fashioned to use different versions of
> Visual C++, some of which are not invoked through a vcvarsall.bat
> file.
> >
> > I appreciate the need to install additional component modules
> for python. But would it not be more tidy to install these during
> the base step of the bootstrap process? Is there any reason why
> this cannot happen?
> >
> >
> > Regards,
> >
> > Rob
> >
> > --
> > Robert Oeffner, Ph.D.
> > Research Associate, The Read Group
> > Department of Haematology,
> > Cambridge Institute for Medical Research
> > University of Cambridge
> > Cambridge Biomedical Campus
> > Wellcome Trust/MRC Building
> > Hills Road
> > Cambridge CB2 0XY
> >
> > www.cimr.cam.ac.uk/investigators/read/index.html
> <http://www.cimr.cam.ac.uk/investigators/read/index.html>
> > tel: +44(0)1223 763234
> >
> > _______________________________________________
> > cctbxbb mailing list
> > cctbxbb at phenix-online.org <mailto:cctbxbb at phenix-online.org>
> > http://phenix-online.org/mailman/listinfo/cctbxbb
> <http://phenix-online.org/mailman/listinfo/cctbxbb>
>
>
> --
> This e-mail and any attachments may contain confidential,
> copyright and or privileged material, and are for the use of the
> intended addressee only. If you are not the intended addressee or
> an authorised recipient of the addressee please notify us of
> receipt by returning the e-mail and do not use, copy, retain,
> distribute or disclose the information in or attached to the e-mail.
> Any opinions expressed within this e-mail are those of the
> individual and not necessarily of Diamond Light Source Ltd.
> Diamond Light Source Ltd. cannot guarantee that this e-mail or any
> attachments are free from viruses and we cannot accept liability
> for any damage which you may sustain as a result of software
> viruses which may be transmitted in or with the message.
> Diamond Light Source Limited (company no. 4375679). Registered in
> England and Wales with its registered office at Diamond House,
> Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11
> 0DE, United Kingdom
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org <mailto:cctbxbb at phenix-online.org>
> http://phenix-online.org/mailman/listinfo/cctbxbb
> <http://phenix-online.org/mailman/listinfo/cctbxbb>
>
>
>
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb
--
Robert Oeffner, Ph.D.
Research Associate, The Read Group
Department of Haematology,
Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY
www.cimr.cam.ac.uk/investigators/read/index.html
tel: +44(0)1223 763234
_______________________________________________
cctbxbb mailing list
cctbxbb at phenix-online.org
http://phenix-online.org/mailman/listinfo/cctbxbb
More information about the cctbxbb
mailing list