Hi Rob, This appears to be a problem with the way we bootstrap windows builds. Firstly, to add to the reasons for installing dependencies in configure, we discovered that some sites were not using base to install - at all - so never got updated packages. Moving to configure was a compromise - and arguably this is the best place in our ecosystem anyway, because a) until you configure you don't know what modules you will have (and thus don't know the external package dependencies) b) you use configure to add new modules - so their dependencies will be picked up at the correct time. Except... Windows seems to have been missed with this. The Windows bootstrap doesn't build python in bootstrap but instead downloads a prebuilt bundle of Python 2.7.12 from LBL - we moved off this version a while ago, so presumably many parts of dials have been broken for you for a while. Only now we explicitly check for missing package dependencies in configure is the problem noticed. So, the new refresh mechanism just showed an old problem rather than causing a new one. I have no idea whose responsibility this package is, so we'll investigate. Nick On Fri, Apr 6, 2018 at 8:45 AM, [email protected] < [email protected]> wrote:

Hi Rob

As Markus mentioned, this was fairly extensively discussed before on the ticket #151 which I have now reopened

The situation we have is this -

- some projects build on CCTBX but are not distributed - we have a bunch of in-house stuff we use for automation, adding dependencies for these to bootstrap would quite reasonably meet resistance as they are not strictly needed for Phenix or DIALS

- with hdf5 for dxtbx the conclusion was to sometimes release a broken cctbx without the dependency, which I continue to think is not a great solution.

- sometimes we have new dependencies - this ordered is a prime example - and in that case we can add them to bootstrap *and also* pip install them if you have not recently run bootstrap base. So, the base install should have the dependency in already. The problem here is that pip tends to assume you have a conventional build environment. If you don’t update base from time to time, things will either be broken or we need to add dependencies.

- some third party projects build on cctbx but use dependencies which are completely outside of scope - not needed for Phenix or DIALS - and this mechanism allows them to be fetched without disrupting anyone’s workflow

I’d like a clear agreement on how things should work here - with the preparations to move to Python 3 there will be a lot of dependency changes (wx etc.) and having an agreed way to work will be critical.

As you can see from the transactions in #151 - this was discussed with Nick & Co at LBL and agreed as a way forward.

Cheers Graeme

...

On 4 Apr 2018, at 17:40, Robert Oeffner wrote:

Hi,

I'm not sure when this has been put into configure.py as I have had taken my eyes of the ball lately regarding Windows builds. But it seems that during bootstrap build the configure.py step is now doing more than just configuring modules. On my PC the excerpt below from stdout shows an error during the configure.py step

Processing: "c:\busers\oeffner\nightlybuild\x32\current\ modules\dials\libtbx_refresh.py" requires mock>=2.0, has 2.0.0 requirement orderedset is not currently met, package not installed attempting install of orderedset... Collecting orderedset Using cached orderedset-2.0.1.tar.gz Installing collected packages: orderedset Running setup.py install for orderedset: started Running setup.py install for orderedset: finished with status 'error' Complete output from command C:\Busers\oeffner\ NightlyBuild\x32\Current\base\bin\python\python.exe -u -c "import setuptools, tokenize;__file__='c:\\users\\oeffner\\appdata\\local\\temp\ \pip-build-smhifw\\orderedset\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record c:\users\oeffner\appdata\local\temp\pip-0seb8q-record\install-record.txt --single-version-externally-managed --compile: running install running build running build_py creating build creating build\lib.win32-2.7 creating build\lib.win32-2.7\orderedset copying lib\orderedset\__init__.py -> build\lib.win32-2.7\orderedset running build_ext building 'orderedset._orderedset' extension error: Microsoft Visual C++ 9.0 is required (Unable to find vcvarsall.bat). Get it from http://aka.ms/vcpython27

---------------------------------------- Command "C:\Busers\oeffner\NightlyBuild\x32\Current\base\bin\python\python.exe -u -c "import setuptools, tokenize;__file__='c:\\users\\ oeffner\\appdata\\local\\temp\\pip-build-smhifw\\orderedset\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record c:\users\oeffner\appdata\local\temp\pip-0seb8q-record\install-record.txt --single-version-externally-managed --compile" failed with error code 1 in c:\users\oeffner\appdata\local\temp\pip-build-smhifw\orderedset\ You are using pip version 9.0.1, however version 9.0.3 is available. You should consider upgrading via the 'python -m pip install --upgrade pip' command. install failed. please check manually

My first concern with this approach is that the bootstrap process doesn't stop. It carries on and eventually builds the Phenix software suite. This may or may not be a broken build as it is not clear whether the above failure to build orderedset was a showstopper for something.

Secondly, I thought that modules would not be installed anywhere else than when building base components or when downloading sources. For a small project it is a nice and sneaky solution to have certain build steps second guessing what dependencies are missing and then installing them "on the fly", if not covertly. But for a large project such as CCTBX with several teams using that as a common framework this becomes messy I would argue.

A quick and dirty fix for the above error might be just to put vcvarsall.bat in the PATH environment. But that is not a solution since bootstrap.py has been fashioned to use different versions of Visual C++, some of which are not invoked through a vcvarsall.bat file.

I appreciate the need to install additional component modules for python. But would it not be more tidy to install these during the base step of the bootstrap process? Is there any reason why this cannot happen?

Regards,

Rob

-- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY

www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Dear all, In my group we are mostly using a Windows environment and recently had reason to set up Dials on an instrument computer. Thus I ended up compiling Dials from scratch on Windows, using the old Windows C++ compiler for Python 2.7 (https://www.microsoft.com/en-us/download/details.aspx?id=44266) and the bootstrap. However, we did notice some issues. If someone starts making a fix I would be happy if the other issues were also fixed. 1. On computers with newer C++ compilers it was not possible to compile. 2. I tried following the instructions at https://dials.github.io/documentation/installation_developer.html#installing.... These are outdated. The correct way would be to make sure that the correct cl.exe file is available, use Python 2.7 and run bootstrap.py. The one thing I can see missing for Dials on Windows right now is an easy way to redistribute it to other computers once compiled. Regarding the comment about shipping Dials with Phenix: I am personally using Ubuntu on Windows subsystem for Linux, but ran into some issues with the Phenix version I installed. Is there any way as a non CCI-member to compile Phenix on my computer, and would it be possible to compile without duplicating the common parts? When I tried bootstrap.py --builder=Phenix there was some issue about not having a CCI account. Best regards, Viktor -----Original Message----- From: [email protected] On Behalf Of Robert Oeffner Sent: den 6 april 2018 18:09 To: cctbx mailing list ; Nicholas Devenish Subject: Re: [cctbxbb] Misuse of libtbx/configure.py Hi Nick, Graeme and Markus, I appreciate your replies and your explanations as I didn't follow the discussions on #151. Regarding the broken build on my PC it appears that Windows builds at CCI are working fine since they have the traditional developer environment with VS2008 installed which succeeds compiling the missing orderedset python module. On my machine the compiler is installed as part of a different package and is not invoked with a vcvarsall.bat file. Hence this step fails. The python package downloaded during bootstrap builds on Windows was created by me and passed on to Billy who put it on the CCI website. The package could be souped up to also contain the additional modules (like orderedset) that are being checked for and installed during the configure.py step and I'll get around to do so in the near future. However, seeing that Dials officially doesn't yet support Windows I guess it is not a high priority, in particular since Dials builds still work fine at CCI. Just to recap some background info: A few years ago when CCI began using Buildbot for building Linux and MacOS versions of Phenix there was a need to also provide a Windows build. I assisted in doing so using Nat Echol's shortcut that on Windows virtually all 3rd party modules for python can be bundled into a souped up python installation (see https://github.com/cctbx/cctbx_project/wiki/Base-components-for-Windows-buil... ). This makes the base step of bootstrap very painless. However, communication between teams is necessary as to ensure that when additional modules become a requirement these are bundled into the python installation package for Windows. Exactly who is responsible for this is not totally clear. Phenix and CCTBX are built for Windows, MacOS and Linux whereas Dials only support MacOS and Linux. Having assisted in providing a Windows build of Phenix and CCTBX it becomes grey area when Dials is also shipped with Phenix. Broken Dials builds within Phenix on Windows is not flagged as showstoppers by anyone and bugs are only discovered by chance as I don't think anyone is glancing over the regression test log files of Dials builds on Windows. Correct me if I'm wrong. There may be better ways than the current one to incorporate Windows builds. In any case once we move to Conda builds this all may become a non-issue. Regards, Rob On 06/04/2018 15:23, Nicholas Devenish wrote:

Hi Rob,

This appears to be a problem with the way we bootstrap windows builds.

Firstly, to add to the reasons for installing dependencies in configure, we discovered that some sites were not using base to install - at all - so never got updated packages. Moving to configure was a compromise - and arguably this is the best place in our ecosystem anyway, because a) until you configure you don't know what modules you will have (and thus don't know the external package dependencies) b) you use configure to add new modules - so their dependencies will be picked up at the correct time.

Except... Windows seems to have been missed with this.

The Windows bootstrap doesn't build python in bootstrap but instead downloads a prebuilt bundle of Python 2.7.12 from LBL - we moved off this version a while ago, so presumably many parts of dials have been broken for you for a while. Only now we explicitly check for missing package dependencies in configure is the problem noticed.

So, the new refresh mechanism just showed an old problem rather than causing a new one.

I have no idea whose responsibility this package is, so we'll investigate.

Nick

On Fri, Apr 6, 2018 at 8:45 AM, [email protected] mailto:[email protected] mailto:[email protected]> wrote:

Hi Rob

As Markus mentioned, this was fairly extensively discussed before on the ticket #151 which I have now reopened

The situation we have is this -

 - some projects build on CCTBX but are not distributed - we have a bunch of in-house stuff we use for automation, adding dependencies for these to bootstrap would quite reasonably meet resistance as they are not strictly needed for Phenix or DIALS

 - with hdf5 for dxtbx the conclusion was to sometimes release a broken cctbx without the dependency, which I continue to think is not a great solution.

 - sometimes we have new dependencies - this ordered is a prime example - and in that case we can add them to bootstrap *and also* pip install them if you have not recently run bootstrap base. So, the base install should have the dependency in already. The problem here is that pip tends to assume you have a conventional build environment. If you don't update base from time to time, things will either be broken or we need to add dependencies.

 - some third party projects build on cctbx but use dependencies which are completely outside of scope - not needed for Phenix or DIALS - and this mechanism allows them to be fetched without disrupting anyone's workflow

I'd like a clear agreement on how things should work here - with the preparations to move to Python 3 there will be a lot of dependency changes (wx etc.) and having an agreed way to work will be critical.

As you can see from the transactions in #151 - this was discussed with Nick & Co at LBL and agreed as a way forward.

Cheers Graeme

> On 4 Apr 2018, at 17:40, Robert Oeffner mailto:[email protected]> wrote: > > Hi, > > I'm not sure when this has been put into configure.py as I have had taken my eyes of the ball lately regarding Windows builds. But it seems that during bootstrap build the configure.py step is now doing more than just configuring modules. On my PC the excerpt below from stdout shows an error during the configure.py step > > Processing: "c:\busers\oeffner\nightlybuild\x32\current\modules\dials\libtbx_refresh.py" > requires mock>=2.0, has 2.0.0 > requirement orderedset is not currently met, package not installed > attempting install of orderedset... > Collecting orderedset >   Using cached orderedset-2.0.1.tar.gz > Installing collected packages: orderedset >   Running setup.py install for orderedset: started >     Running setup.py install for orderedset: finished with status 'error' >     Complete output from command C:\Busers\oeffner\NightlyBuild\x32\Current\base\bin\python\python.exe -u -c "import setuptools, tokenize;__file__='c:\\users\\oeffner\\appdata\\local\\temp\\pip-build-smhifw\\orderedset\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record c:\users\oeffner\appdata\local\temp\pip-0seb8q-record\install-record.txt --single-version-externally-managed --compile: >     running install >     running build >     running build_py >     creating build >     creating build\lib.win32-2.7 >     creating build\lib.win32-2.7\orderedset >     copying lib\orderedset\__init__.py -> build\lib.win32-2.7\orderedset >     running build_ext >     building 'orderedset._orderedset' extension >     error: Microsoft Visual C++ 9.0 is required (Unable to find vcvarsall.bat). Get it from http://aka.ms/vcpython27 > > ---------------------------------------- > Command "C:\Busers\oeffner\NightlyBuild\x32\Current\base\bin\python\python.exe -u -c "import setuptools, tokenize;__file__='c:\\users\\oeffner\\appdata\\local\\temp\\pip-build-smhifw\\orderedset\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record c:\users\oeffner\appdata\local\temp\pip-0seb8q-record\install-record.txt --single-version-externally-managed --compile" failed with error code 1 in c:\users\oeffner\appdata\local\temp\pip-build-smhifw\orderedset\ > You are using pip version 9.0.1, however version 9.0.3 is available. > You should consider upgrading via the 'python -m pip install --upgrade pip' command. > install failed. please check manually > > My first concern with this approach is that the bootstrap process doesn't stop. It carries on and eventually builds the Phenix software suite. This may or may not be a broken build as it is not clear whether the above failure to build orderedset was a showstopper for something. > > Secondly, I thought that modules would not be installed anywhere else than when building base components or when downloading sources. For a small project it is a nice and sneaky solution to have certain build steps second guessing what dependencies are missing and then installing them "on the fly", if not covertly. But for a large project such as CCTBX with several teams using that as a common framework this becomes messy I would argue. > > A quick and dirty fix for the above error might be just to put vcvarsall.bat in the PATH environment. But that is not a solution since bootstrap.py has been fashioned to use different versions of Visual C++, some of which are not invoked through a vcvarsall.bat file. > > I appreciate the need to install additional component modules for python. But would it not be more tidy to install these during the base step of the bootstrap process? Is there any reason why this cannot happen? > > > Regards, > > Rob > > -- > Robert Oeffner, Ph.D. > Research Associate, The Read Group > Department of Haematology, > Cambridge Institute for Medical Research > University of Cambridge > Cambridge Biomedical Campus > Wellcome Trust/MRC Building > Hills Road > Cambridge CB2 0XY > > www.cimr.cam.ac.uk/investigators/read/index.html http://www.cimr.cam.ac.uk/investigators/read/index.html > tel: +44(0)1223 763234 > > _______________________________________________ > cctbxbb mailing list > [email protected] mailto:[email protected] > http://phenix-online.org/mailman/listinfo/cctbxbb http://phenix-online.org/mailman/listinfo/cctbxbb

-- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

_______________________________________________ cctbxbb mailing list [email protected] mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234 _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb