Hallo Rob, Richard (Gildea) has figured this out for me -- I was using python 2.7.8 and that's the first part of the problem. I updated to 2.7.13 and the initial issues to do with ssl are solved. I am still stuck at the end: **************************************************************************** Automated CCTBX dependencies build script report problems to [email protected] **************************************************************************** Base directory already exists and --skip-if-exists set; exiting. ===== Running in build: run configure.py Traceback (most recent call last): File "bootstrap.py", line 2113, in <module> run() File "bootstrap.py", line 2108, in run enable_shared=options.enable_shared, File "bootstrap.py", line 1036, in run i.run() File "bootstrap.py", line 189, in run raise RuntimeError("Could not run %s: File not found" % executable) RuntimeError: Could not run base\bin\python: File not found What is it looking for? Please note: this is *not* a development machine. Greetings Horst On 6 April 2017 at 10:17, R. D. Oeffner wrote:

Dear Horst,

I'll look into that. FYI you can also get Windows builds of CCTBX. See email below. You may have to resort using bundles from a few days ago because the two last nightly builds on windows are broken.

Rob

-----Original Message----- From: R. D. Oeffner Sent: Sunday, March 19, 2017 3:49 PM To: Billy Poon ; [email protected] Subject: Re: Windows builds for cctbx

There should now be CCTBX builds for Windows in the most recent folders on http://cci.lbl.gov/cctbx_build . They are just the sources and the build directory zipped into an archive and does not include an installation program like the graphical Phenix installer for Windows. Presumably people wanting to use only CCTBX are sufficiently computer savvy to cope with this.

Rob

-- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY

www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234 mobile: +44(0)7712 887162 -----Original Message----- From: Horst Puschmann Sent: Thursday, April 6, 2017 9:59 AM To: [email protected] Subject: [cctbxbb] bootstrap.py on windows

Hello

I am trying to install the cctbx on a Windows 7 64 bit machine using the following bootstrap.py file:

https://raw.githubusercontent.com/cctbx/cctbx_project/master /libtbx/auto_build/bootstrap.py

The script will fail, unless I comment out lines 1101 and 1102:

#if self.isPlatformWindows(): #tarurl, arxname, dirpath = MODULES.get_module(module)().g et_tarauthenticated(auth=self.get_auth())

If it tries to execute the commented lines, the error will be "KeyError: 'cciuser' in line 590"

After commenting this out, it will fail in line 271, with "AttributeError: 'module' object has no attribute '_create_unverified_context'"

I can get round that with disabling line 247:

if sys.platform == "win32":

(i.e. change this to if sys.platform == "xxx":)

After that, things start downloading. All appears well until the same thing happens again in another bootstrap.py file in \modules\cctbx_project\libtbx\auto_build\bootstrap.py

If I disable *that*, it goes further but fails finally with

===== Running in build: run configure.py Traceback (most recent call last): File "bootstrap.py", line 2113, in <module> run() File "bootstrap.py", line 2108, in run enable_shared=options.enable_shared, File "bootstrap.py", line 1036, in run i.run() File "bootstrap.py", line 189, in run raise RuntimeError("Could not run %s: File not found" % executable) RuntimeError: Could not run base\bin\python: File not found

I guess the automatic tests haven't picked this up, because no authentication is needed, maybe?

Greetings Horst

Virus-free. www.avg.com

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Thanks Richard, Rob and Markus! I cleared the entire directory and ran bootstrap.py again (Python 2.7.13) -- It's the same at the end: ===== Running in build: run configure.py Traceback (most recent call last): File "bootstrap.py", line 2113, in <module> run() File "bootstrap.py", line 2108, in run enable_shared=options.enable_shared, File "bootstrap.py", line 1036, in run i.run() File "bootstrap.py", line 189, in run raise RuntimeError("Could not run %s: File not found" % executable) RuntimeError: Could not run base\bin\python: File not found Just to show that the '.exe' is not required on Windows: C:\cctbx_17>c:\Python27\python Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:53:40) [MSC v.1500 64 bit ( AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information.

...

...

There is a python.exe in \base\bin\python, btw. Horst

Hi Horst, Rob I had a brief look at the bootstrap script. If I understand it correctly only authenticated access is allowed at the moment due to the unique way the download archives are constructed. I am not entirely sure why the sequence to download the 'hot' packages (I did not venture further into the bootstrap process) consists of * ssh'ing to CCI servers * creating a TAR archive * downloading said archive via SCP * unpacking the archive instead of going down the conventional route of * downloading the default TAR archive via HTTP (yes, they're called .gz because someone was lazy, but they are .tar.gz files) * unpacking the archive I suspect that this is due to the absence of CURL on Windows machines combined with how the bootstrap script is intertwined with the buildbot testing system. Internally, the bootstrap script uses an "_add_curl" call to download files. When run within the buildbot testing system this presumably gets replaced by an actual call to the 'curl' program, which doesn't exist on Windows, and so would fail. However, when you run bootstrap.py directly _add_curl actually uses an internal downloading routine, which works. (If true, this would simply be one more example of why buildbot determining the layout of bootstrap is both limiting and dangerous. The solution for that of course would be to install CURL for Windows (and/or replace buildbot)) So I guess the two lines in add_module, https://github.com/cctbx/cctbx_project/blob/master/libtbx/auto_build/bootstr... if self.isPlatformWindows(): tarurl, arxname, dirpath = MODULES.get_module(module)().get_tarauthenticated(auth=self.get_auth()) could be removed along with all references to tarauthenticated? As Horst already said there is more going wrong here. The ROBOCOPY calls are *incredibly* slow for something that could/should be done in pure python instead (shutil.rmtree?). The ssl 'workaround' uses internal methods, which are not available on older versions of Python. The error at the end is presumably that the python interpreter is called python.exe on Windows? -Markus -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of R. D. Oeffner Sent: 06 April 2017 10:18 To: [email protected]; [email protected] Subject: Re: [cctbxbb] bootstrap.py on windows Dear Horst, I'll look into that. FYI you can also get Windows builds of CCTBX. See email below. You may have to resort using bundles from a few days ago because the two last nightly builds on windows are broken. Rob -----Original Message----- From: R. D. Oeffner Sent: Sunday, March 19, 2017 3:49 PM To: Billy Poon ; [email protected] Subject: Re: Windows builds for cctbx There should now be CCTBX builds for Windows in the most recent folders on http://cci.lbl.gov/cctbx_build . They are just the sources and the build directory zipped into an archive and does not include an installation program like the graphical Phenix installer for Windows. Presumably people wanting to use only CCTBX are sufficiently computer savvy to cope with this. Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234 mobile: +44(0)7712 887162 -----Original Message----- From: Horst Puschmann Sent: Thursday, April 6, 2017 9:59 AM To: [email protected] Subject: [cctbxbb] bootstrap.py on windows Hello I am trying to install the cctbx on a Windows 7 64 bit machine using the following bootstrap.py file: https://raw.githubusercontent.com/cctbx/cctbx_project/master/libtbx/auto_bui... The script will fail, unless I comment out lines 1101 and 1102: #if self.isPlatformWindows(): #tarurl, arxname, dirpath = MODULES.get_module(module)().get_tarauthenticated(auth=self.get_auth()) If it tries to execute the commented lines, the error will be "KeyError: 'cciuser' in line 590" After commenting this out, it will fail in line 271, with "AttributeError: 'module' object has no attribute '_create_unverified_context'" I can get round that with disabling line 247: if sys.platform == "win32": (i.e. change this to if sys.platform == "xxx":) After that, things start downloading. All appears well until the same thing happens again in another bootstrap.py file in \modules\cctbx_project\libtbx\auto_build\bootstrap.py If I disable *that*, it goes further but fails finally with ===== Running in build: run configure.py Traceback (most recent call last): File "bootstrap.py", line 2113, in <module> run() File "bootstrap.py", line 2108, in run enable_shared=options.enable_shared, File "bootstrap.py", line 1036, in run i.run() File "bootstrap.py", line 189, in run raise RuntimeError("Could not run %s: File not found" % executable) RuntimeError: Could not run base\bin\python: File not found I guess the automatic tests haven't picked this up, because no authentication is needed, maybe? Greetings Horst Virus-free. www.avg.com _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

Hello Rob, Yes, this is the command I am using: C:\Python27\python.exe bootstrap.py --builder=cctbx Everything works fine (and I don't think it's too slow!) -- but that last call to python can not work, since it calls a directory: raise RuntimeError("Could not run %s: File not found" % executable) RuntimeError: Could not run base\bin\python: File not found if it called base\bin\python\python instead, it would work. Of course, I don't know what might not work after that step.... Horst On 6 April 2017 at 12:17, R.D. Oeffner wrote:

Hi Horst and Markus,

Horst are you using a command line such as:

C:\Python\python.exe bootstrap.py --builder=cctbx

I realise this is broken because I never tested rigorously for anonymous builds of CCTBX on Windows. Am in the process of fixing this.

Markus you are correct that the roundabout way to get hot packages from CCI is because rsync does not exist on Windows. With Git installed on windows curl and a few other Unix tools do exist. This was a quick solution for making a Windows build Phenix, Dials and CCTBX where it has been running smoothly since August last year. The Robocopy calls for deleting files is a solution, albeit cumbersome for deleting very long file paths on Windows (>255 characters). Pythons shutil.rmtree() will fail in such cases. File deletion on Windows is always much slower than on UNIX by design.

Making bootstrap.py and Buildbot work on Windows has been taken up a lot of work given that Buildbot seems to has been designed in denial of the existence of Windows and Microsoft until recently have been in denial of open source solutions.

Rob

On 06/04/2017 10:50, [email protected] wrote:

Hi Horst, Rob

I had a brief look at the bootstrap script. If I understand it correctly only authenticated access is allowed at the moment due to the unique way the download archives are constructed.

I am not entirely sure why the sequence to download the 'hot' packages (I did not venture further into the bootstrap process) consists of * ssh'ing to CCI servers * creating a TAR archive * downloading said archive via SCP * unpacking the archive instead of going down the conventional route of * downloading the default TAR archive via HTTP (yes, they're called .gz because someone was lazy, but they are .tar.gz files) * unpacking the archive

I suspect that this is due to the absence of CURL on Windows machines combined with how the bootstrap script is intertwined with the buildbot testing system. Internally, the bootstrap script uses an "_add_curl" call to download files. When run within the buildbot testing system this presumably gets replaced by an actual call to the 'curl' program, which doesn't exist on Windows, and so would fail. However, when you run bootstrap.py directly _add_curl actually uses an internal downloading routine, which works. (If true, this would simply be one more example of why buildbot determining the layout of bootstrap is both limiting and dangerous. The solution for that of course would be to install CURL for Windows (and/or replace buildbot))

So I guess the two lines in add_module, https://github.com/cctbx/cctbx_project/blob/master/libtbx/auto_build/bootstr... if self.isPlatformWindows(): tarurl, arxname, dirpath = MODULES.get_module(module)().get_tarauthenticated(auth=self.get_auth()) could be removed along with all references to tarauthenticated?

As Horst already said there is more going wrong here. The ROBOCOPY calls are *incredibly* slow for something that could/should be done in pure python instead (shutil.rmtree?). The ssl 'workaround' uses internal methods, which are not available on older versions of Python.

The error at the end is presumably that the python interpreter is called python.exe on Windows?

-Markus

-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of R. D. Oeffner Sent: 06 April 2017 10:18 To: [email protected]; [email protected] Subject: Re: [cctbxbb] bootstrap.py on windows

Dear Horst,

I'll look into that. FYI you can also get Windows builds of CCTBX. See email below. You may have to resort using bundles from a few days ago because the two last nightly builds on windows are broken.

Rob

-----Original Message----- From: R. D. Oeffner Sent: Sunday, March 19, 2017 3:49 PM To: Billy Poon ; [email protected] Subject: Re: Windows builds for cctbx

There should now be CCTBX builds for Windows in the most recent folders on http://cci.lbl.gov/cctbx_build . They are just the sources and the build directory zipped into an archive and does not include an installation program like the graphical Phenix installer for Windows. Presumably people wanting to use only CCTBX are sufficiently computer savvy to cope with this.

Rob

-- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234 <+44%201223%20763234> mobile: +44(0)7712 887162 <+44%207712%20887162> -----Original Message----- From: Horst Puschmann Sent: Thursday, April 6, 2017 9:59 AM To: [email protected] Subject: [cctbxbb] bootstrap.py on windows

Hello

I am trying to install the cctbx on a Windows 7 64 bit machine using the following bootstrap.py file: https://raw.githubusercontent.com/cctbx/cctbx_project/master/libtbx/auto_bui...

The script will fail, unless I comment out lines 1101 and 1102:

#if self.isPlatformWindows(): #tarurl, arxname, dirpath = MODULES.get_module(module)().get_tarauthenticated(auth=self.get_auth())

If it tries to execute the commented lines, the error will be "KeyError: 'cciuser' in line 590"

After commenting this out, it will fail in line 271, with "AttributeError: 'module' object has no attribute '_create_unverified_context'"

I can get round that with disabling line 247:

if sys.platform == "win32":

(i.e. change this to if sys.platform == "xxx":)

After that, things start downloading. All appears well until the same thing happens again in another bootstrap.py file in \modules\cctbx_project\libtbx\auto_build\bootstrap.py

If I disable *that*, it goes further but fails finally with

===== Running in build: run configure.py Traceback (most recent call last): File "bootstrap.py", line 2113, in <module> run() File "bootstrap.py", line 2108, in run enable_shared=options.enable_shared, File "bootstrap.py", line 1036, in run i.run() File "bootstrap.py", line 189, in run raise RuntimeError("Could not run %s: File not found" % executable) RuntimeError: Could not run base\bin\python: File not found

I guess the automatic tests haven't picked this up, because no authentication is needed, maybe?

Greetings Horst

Virus-free. www.avg.com

_______________________________________________ cctbxbb mailing [email protected]http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing [email protected]http://phenix-online.org/mailman/listinfo/cctbxbb

Hello Rob, This works fine -- Thank You! ... but then something goes wrong in the compilation: pycbf_wrap.c c:\cctbx_17\base\hdf5-1.8.16\include\H5public.h(158) : error C2371: 'ssize_t' : redefinition; different basic types c:\cctbx_17\base\bin\python\include\pyconfig.h(205) : see declaration of 'ssize_t' scons: *** [cbflib\pycbf\pycbf_wrap.obj] Error 2 scons: building terminated because of errors. usr+sys time: 9.47 seconds wall clock time: 4 minutes 22.11 seconds (262.11 seconds total) Process failed with return code 2 ;) Horst On 6 April 2017 at 14:37, R.D. Oeffner wrote:

Hi Horst,

I just committed a fix to bootstrap.py to allow anonymous build. If you get it with:

curl https://raw.githubusercontent.com/cctbx/cctbx_project/ master/libtbx/auto_build/bootstrap.py > bootstrap.py

and then do the build with:

C:\Python27\python.exe bootstrap.py --builder=cctbx

this works for me. I did not encounter the RuntimeError you mentioned.

Let me know if it doesn't work,

Rob

On 06/04/2017 12:28, Horst Puschmann wrote:

Hello Rob,

Yes, this is the command I am using:

C:\Python27\python.exe bootstrap.py --builder=cctbx

Everything works fine (and I don't think it's too slow!) -- but that last call to python can not work, since it calls a directory:

raise RuntimeError("Could not run %s: File not found" % executable) RuntimeError: Could not run base\bin\python: File not found

if it called

base\bin\python\python instead, it would work.

Of course, I don't know what might not work after that step....

Horst

On 6 April 2017 at 12:17, R.D. Oeffner wrote:

...

Hi Horst and Markus,

Horst are you using a command line such as:

C:\Python\python.exe bootstrap.py --builder=cctbx

I realise this is broken because I never tested rigorously for anonymous builds of CCTBX on Windows. Am in the process of fixing this.

Markus you are correct that the roundabout way to get hot packages from CCI is because rsync does not exist on Windows. With Git installed on windows curl and a few other Unix tools do exist. This was a quick solution for making a Windows build Phenix, Dials and CCTBX where it has been running smoothly since August last year. The Robocopy calls for deleting files is a solution, albeit cumbersome for deleting very long file paths on Windows (>255 characters). Pythons shutil.rmtree() will fail in such cases. File deletion on Windows is always much slower than on UNIX by design.

Making bootstrap.py and Buildbot work on Windows has been taken up a lot of work given that Buildbot seems to has been designed in denial of the existence of Windows and Microsoft until recently have been in denial of open source solutions.

Rob

On 06/04/2017 10:50, [email protected] wrote:

Hi Horst, Rob

I had a brief look at the bootstrap script. If I understand it correctly only authenticated access is allowed at the moment due to the unique way the download archives are constructed.

I am not entirely sure why the sequence to download the 'hot' packages (I did not venture further into the bootstrap process) consists of * ssh'ing to CCI servers * creating a TAR archive * downloading said archive via SCP * unpacking the archive instead of going down the conventional route of * downloading the default TAR archive via HTTP (yes, they're called .gz because someone was lazy, but they are .tar.gz files) * unpacking the archive

I suspect that this is due to the absence of CURL on Windows machines combined with how the bootstrap script is intertwined with the buildbot testing system. Internally, the bootstrap script uses an "_add_curl" call to download files. When run within the buildbot testing system this presumably gets replaced by an actual call to the 'curl' program, which doesn't exist on Windows, and so would fail. However, when you run bootstrap.py directly _add_curl actually uses an internal downloading routine, which works. (If true, this would simply be one more example of why buildbot determining the layout of bootstrap is both limiting and dangerous. The solution for that of course would be to install CURL for Windows (and/or replace buildbot))

So I guess the two lines in add_module, https://github.com/cctbx/cctbx_project/blob/master/libtbx/auto_build/bootstr... if self.isPlatformWindows(): tarurl, arxname, dirpath = MODULES.get_module(module)().get_tarauthenticated(auth=self.get_auth()) could be removed along with all references to tarauthenticated?

As Horst already said there is more going wrong here. The ROBOCOPY calls are *incredibly* slow for something that could/should be done in pure python instead (shutil.rmtree?). The ssl 'workaround' uses internal methods, which are not available on older versions of Python.

The error at the end is presumably that the python interpreter is called python.exe on Windows?

-Markus

-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of R. D. Oeffner Sent: 06 April 2017 10:18 To: [email protected]; [email protected] Subject: Re: [cctbxbb] bootstrap.py on windows

Dear Horst,

I'll look into that. FYI you can also get Windows builds of CCTBX. See email below. You may have to resort using bundles from a few days ago because the two last nightly builds on windows are broken.

Rob

-----Original Message----- From: R. D. Oeffner Sent: Sunday, March 19, 2017 3:49 PM To: Billy Poon ; [email protected] Subject: Re: Windows builds for cctbx

There should now be CCTBX builds for Windows in the most recent folders on http://cci.lbl.gov/cctbx_build . They are just the sources and the build directory zipped into an archive and does not include an installation program like the graphical Phenix installer for Windows. Presumably people wanting to use only CCTBX are sufficiently computer savvy to cope with this.

Rob

-- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234 <+44%201223%20763234> mobile: +44(0)7712 887162 <+44%207712%20887162> -----Original Message----- From: Horst Puschmann Sent: Thursday, April 6, 2017 9:59 AM To: [email protected] Subject: [cctbxbb] bootstrap.py on windows

Hello

I am trying to install the cctbx on a Windows 7 64 bit machine using the following bootstrap.py file: https://raw.githubusercontent.com/cctbx/cctbx_project/master/libtbx/auto_bui...

The script will fail, unless I comment out lines 1101 and 1102:

#if self.isPlatformWindows(): #tarurl, arxname, dirpath = MODULES.get_module(module)().get_tarauthenticated(auth=self.get_auth())

If it tries to execute the commented lines, the error will be "KeyError: 'cciuser' in line 590"

After commenting this out, it will fail in line 271, with "AttributeError: 'module' object has no attribute '_create_unverified_context'"

I can get round that with disabling line 247:

if sys.platform == "win32":

(i.e. change this to if sys.platform == "xxx":)

After that, things start downloading. All appears well until the same thing happens again in another bootstrap.py file in \modules\cctbx_project\libtbx\auto_build\bootstrap.py

If I disable *that*, it goes further but fails finally with

===== Running in build: run configure.py Traceback (most recent call last): File "bootstrap.py", line 2113, in <module> run() File "bootstrap.py", line 2108, in run enable_shared=options.enable_shared, File "bootstrap.py", line 1036, in run i.run() File "bootstrap.py", line 189, in run raise RuntimeError("Could not run %s: File not found" % executable) RuntimeError: Could not run base\bin\python: File not found

I guess the automatic tests haven't picked this up, because no authentication is needed, maybe?

Greetings Horst

Virus-free. www.avg.com

_______________________________________________ cctbxbb mailing [email protected]http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing [email protected]http://phenix-online.org/mailman/listinfo/cctbxbb

Hi Horst, I encountered this error too and committed a fix half an hour ago to cctbx_projectcbflib_adaptbxmsvc_includeunistd.h. If you git pull cctbx_project and build with libtbx.scons again it should compile. Rob On 06/04/2017 15:54, Horst Puschmann wrote:

Hello Rob,

This works fine -- Thank You!

... but then something goes wrong in the compilation:

pycbf_wrap.c c:cctbx_17basehdf5-1.8.16includeH5public.h(158) : error C2371: 'ssize_t' : redefinition; different basic types c:cctbx_17basebinpythonincludepyconfig.h(205) : see declaration of 'ssize_t' scons: *** [cbflibpycbfpycbf_wrap.obj] Error 2 scons: building terminated because of errors. usr+sys time: 9.47 seconds wall clock time: 4 minutes 22.11 seconds (262.11 seconds total) Process failed with return code 2

;)

Horst

On 6 April 2017 at 14:37, R.D. Oeffner wrote:

Hi Horst,

I just committed a fix to bootstrap.py to allow anonymous build. If you get it with:

curl https://raw.githubusercontent.com/cctbx/cctbx_project/master/libtbx/auto_bui... [6] > bootstrap.py

and then do the build with:

C:Python27python.exe bootstrap.py --builder=cctbx

this works for me. I did not encounter the RuntimeError you mentioned.

Let me know if it doesn't work,

Rob

On 06/04/2017 12:28, Horst Puschmann wrote: Hello Rob,

Yes, this is the command I am using:

C:Python27python.exe bootstrap.py --builder=cctbx

Everything works fine (and I don't think it's too slow!) -- but that last call to python can not work, since it calls a directory:

raise RuntimeError("Could not run %s: File not found" % executable) RuntimeError: Could not run basebinpython: File not found

if it called

basebinpythonpython instead, it would work.

Of course, I don't know what might not work after that step....

Horst

On 6 April 2017 at 12:17, R.D. Oeffner wrote:

Hi Horst and Markus,

Horst are you using a command line such as:

C:Pythonpython.exe bootstrap.py --builder=cctbx

I realise this is broken because I never tested rigorously for anonymous builds of CCTBX on Windows. Am in the process of fixing this.

Markus you are correct that the roundabout way to get hot packages from CCI is because rsync does not exist on Windows. With Git installed on windows curl and a few other Unix tools do exist. This was a quick solution for making a Windows build Phenix, Dials and CCTBX where it has been running smoothly since August last year. The Robocopy calls for deleting files is a solution, albeit cumbersome for deleting very long file paths on Windows (>255 characters). Pythons shutil.rmtree() will fail in such cases. File deletion on Windows is always much slower than on UNIX by design.

Making bootstrap.py and Buildbot work on Windows has been taken up a lot of work given that Buildbot seems to has been designed in denial of the existence of Windows and Microsoft until recently have been in denial of open source solutions.

Rob

On 06/04/2017 10:50, [email protected] wrote:

Hi Horst, Rob

I had a brief look at the bootstrap script. If I understand it correctly only authenticated access is allowed at the moment due to the unique way the download archives are constructed.

I am not entirely sure why the sequence to download the 'hot' packages (I did not venture further into the bootstrap process) consists of * ssh'ing to CCI servers * creating a TAR archive * downloading said archive via SCP * unpacking the archive instead of going down the conventional route of * downloading the default TAR archive via HTTP (yes, they're called .gz because someone was lazy, but they are .tar.gz files) * unpacking the archive

I suspect that this is due to the absence of CURL on Windows machines combined with how the bootstrap script is intertwined with the buildbot testing system. Internally, the bootstrap script uses an "_add_curl" call to download files. When run within the buildbot testing system this presumably gets replaced by an actual call to the 'curl' program, which doesn't exist on Windows, and so would fail. However, when you run bootstrap.py directly _add_curl actually uses an internal downloading routine, which works. (If true, this would simply be one more example of why buildbot determining the layout of bootstrap is both limiting and dangerous. The solution for that of course would be to install CURL for Windows (and/or replace buildbot))

So I guess the two lines in add_module, https://github.com/cctbx/cctbx_project/blob/master/libtbx/auto_build/bootstr... [1] if self.isPlatformWindows(): tarurl, arxname, dirpath = MODULES.get_module(module)().get_tarauthenticated(auth=self.get_auth()) could be removed along with all references to tarauthenticated?

As Horst already said there is more going wrong here. The ROBOCOPY calls are *incredibly* slow for something that could/should be done in pure python instead (shutil.rmtree?). The ssl 'workaround' uses internal methods, which are not available on older versions of Python.

The error at the end is presumably that the python interpreter is called python.exe on Windows?

-Markus

-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of R. D. Oeffner Sent: 06 April 2017 10:18 To: [email protected]; [email protected] Subject: Re: [cctbxbb] bootstrap.py on windows

Dear Horst,

I'll look into that. FYI you can also get Windows builds of CCTBX. See email below. You may have to resort using bundles from a few days ago because the two last nightly builds on windows are broken.

Rob

-----Original Message----- From: R. D. Oeffner Sent: Sunday, March 19, 2017 3:49 PM To: Billy Poon ; [email protected] Subject: Re: Windows builds for cctbx

There should now be CCTBX builds for Windows in the most recent folders on http://cci.lbl.gov/cctbx_build [2] . They are just the sources and the build directory zipped into an archive and does not include an installation program like the graphical Phenix installer for Windows. Presumably people wanting to use only CCTBX are sufficiently computer savvy to cope with this.

Rob

-- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY

www.cimr.cam.ac.uk/investigators/read/index.html [3] tel: +44(0)1223 763234 [4] mobile: +44(0)7712 887162 [5] -----Original Message----- From: Horst Puschmann Sent: Thursday, April 6, 2017 9:59 AM To: [email protected] Subject: [cctbxbb] bootstrap.py on windows

Hello

I am trying to install the cctbx on a Windows 7 64 bit machine using the following bootstrap.py file:

https://raw.githubusercontent.com/cctbx/cctbx_project/master/libtbx/auto_bui... [6]

The script will fail, unless I comment out lines 1101 and 1102:

#if self.isPlatformWindows(): #tarurl, arxname, dirpath = MODULES.get_module(module)().get_tarauthenticated(auth=self.get_auth())

If it tries to execute the commented lines, the error will be "KeyError: 'cciuser' in line 590"

After commenting this out, it will fail in line 271, with "AttributeError: 'module' object has no attribute '_create_unverified_context'"

I can get round that with disabling line 247:

if sys.platform == "win32":

(i.e. change this to if sys.platform == "xxx":)

After that, things start downloading. All appears well until the same thing happens again in another bootstrap.py file in modulescctbx_projectlibtbxauto_buildbootstrap.py

If I disable *that*, it goes further but fails finally with

===== Running in build: run configure.py Traceback (most recent call last): File "bootstrap.py", line 2113, in <module> run() File "bootstrap.py", line 2108, in run enable_shared=options.enable_shared, File "bootstrap.py", line 1036, in run i.run() File "bootstrap.py", line 189, in run raise RuntimeError("Could not run %s: File not found" % executable) RuntimeError: Could not run basebinpython: File not found

I guess the automatic tests haven't picked this up, because no authentication is needed, maybe?

Greetings Horst

Virus-free. www.avg.com [7]

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb [8]

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb [8]

Links: ------ [1] https://github.com/cctbx/cctbx_project/blob/master/libtbx/auto_build/bootstr... [2] http://cci.lbl.gov/cctbx_build [3] http://www.cimr.cam.ac.uk/investigators/read/index.html [4] tel:+44%201223%20763234 [5] tel:+44%207712%20887162 [6] https://raw.githubusercontent.com/cctbx/cctbx_project/master/libtbx/auto_bui... [7] http://www.avg.com [8] http://phenix-online.org/mailman/listinfo/cctbxbb

Hi Horst, As far as I can tell your build platform is not configured properly. Git has to be installed before running bootstrap. The old documents on how to build on windows are very out of date. I have attached some new notes to that effect. Markus and Richard, Do you think these could go on the wiki somewhere obvious to people wanting to build CCTBX on Windows. The source is in rst format if that is any good for the wiki. Rob On 07/04/2017 13:46, Horst Puschmann wrote:

Hello Rob,

It turns out that the bootstrap script does not include the git repository. So there is nothing to 'git pull' on. With Luc's help (Thank you, Luc!) we got everything going up to the point where libtbx.scons failed.

c:cctbx_17build>binlibtbx.scons -j4 TERSE=1 [...]

c:cctbx_17basehdf5-1.8.16includeH5public.h(158) : error C2371: 'ssize_t' : redefinition; different basic types c:cctbx_17basebinpythonincludepyconfig.h(205) : see declaration of 'ssize_t' scons: *** [cbflibpycbfpycbf_wrap.obj] Error 2 Compiling cbflibsrccbf_compress.obj

Luc suggested to fun binlibtbx.configure --only smtbx iotbx (the only things I am interested in right now) -- but it exits with

Copying to lib: "c:cctbx_17modulescbflibpycbfpycbf.py"

Processing: "c:cctbx_17modulescctbx_projectiotbxlibtbx_refresh.py" Using fable to convert iotbxpdbhybrid_36_f.f Writing: iotbxpdbhybrid_36_fem.cpp This script is only applicable to Linux - exiting. Done.

Luc pinpointed the problem down to:

the problem is, I believe, in libtbx/auto_build/regenerate_module_files.py

at line 146

I guess I better leave it at that right now!

Greetings Horst

On 6 April 2017 at 16:21, R.D. Oeffner wrote:

Hi Horst,

I encountered this error too and committed a fix half an hour ago to cctbx_projectcbflib_adaptbxmsvc_includeunistd.h. If you git pull cctbx_project and build with libtbx.scons again it should compile.

Rob

On 06/04/2017 15:54, Horst Puschmann wrote: Hello Rob,

This works fine -- Thank You!

... but then something goes wrong in the compilation:

pycbf_wrap.c c:cctbx_17basehdf5-1.8.16includeH5public.h(158) : error C2371: 'ssize_t' : redefinition; different basic types c:cctbx_17basebinpythonincludepyconfig.h(205) : see declaration of 'ssize_t' scons: *** [cbflibpycbfpycbf_wrap.obj] Error 2 scons: building terminated because of errors. usr+sys time: 9.47 seconds wall clock time: 4 minutes 22.11 seconds (262.11 seconds total) Process failed with return code 2

;)

Horst

On 6 April 2017 at 14:37, R.D. Oeffner wrote:

Hi Horst,

I just committed a fix to bootstrap.py to allow anonymous build. If you get it with:

curl https://raw.githubusercontent.com/cctbx/cctbx_project/master/libtbx/auto_bui... [6] > bootstrap.py

and then do the build with:

C:Python27python.exe bootstrap.py --builder=cctbx

this works for me. I did not encounter the RuntimeError you mentioned.

Let me know if it doesn't work,

Rob

On 06/04/2017 12:28, Horst Puschmann wrote: Hello Rob,

Yes, this is the command I am using:

C:Python27python.exe bootstrap.py --builder=cctbx

Everything works fine (and I don't think it's too slow!) -- but that last call to python can not work, since it calls a directory:

raise RuntimeError("Could not run %s: File not found" % executable) RuntimeError: Could not run basebinpython: File not found

if it called

basebinpythonpython instead, it would work.

Of course, I don't know what might not work after that step....

Horst

On 6 April 2017 at 12:17, R.D. Oeffner wrote:

Hi Horst and Markus,

Horst are you using a command line such as:

C:Pythonpython.exe bootstrap.py --builder=cctbx

I realise this is broken because I never tested rigorously for anonymous builds of CCTBX on Windows. Am in the process of fixing this.

Markus you are correct that the roundabout way to get hot packages from CCI is because rsync does not exist on Windows. With Git installed on windows curl and a few other Unix tools do exist. This was a quick solution for making a Windows build Phenix, Dials and CCTBX where it has been running smoothly since August last year. The Robocopy calls for deleting files is a solution, albeit cumbersome for deleting very long file paths on Windows (>255 characters). Pythons shutil.rmtree() will fail in such cases. File deletion on Windows is always much slower than on UNIX by design.

Making bootstrap.py and Buildbot work on Windows has been taken up a lot of work given that Buildbot seems to has been designed in denial of the existence of Windows and Microsoft until recently have been in denial of open source solutions.

Rob

On 06/04/2017 10:50, [email protected] wrote:

Hi Horst, Rob

I had a brief look at the bootstrap script. If I understand it correctly only authenticated access is allowed at the moment due to the unique way the download archives are constructed.

I am not entirely sure why the sequence to download the 'hot' packages (I did not venture further into the bootstrap process) consists of * ssh'ing to CCI servers * creating a TAR archive * downloading said archive via SCP * unpacking the archive instead of going down the conventional route of * downloading the default TAR archive via HTTP (yes, they're called .gz because someone was lazy, but they are .tar.gz files) * unpacking the archive

I suspect that this is due to the absence of CURL on Windows machines combined with how the bootstrap script is intertwined with the buildbot testing system. Internally, the bootstrap script uses an "_add_curl" call to download files. When run within the buildbot testing system this presumably gets replaced by an actual call to the 'curl' program, which doesn't exist on Windows, and so would fail. However, when you run bootstrap.py directly _add_curl actually uses an internal downloading routine, which works. (If true, this would simply be one more example of why buildbot determining the layout of bootstrap is both limiting and dangerous. The solution for that of course would be to install CURL for Windows (and/or replace buildbot))

So I guess the two lines in add_module, https://github.com/cctbx/cctbx_project/blob/master/libtbx/auto_build/bootstr... [1] if self.isPlatformWindows(): tarurl, arxname, dirpath = MODULES.get_module(module)().get_tarauthenticated(auth=self.get_auth()) could be removed along with all references to tarauthenticated?

As Horst already said there is more going wrong here. The ROBOCOPY calls are *incredibly* slow for something that could/should be done in pure python instead (shutil.rmtree?). The ssl 'workaround' uses internal methods, which are not available on older versions of Python.

The error at the end is presumably that the python interpreter is called python.exe on Windows?

-Markus

-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of R. D. Oeffner Sent: 06 April 2017 10:18 To: [email protected]; [email protected] Subject: Re: [cctbxbb] bootstrap.py on windows

Dear Horst,

I'll look into that. FYI you can also get Windows builds of CCTBX. See email below. You may have to resort using bundles from a few days ago because the two last nightly builds on windows are broken.

Rob

-----Original Message----- From: R. D. Oeffner Sent: Sunday, March 19, 2017 3:49 PM To: Billy Poon ; [email protected] Subject: Re: Windows builds for cctbx

There should now be CCTBX builds for Windows in the most recent folders on http://cci.lbl.gov/cctbx_build [2] . They are just the sources and the build directory zipped into an archive and does not include an installation program like the graphical Phenix installer for Windows. Presumably people wanting to use only CCTBX are sufficiently computer savvy to cope with this.

Rob

-- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY

www.cimr.cam.ac.uk/investigators/read/index.html [3] tel: +44(0)1223 763234 [4] mobile: +44(0)7712 887162 [5] -----Original Message----- From: Horst Puschmann Sent: Thursday, April 6, 2017 9:59 AM To: [email protected] Subject: [cctbxbb] bootstrap.py on windows

Hello

I am trying to install the cctbx on a Windows 7 64 bit machine using the following bootstrap.py file:

https://raw.githubusercontent.com/cctbx/cctbx_project/master/libtbx/auto_bui... [6]

The script will fail, unless I comment out lines 1101 and 1102:

#if self.isPlatformWindows(): #tarurl, arxname, dirpath = MODULES.get_module(module)().get_tarauthenticated(auth=self.get_auth())

If it tries to execute the commented lines, the error will be "KeyError: 'cciuser' in line 590"

After commenting this out, it will fail in line 271, with "AttributeError: 'module' object has no attribute '_create_unverified_context'"

I can get round that with disabling line 247:

if sys.platform == "win32":

(i.e. change this to if sys.platform == "xxx":)

After that, things start downloading. All appears well until the same thing happens again in another bootstrap.py file in modulescctbx_projectlibtbxauto_buildbootstrap.py

If I disable *that*, it goes further but fails finally with

===== Running in build: run configure.py Traceback (most recent call last): File "bootstrap.py", line 2113, in <module> run() File "bootstrap.py", line 2108, in run enable_shared=options.enable_shared, File "bootstrap.py", line 1036, in run i.run() File "bootstrap.py", line 189, in run raise RuntimeError("Could not run %s: File not found" % executable) RuntimeError: Could not run basebinpython: File not found

I guess the automatic tests haven't picked this up, because no authentication is needed, maybe?

Greetings Horst

Virus-free. www.avg.com [7]

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb [8]

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb [8]

Links: ------ [1] https://github.com/cctbx/cctbx_project/blob/master/libtbx/auto_build/bootstr... [2] http://cci.lbl.gov/cctbx_build [3] http://www.cimr.cam.ac.uk/investigators/read/index.html [4] tel:+44%201223%20763234 [5] tel:+44%207712%20887162 [6] https://raw.githubusercontent.com/cctbx/cctbx_project/master/libtbx/auto_bui... [7] http://www.avg.com [8] http://phenix-online.org/mailman/listinfo/cctbxbb

Hi Rob,

...

Luc suggested to fun bin\libtbx.configure --only smtbx iotbx (the only things I am interested in right now) -- but it exits with

Copying to lib: "c:\cctbx_17\modules\cbflib\pycbf\pycbf.py" Processing: "c:\cctbx_17\modules\cctbx_project\iotbx\libtbx_refresh.py" Using fable to convert iotbx\pdb\hybrid_36_f.f Writing: iotbx\pdb\hybrid_36_fem.cpp This script is only applicable to Linux - exiting. Done.

As far as I can tell your build platform is not configured properly. Git has to be installed before running bootstrap. The old documents on how to build on windows are very out of date. I have attached some new notes to that effect.

Do you mean that libtbx.configure is not supposed to work anymore on an existing installation? Really? Best wishes, Luc