[cctbxbb] bootstrap.py on windows

Thu Apr 6 02:50:52 PDT 2017

Hi Horst, Rob

I had a brief look at the bootstrap script. If I understand it correctly only authenticated access is allowed at the moment due to the unique way the download archives are constructed.

I am not entirely sure why the sequence to download the 'hot' packages (I did not venture further into the bootstrap process) consists of
 * ssh'ing to CCI servers
 * creating a TAR archive
 * downloading said archive via SCP
 * unpacking the archive
instead of going down the conventional route of
 * downloading the default TAR archive via HTTP (yes, they're called .gz because someone was lazy, but they are .tar.gz files)
 * unpacking the archive

I suspect that this is due to the absence of CURL on Windows machines combined with how the bootstrap script is intertwined with the buildbot testing system. Internally, the bootstrap script uses an "_add_curl" call to download files. When run within the buildbot testing system this presumably gets replaced by an actual call to the 'curl' program, which doesn't exist on Windows, and so would fail. However, when you run bootstrap.py directly _add_curl actually uses an internal downloading routine, which works.
(If true, this would simply be one more example of why buildbot determining the layout of bootstrap is both limiting and dangerous. The solution for that of course would be to install CURL for Windows (and/or replace buildbot))

So I guess the two lines in add_module, https://github.com/cctbx/cctbx_project/blob/master/libtbx/auto_build/bootstrap.py#L1124
    if self.isPlatformWindows():
       tarurl, arxname, dirpath = MODULES.get_module(module)().get_tarauthenticated(auth=self.get_auth())
could be removed along with all references to tarauthenticated?

As Horst already said there is more going wrong here. The ROBOCOPY calls are *incredibly* slow for something that could/should be done in pure python instead (shutil.rmtree?). The ssl 'workaround' uses internal methods, which are not available on older versions of Python.

The error at the end is presumably that the python interpreter is called python.exe on Windows?

-Markus

-----Original Message-----
From: cctbxbb-bounces at phenix-online.org [mailto:cctbxbb-bounces at phenix-online.org] On Behalf Of R. D. Oeffner
Sent: 06 April 2017 10:18
To: horst.puschmann at gmail.com; cctbxbb at phenix-online.org
Subject: Re: [cctbxbb] bootstrap.py on windows

Dear Horst,

I'll look into that. FYI you can also get Windows builds of CCTBX. See email below. You may have to resort using bundles from a few days ago because the two last nightly builds on windows are broken.

Rob

-----Original Message-----
From: R. D. Oeffner
Sent: Sunday, March 19, 2017 3:49 PM
To: Billy Poon ; nwmoriarty at lbl.gov
Subject: Re: Windows builds for cctbx

There should now be CCTBX builds for Windows in the most recent folders on http://cci.lbl.gov/cctbx_build . They are just the sources and the build directory zipped into an archive and does not include an installation program like the graphical Phenix installer for Windows. Presumably people wanting to use only CCTBX are sufficiently computer savvy to cope with this.

Rob

--
Robert Oeffner, Ph.D.
Research Associate, The Read Group
Department of Haematology,
Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY

www.cimr.cam.ac.uk/investigators/read/index.html
tel: +44(0)1223 763234
mobile: +44(0)7712 887162
-----Original Message-----
From: Horst Puschmann
Sent: Thursday, April 6, 2017 9:59 AM
To: cctbxbb at phenix-online.org
Subject: [cctbxbb] bootstrap.py on windows

Hello

I am trying to install the cctbx on a Windows 7 64 bit machine using the 
following bootstrap.py file:

https://raw.githubusercontent.com/cctbx/cctbx_project/master/libtbx/auto_build/bootstrap.py

The script will fail, unless I comment out lines 1101 and 1102:

    #if self.isPlatformWindows():
      #tarurl, arxname, dirpath = 
MODULES.get_module(module)().get_tarauthenticated(auth=self.get_auth())

If it tries to execute the commented lines, the error will be "KeyError: 
'cciuser' in line 590"

After commenting this out, it will fail in line 271, with "AttributeError: 
'module' object has no attribute '_create_unverified_context'"

I can get round that with disabling line 247:

      if sys.platform == "win32":

(i.e. change this to       if sys.platform == "xxx":)

After that, things start downloading. All appears well until the same thing 
happens again in another bootstrap.py file in 
\modules\cctbx_project\libtbx\auto_build\bootstrap.py

If I disable *that*, it goes further but fails finally with

===== Running in build: run configure.py
Traceback (most recent call last):
  File "bootstrap.py", line 2113, in <module>
    run()
  File "bootstrap.py", line 2108, in run
    enable_shared=options.enable_shared,
  File "bootstrap.py", line 1036, in run
    i.run()
  File "bootstrap.py", line 189, in run
    raise RuntimeError("Could not run %s: File not found" % executable)
RuntimeError: Could not run base\bin\python: File not found

I guess the automatic tests haven't picked this up, because no 
authentication is needed, maybe?

Greetings
Horst

Virus-free. www.avg.com

_______________________________________________
cctbxbb mailing list
cctbxbb at phenix-online.org
http://phenix-online.org/mailman/listinfo/cctbxbb 

_______________________________________________
cctbxbb mailing list
cctbxbb at phenix-online.org
http://phenix-online.org/mailman/listinfo/cctbxbb

-- 
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. 
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom