==============================
How to build CCTBX on Windows
==============================

.. contents::

Introduction
============

This document describes how to build CCTBX from sources on Windows 7 or more recent versions. 
We cover how to build manually as well how to build automatically with BuildBot.
Given that Windows does not have all the necessary tools for building CCTBX, how to set up
these tools is covered here in some detail.

A typical CCTBX installation takes up the order of 1Gb disk space. If you build CCTBX 
from scratch including an installer this will be of the order of 6Gb.
Throughout this document the "command prompt" (archaichally termed the "DOS prompt" by some)
refers to the commandline interpreter program, cmd.exe, which serves the
same purpose as bash or tcsh on Unix. On the Startup menu it is located in the "Accessories" 
group.


Necessary prerequisites
=======================

To build CCTBX from scratch the following tools are required:

 - Python 2.7 (any version)
 - Microsoft Visual C++ 9.0 / Microsoft Visual Studio 2008 
 - TortoiseSVN with command line tools ( for updating sources )
 - TortoiseGit (optionally)
 - Git ( for updating sources and providing some unix command line tools)
 - mtee.exe (optionally for duplicating stdout to file and console)


Visual C++ 9.0
---------------

Microsoft Visual Studio 2008 compiler is available for free in the Windows 7 
SDK from the Microsoft website:
https://www.microsoft.com/en-gb/download/details.aspx?id=3138&fa43d42b-25b5-4a42-fe9b-1634f450f5ee=True . 
It should be titled "Microsoft Windows SDK for Windows 7 and .NET Framework 3.5 SP1". 
Rather than doing the automated installation from the website it may be convenient to 
download the DVD iso image of the SDK and then do the installation
from that. The 32 bit version is named
``GRMSDK_EN_DVD.iso`` whereas the 64 bit version is named ``GRMSDKX_EN_DVD.iso``.
There are newer versions of this SDK for Windows 7. But as these use more recent 
C-runtime libraries compiled programs may not be binary compatible with Python 2.7 which
is compiled with Visual C++ 9.0. You would still be able to build individual statically 
linked executables with those newer compilers. But whether or not compiled python modules 
can run without crashing using a newer runtime is questionable. After installing the 
compiler it can be invoked once environment variables have been set by calling scripts 
such as ``vcvars32.bat`` or ``vcvars64.bat`` from a command prompt or in a build script prior 
to compiling. These files live in the folder
::

  C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin

OpenSSH tools
--------------

The OpenSSH tools used for the Windows build are the ones that come with Git for Windows.
If Git has been installed in 
::

  C:\Program Files\Git

then OpenSSH tools live in the directory 
::

  C:\Program Files\Git\usr\bin

which you must add to the %PATH% environment variable. To have the build run without
interrupts the private keys matching your public keys on the remote svn or git repositories
must be loaded prior to the build with
ssh-agent. Create a .ssh directory in the directory pointed to by the 
%USERPROFILE% environment variable and store your private keys there.
Then add the lines

.. raw:: html  

  <div id="Layer1" style=
  "position:relative; width:90%; height:70px;  left: 4px; top: 0px; overflow: auto; background: #eee;">
  <pre class="code">
  call "C:\Program Files\Git\cmd\start-ssh-agent.cmd"
  SETX SSH_AUTH_SOCK "%SSH_AUTH_SOCK%"
  SETX SSH_AGENT_PID "%SSH_AGENT_PID%"
  </pre>
  </div>

to your **logon.cmd** file. When you log on it will pick up the private keys in your %USERPROFILE%\\.ssh 
folder. 
If keys require password you will be prompted for them when you logon to the PC. If a key, say 
``mykey_rsa``, is not picked up it can be loaded explicitly by adding the line

.. raw:: html  

  <div id="Layer1" style=
  "position:relative; width:90%; height:30px;  left: 4px; top: 0px; overflow: auto; background: #eee;">
  <pre class="code">
  ssh-add mykey_rsa
  </pre>
  </div>

just below the above 3 lines.



Manual build
============

From an existing CCTBX installation
------------------------------------

If you want to build CCTBX from an existing installation on Windows located say in 
``C:\Users\Mike\CCTBX\cctbx-installer-dev-983`` and do not intend to get updated sources
from remote repositories follow these steps:

  1. Open a command prompt and go to ``C:\users\Mike\CCTBX\cctbx-installer-dev-983\build``.
  #. Run one of the ``vcvarsXX.bat`` scripts `depending on the platform <CCTBXBuild.html#Visual-c-9-0>`__ .
  #. Run ``setpaths.bat``
  #. Run libtbx.scons -j <nproc>.

From bootstrap.py
-------------------

Below is a cmd script for manually building CCTBX with bootstrap.py. It first downloads the base
components, HDF5 and Python, installs them, then downloads sources
from external repositories, compiles them, runs tests and finally creates a zipped up bundle
of the sources and the compiled CCTBX files. It is assumed the script is started with command 
line arguments specifying platform and ooptionally a version label.

**CCTBXBuild.cmd**

.. raw:: html  

  <div id="Layer1" style=
  "position:relative; width:90%; height:400px;  left: 4px; top: 0px; overflow: auto; background: #eee;">
  <pre class="code">
  @REM Call script with argument x32 or x64 to specify platform. Optional 2nd argument is the release number, e.g. 123.456
  @REM
  
  SETLOCAL ENABLEDELAYEDEXPANSION
  
  @REM enable the platform specific compiler and python executable
  
  set MYPLATFORM=%1
  
  IF DEFINED MYPLATFORM (
  IF %MYPLATFORM% == x32 (
    call "C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\vcvars32.bat"
    set PYTHONEXE=C:\Python27_32\python.exe
  )
  
  IF %MYPLATFORM% == x64 (
    call "C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\vcvars64.bat"
    set PYTHONEXE=C:\Python27_64\python.exe
  )
  )
  
  @REM                  Platform specific compiler and python executable now set
  mkdir %MYPLATFORM%
  cd %MYPLATFORM%
  
  %PYTHONEXE% -c "import time; print '-' + str((int((time.time() - 1404950400) / (24*60*60))))" > buildnumber.txt
  SET /p BUILDNUMBER= < buildnumber.txt
  %PYTHONEXE% -c "import time; print 'dev-' + str((int((time.time() - 1404950400) / (24*60*60))))" > version.txt
  
  SET RELEASENUMBER=%2
  IF DEFINED RELEASENUMBER (
    SET CCTBXVERSION=!RELEASENUMBER!!BUILDNUMBER!
  ) ELSE (
    SET CCTBXVERSION=dev!BUILDNUMBER!
  )
  
  title Bootstrap %CCTBXVERSION% on %MYPLATFORM%
  
  @REM mkdir %CCTBXVERSION%
  @REM cd %CCTBXVERSION%
  mkdir Current
  del *.log
  cd Current
  
  @echo %DATE% %TIME% > ..\build%CCTBXVERSION%-%MYPLATFORM%.log
  
  @REM                           get latest bootstrap.py file
  set GETBOOTSTRAP=%3
  IF DEFINED GETBOOTSTRAP (
     @echo Get bootstrap.py | mtee /+ ..\build%CCTBXVERSION%-%MYPLATFORM%.log
     ( curl https://raw.githubusercontent.com/cctbx/cctbx_project/master/libtbx/auto_build/bootstrap.py > bootstrap.py ) 2>&1 ^
     | mtee /+ ..\build%CCTBXVERSION%-%MYPLATFORM%.log
  )
  
  @REM   with no flags bootstrap defaults to doing cleanup, hot, update, base and build stages
  title Bootstrap %CCTBXVERSION% on %MYPLATFORM%  build
  @echo %DATE% %TIME% >> ..\build%CCTBXVERSION%-%MYPLATFORM%.log
  %PYTHONEXE% bootstrap.py --builder=cctbx --nproc=10  2>&1 ^
   | mtee /+ ..\build%CCTBXVERSION%-%MYPLATFORM%.log
  IF ERRORLEVEL 1 (
    GOTO Makesummary
  ) 
  @echo %DATE% %TIME% | mtee /+ ..\build%CCTBXVERSION%-%MYPLATFORM%.log
  
  @REM                             run tests
  title Bootstrap %CCTBXVERSION% on %MYPLATFORM% tests
  @echo %DATE% %TIME% > ..\tests%CCTBXVERSION%-%MYPLATFORM%.log
  %PYTHONEXE% bootstrap.py --builder=cctbx --nproc=10 tests 2>&1 ^
   | mtee /+ ..\tests%CCTBXVERSION%-%MYPLATFORM%.log
  IF ERRORLEVEL 1 (
    GOTO Makesummary
  ) 
  @echo %DATE% %TIME% | mtee /+ ..\tests%CCTBXVERSION%-%MYPLATFORM%.log
  
  @REM                            create installer
  title Bootstrap %CCTBXVERSION% on %MYPLATFORM% create_installer
  @echo %DATE% %TIME% > ..\CreateInstaller%CCTBXVERSION%-%MYPLATFORM%.log
  call build\bin\libtbx.create_installer.bat --binary --version %CCTBXVERSION% ^
   --install_script modules\cctbx_project\libtbx\auto_build\plus_installer.py ^
   --dist_dir dist\%CCTBXVERSION% tmp/cctbx-installer-%CCTBXVERSION%-win7vc90 2>&1 ^
    | mtee /+ ..\CreateInstaller%CCTBXVERSION%-%MYPLATFORM%.log
  IF ERRORLEVEL 1 (
    GOTO Makesummary
  ) 
  @echo %DATE% %TIME% | mtee /+ ..\CreateInstaller%CCTBXVERSION%-%MYPLATFORM%.log
  
  :Makesummary
  @echo %CCTBXVERSION%-%MYPLATFORM% > ..\summary.log
  %PYTHONEXE% -c "lines = open('..\\build%CCTBXVERSION%-%MYPLATFORM%.log','r').readlines(); lastlines = lines[(len(lines) - 5): ]; print ''.join(lastlines); " >> ..\summary.log
  %PYTHONEXE% -c "lines = open('..\\tests%CCTBXVERSION%-%MYPLATFORM%.log','r').readlines(); lastlines = lines[(len(lines) - 5): ]; print ''.join(lastlines); " >> ..\summary.log
  %PYTHONEXE% -c "lines = open('..\\CreateInstaller%CCTBXVERSION%-%MYPLATFORM%.log','r').readlines(); lastlines = lines[(len(lines) - 20): ]; print ''.join(lastlines); " >> ..\summary.log
  
  REM print concatenated summary of logfiles in a message box
  type ..\summary.log | msg %USERNAME% /time:86400
  
  @ENDLOCAL
  EXIT
  </pre>
  </div>
  
In the above script mtee.exe from https://ritchielawrence.github.io/mtee/ is used for piping stdout to a 
file as well as to the console. 
It works similar to tee on Unix platforms. A newer version has been compiled that also retains the
error code of the program that streams to stdout. This is useful when checking for the success of 
the builds. Source code and executable is available from http://oeffner.net/development/Stuff/mtee.zip.


Useful tweaks for running as a Buildbot slave
=================================================================

Automatically run script for every command prompt
--------------------------------------------------

On Unix a startup script such as .bashrc or .tcshrc
are convenient for initiating commonly used environment variables for a user.
Windows has a different technique for achieving the same for the command prompt. 
In the registry editor, regedit.exe, locate the AutoRun registry key:
::

  HKEY_CURRENT_USER\Software\Microsoft\Command Processor\AutoRun

If there is no AutoRun key present under the "Command Processor" key then create a new one
of type string. The value of this key can be set to the full path of a cmd script you have 
written that does something such as defining environment variables for that command prompt. 
This script can have any name but we will call it **ntshell.cmd** and store it in path 
pointed to by the %HOMEDRIVE%%HOMEPATH% environment variables.

.. warning:: Entering wrong values in the registry or accidentally
  deleting values may render Windows bricked. Always ensure you have a recent "System Restore"
  point in your Windows backup. This will allow you to boot Windows to 
  "Last known good configuration" if the registry gets corrupted.

Alternatively one can also define environment variables through the 
Control Panel\\All Control Panel Items\\User Accounts and click on 
"Change my enviroment variables".
But this method is not feasible if there are many variables or some variable values are 
deduced on the fly from scripts. This is the case when initialising environment variables for
the compiler.

Run a script once at every logon
---------------------------------

Say you have a script in the directory pointed to by the %HOMEDRIVE%%HOMEPATH% environment 
variables named **logon.cmd** or whatever.
To make this script run once every time you log on to your account, open the Task Scheduler 
in the "Administrative tools" group in Control Panel, 
click "Create Task", set "Triggers" to "At logon" and set "Action" to the full path of the 
logon.cmd file. This script will then run next time you log on. 


Automated build with Buildbot
=============================

This section is relevant only if you plan to do automated builds using BuildBot.
Assuming the PC runs as a Buildbot slave builds are automated from the Buildbot master
machine. Required private keys should be loaded in a **logon.cmd** script with ssh-agent and 
command prompts must set the compiler environment variables through an **ntshell.cmd** script
as `detailed above <CCTBXBuild.html#automatically-run-script-for-every-command-prompt>`__.


Issue defining environment variables for the compiler
------------------------------------------------------

If doing an automated build with Buildbot
all the necessary prerequisites must be accessible from the %PATH% environment from a command prompt.
It is tempting to add the full path of one of the `vcvarsXX.bat scripts <CCTBXBuild.html#Visual-c-9-0>`__  to
**ntshell.cmd**. However, due to a bug in those files this may lead all command prompts to hang in
an infinite loop.
A workaround is to copy vcvars32.bat and vcvars64.bat to the %HOMEDRIVE%%HOMEPATH% folder and
rename them myvcvars32.bat and myvcvars64.bat, respectively. Then replace the lines:

.. raw:: html  

  <div id="Layer1" style=
  "position:relative; width:90%; height:150px;  left: 4px; top: 0px; overflow: auto; background: #eee;">
  <pre class="code">
  :GetWindowsSdkDirHelper
  @for /F "tokens=1,2*" %%i in ('reg query "%1\SOFTWARE\Microsoft\Microsoft SDKs\Windows" /v "CurrentInstallFolder"') DO (
    if "%%i"=="CurrentInstallFolder" (
      SET "WindowsSdkDir=%%k"
    )
  )
  </pre>
  </div>

with

.. raw:: html  

  <div id="Layer1" style=
  "position:relative; width:90%; height:220px;  left: 4px; top: 0px; overflow: auto; background: #eee;">
  <pre class="code">
  :GetWindowsSdkDirHelper
  SET fname=%TEMP%\tmpSDKvars.txt
  reg query "%1\SOFTWARE\Microsoft\Microsoft SDKs\Windows" /v "CurrentInstallFolder" > %fname%
  
  @SET WindowsSdkDir=
  @for /F "tokens=1,2*" %%i in (%fname%) DO (
    if "%%i"=="CurrentInstallFolder" (
      SET "WindowsSdkDir=%%k"
    )
  )
  del %fname%
  </pre>
  </div>

then add a line to your **ntshell.cmd** with the full path of myvcvars32.bat or myvcvars64.bat
like:
::

  call "C:\Users\builder\myvcvars32.bat"




Setting up a PC as a Buildbot slave
------------------------------------

The Buildbot slave is installed as per  
`online documentation <http://trac.buildbot.net/wiki/DownloadInstall>`__. For instance 
for installing a slave named intel-windows-x86_64 in C:\\Users\\builder\\slave chdir to python
installed in C:\\Python2.7.9x64 type:

.. raw:: html  

  <div id="Layer1" style=
  "position:relative; width:90%; height:50px;  left: 4px; top: 0px; overflow: auto; background: #eee;">
  <pre class="code">
  C:\Python2.7.9x64\Scripts\easy_install.exe buildbot-slave
  </pre>
  </div>

and then:

.. raw:: html  

  <div id="Layer1" style=
  "position:relative; width:90%; height:50px;  left: 4px; top: 0px; overflow: auto; background: #eee;">
  <pre class="code">
  C:\Python2.7.9x64\Scripts\buildslave create-slave C:\Users\builder\slave cci-vm-6.lbl.gov:9989 intel-windows-x86_64 pass
  </pre>
  </div>

which will make it listen to a master on port 9989 from cci-vm-6.lbl.gov. It is possible to
start this slave automatically as a Windows service (equivalent of a Unix daemon) but a simpler
procedure is to add the following line:

.. raw:: html  

  <div id="Layer1" style=
  "position:relative; width:90%; height:50px;  left: 4px; top: 0px; overflow: auto; background: #eee;">
  <pre class="code">
  START "Running BuildBot Slave. Do not close!" /MIN cmd /c C:\Python2.7.9x64\Scripts\buildslave start C:\Users\builder\slave
  </pre>
  </div>

to your **logon.cmd** script. Once you have logged on this will place an icon of a minimised command 
prompt on the taskbar which must not be interfered with, let alone closed. The account can now be 
locked (Windows key + L) and is ready for doing regular builds from a Buildbot master.

Bundling base components together as zip files
===============================================

The Windows build does not need to compile Python or the HDF5 library from scratch.
Instead these two components are bundled as separate zip files which bootstrap.py 
will download from
`http://cci.lbl.gov/cctbx_dependencies <http://cci.lbl.gov/cctbx_dependencies>`__ during the
first stages of the build. The Python bundle has had all the necessary 3rd party modules 
added to it for running CCTBX on Windows.
`Click here <cctbx_python_HDF5_bundles.html>`__ for details on how these bundles were created
or on how to update this Python bundle with additional modules.



Important tweaks for shortening duration of builds
===================================================

Clear virtual memory at shutdown:
Go to Control Panel\\All Control Panel Items\\Administrative Tools->Local Security Policy->Local Policies->Security Options->Shutdown: Clear virtual memory pagefile

Adjust for best performance of background processes:
Check Control Panel\\All Control Panel Items\\System->Advanced system settings->System Properties | Advanced tab->Performance settings->Performance Options | Advanced tab->Adjust for best performance of Background services

Ensure at least weekly disk defragmentation which does not coincide with doing builds.

If used for nightly builds then disable automatic installation of updates as this often triggers an 
automatic reboot of the PC. This can be done from the control panel on Windows 7. On
Windows 10 follow the steps described on http://www.howtogeek.com/224471/how-to-prevent-windows-10-from-automatically-downloading-updates/

For the directory where nightly builds are done:

 - Exclude it from Windows indexing. 
 - Exclude it from on-access virus scan (both read and write)
 - Avoid storing several previous builds since NTFS doesn't perform well 
   when zillions of files are present on the same disc. Zip up previous builds 
   into single files or move them to a different partition if they have to be retained.


Failure to follow these steps could over time make successive builds very slow (+24 hours)