Python-based
Hierarchical
ENvironment
for
Integrated
Xtallography
Contents
This document details how to create zip file bundles of python and the HDF5 library used for the bootstrap build on Windows of CCTBX as described here and other software relying on CCTBX such as Phenix or Dials. Unless you want to replace the Python or HDF5 bundle described here with your own versions you do not need to read this document. The bootstrap build will just download existing bundles from cci.lbl.gov.
The two base components are bundled as zip files to be downloaded when building for Windows. This is possible because binary compatibility of executables with Windows has been retained over successive versions of Windows apart from the distinction between the 32 and 64 bit platforms. Therefore these base components are necessary to build one time only. Zipped up bundles of these builds are then stored on http://cci.lbl.gov/cctbx_dependencies from where bootstrap.py will download them during the base stage of the build. This happens around lines 136 - 165 in cctbx_project\libtbx\auto_build\install_base_packages.py, as of 07 April 2017, git SHA-1: 6c494c2... The bootstrap build will uncompress the python bundle into base\bin\python and the HDF5 bundle into base\HDF5-1.8.16 relative to where the bootstrap build is done. Unlike the directory structure on Unix the respective library and header files of these two bundles are stored as sub directories within these directories and not in common base\include and base\lib directories.
The first base component bundled is Python 2.7.12 with installed modules such as matplotlib, numpy, biopython, wxPython, etc. Version 2.7.9 is the minimum version that provides tools like "pip install" which makes it much easier to enhance python with 3rd party modules from https://pypi.python.org.
The first step is to download and install python2.7.9 (or higher) to your PC. Unfortunately the Python for Windows installer is too clever for its own good. It will try to overwrite any existing Python installation on your PC as well as the registry key that specifies the existing installation, HKEY_CURRENT_USER\Software\Python\PythonCore\2.7\InstallPath
To deal with this the easiest thing may be to either uninstall the existing python or to make a backup of the registry key
HKEY_CURRENT_USER\Software\Python
and then delete it. After that, install Python into a folder containing no blank spaces in its name.
Warning
Entering wrong values in the registry or accidentally deleting values may render Windows bricked. Always ensure you have a recent "System Restore" point in your Windows backup. This will allow you to boot Windows to "Last known good configuration" if the registry gets corrupted.
Next we need to install wxPython package for the GUI. This is available as a binary installer from http://www.wxpython.org. Get version 3.0.2 and install it into your python installation. It should default to live in <your python folder>\\Lib\\site-packages.
The list of additional 3rd party python modules and their versions necessary for the python for Windows distributed with Phenix is:
Cython==0.22 PIL==1.1.7 Pillow==2.8.1 PyOpenGL==3.1.1a1 PyRTF==0.45 Pygments==2.1.3 biopython==1.65 comtypes==1.1.1 docutils==0.12 h5py==2.5.0 libsvm==3.20 matplotlib==1.4.3 numpy==1.9.2 psutil==2.2.1 py2exe==0.6.9 pyparsing==2.0.3 python-dateutil==2.4.2 pytz==2015.2 reportlab==3.1.44 scipy==0.15.1 six==1.9.0 jinja2==2.8
These can be added to your python installation as follows: In a command prompt change directory to the toplevel directory of your python installation and type:
python -m pip install moduleXYZ
where "moduleXYZ" is one of the above. Modules not found through the online database of python modules or those which have not been compiled with Visual C++ 2008 may be found on Christoph Gohlke's site. Make sure that the python version and compiler version of the modules matches 2.7 and Visual Studio 2008 respectively.
Now your customised python is ready to be bundled into a zip file. Open Windows Explorer, locate the folder above your python, right-click on your python folder and select "Send to Compressed (zipped) Folder". This will create a zip file named python.zip which contains everything in your customised python. Depending on the platform you are targetting rename this file to python2.7.9_x86_64_plus_relocatable.zip or python2.7.9_x86_32_plus_relocatable.zip. If you want to use make this zip bundle available to others then upload it to http://cci.lbl.gov/cctbx_dependencies . When a boostrap build is started bootstrap.py will download it during the base stage and uncompress it in the base subdirectory of your build location.
The second base component bundled is the HDF5 library which is necesary for Dials and Xia2. On Windows the build tool CMake from https://cmake.org/ must be installed first.
On this occassion it seems to be necessary to have the full Visual Studio 2008 installed. The free version of the compiler in the Windows 7 SDK seems to confuse CMake if other and newer Visual Studio compilers are present. This leads to error messages when attempting to build the library.
HDF5 can be downloaded from http://www.hdfgroup.org/ftp/HDF5/current/src/ such as a zipped file like CMake-hdf5-1.8.16.zip. Uncompress it on your PC.
In the folder CMake-hdf5-1.8.16 there should be cmd scripts such as build-VS2012-32.bat, build-VS2013-32.bat etc. but none targetting Visual Studio 2008, i.e. with "VS2008" as part of the filename. We need to adapt HDF5 to build with Visual Studio 2008. To do this open one of the existing cmd scripts such as build-VS2012-32.bat and replace the line
ctest -S HDF518config.cmake,BUILD_GENERATOR=VS2012 -C Release -V -O hdf518.log
with
ctest -S HDF518config.cmake,BUILD_GENERATOR=VS2008 -C Release -V -O hdf518.log
Save this file with the name "build-VS2008-32.bat". Now open the file "HDF518config.cmake". Locate the section:
# build generator must be defined
if(NOT DEFINED BUILD_GENERATOR)
message(FATAL_ERROR "BUILD_GENERATOR must be defined - Unix, VS200864, VS2013, VS201364, VS2012, or VS201264")
else()
if(${BUILD_GENERATOR} STREQUAL "Unix")
set(CTEST_CMAKE_GENERATOR "Unix Makefiles")
elseif(${BUILD_GENERATOR} STREQUAL "VS2015")
set(CTEST_CMAKE_GENERATOR "Visual Studio 14 2015")
elseif(${BUILD_GENERATOR} STREQUAL "VS201564")
set(CTEST_CMAKE_GENERATOR "Visual Studio 14 2015 Win64")
elseif(${BUILD_GENERATOR} STREQUAL "VS2013")
set(CTEST_CMAKE_GENERATOR "Visual Studio 12 2013")
elseif(${BUILD_GENERATOR} STREQUAL "VS201364")
set(CTEST_CMAKE_GENERATOR "Visual Studio 12 2013 Win64")
elseif(${BUILD_GENERATOR} STREQUAL "VS2012")
set(CTEST_CMAKE_GENERATOR "Visual Studio 11 2012")
elseif(${BUILD_GENERATOR} STREQUAL "VS201264")
set(CTEST_CMAKE_GENERATOR "Visual Studio 11 2012 Win64")
else()
message(FATAL_ERROR "Invalid BUILD_GENERATOR must be - Unix, VS2013, VS201364, VS2012, or VS201264")
endif()
endif()
and add entries for VS2008 like:
# build generator must be defined
if(NOT DEFINED BUILD_GENERATOR)
message(FATAL_ERROR "BUILD_GENERATOR must be defined - Unix, VS2008, VS200864, VS2013, VS201364, VS2012, or VS201264")
else()
if(${BUILD_GENERATOR} STREQUAL "Unix")
set(CTEST_CMAKE_GENERATOR "Unix Makefiles")
elseif(${BUILD_GENERATOR} STREQUAL "VS2015")
set(CTEST_CMAKE_GENERATOR "Visual Studio 14 2015")
elseif(${BUILD_GENERATOR} STREQUAL "VS201564")
set(CTEST_CMAKE_GENERATOR "Visual Studio 14 2015 Win64")
elseif(${BUILD_GENERATOR} STREQUAL "VS2013")
set(CTEST_CMAKE_GENERATOR "Visual Studio 12 2013")
elseif(${BUILD_GENERATOR} STREQUAL "VS201364")
set(CTEST_CMAKE_GENERATOR "Visual Studio 12 2013 Win64")
elseif(${BUILD_GENERATOR} STREQUAL "VS2012")
set(CTEST_CMAKE_GENERATOR "Visual Studio 11 2012")
elseif(${BUILD_GENERATOR} STREQUAL "VS201264")
set(CTEST_CMAKE_GENERATOR "Visual Studio 11 2012 Win64")
elseif(${BUILD_GENERATOR} STREQUAL "VS2008")
set(CTEST_CMAKE_GENERATOR "Visual Studio 9 2008")
elseif(${BUILD_GENERATOR} STREQUAL "VS200864")
set(CTEST_CMAKE_GENERATOR "Visual Studio 9 2008 Win64")
else()
message(FATAL_ERROR "Invalid BUILD_GENERATOR must be - Unix, VS2013, VS201364, VS2012, or VS201264")
endif()
endif()
Now locate the section:
######### Following describes compiler ############
if(WIN32)
set(SITE_OS_NAME "Windows")
set(SITE_OS_VERSION "WIN7")
if(${BUILD_GENERATOR} STREQUAL "VS201564")
set(SITE_OS_BITS "64")
set(SITE_COMPILER_NAME "vs2015")
set(SITE_COMPILER_VERSION "14")
elseif(${BUILD_GENERATOR} STREQUAL "VS2015")
set(SITE_OS_BITS "32")
set(SITE_COMPILER_NAME "vs2015")
set(SITE_COMPILER_VERSION "14")
elseif(${BUILD_GENERATOR} STREQUAL "VS201364")
set(SITE_OS_BITS "64")
set(SITE_COMPILER_NAME "vs2013")
set(SITE_COMPILER_VERSION "12")
elseif(${BUILD_GENERATOR} STREQUAL "VS2013")
set(SITE_OS_BITS "32")
set(SITE_COMPILER_NAME "vs2013")
set(SITE_COMPILER_VERSION "12")
elseif(${BUILD_GENERATOR} STREQUAL "VS201264")
set(SITE_OS_BITS "64")
set(SITE_COMPILER_NAME "vs2012")
set(SITE_COMPILER_VERSION "11")
elseif(${BUILD_GENERATOR} STREQUAL "VS2012")
set(SITE_OS_BITS "32")
set(SITE_COMPILER_NAME "vs2012")
set(SITE_COMPILER_VERSION "11")
endif()
## Set the following to unique id your computer ##
and add entries for the VS2008 compiler like:
######### Following describes compiler ############
if(WIN32)
set(SITE_OS_NAME "Windows")
set(SITE_OS_VERSION "WIN7")
if(${BUILD_GENERATOR} STREQUAL "VS201564")
set(SITE_OS_BITS "64")
set(SITE_COMPILER_NAME "vs2015")
set(SITE_COMPILER_VERSION "14")
elseif(${BUILD_GENERATOR} STREQUAL "VS2015")
set(SITE_OS_BITS "32")
set(SITE_COMPILER_NAME "vs2015")
set(SITE_COMPILER_VERSION "14")
elseif(${BUILD_GENERATOR} STREQUAL "VS201364")
set(SITE_OS_BITS "64")
set(SITE_COMPILER_NAME "vs2013")
set(SITE_COMPILER_VERSION "12")
elseif(${BUILD_GENERATOR} STREQUAL "VS2013")
set(SITE_OS_BITS "32")
set(SITE_COMPILER_NAME "vs2013")
set(SITE_COMPILER_VERSION "12")
elseif(${BUILD_GENERATOR} STREQUAL "VS201264")
set(SITE_OS_BITS "64")
set(SITE_COMPILER_NAME "vs2012")
set(SITE_COMPILER_VERSION "11")
elseif(${BUILD_GENERATOR} STREQUAL "VS2012")
set(SITE_OS_BITS "32")
set(SITE_COMPILER_NAME "vs2012")
set(SITE_COMPILER_VERSION "11")
elseif(${BUILD_GENERATOR} STREQUAL "VS200864")
set(SITE_OS_BITS "64")
set(SITE_COMPILER_NAME "vs2008")
set(SITE_COMPILER_VERSION "9")
elseif(${BUILD_GENERATOR} STREQUAL "VS2008")
set(SITE_OS_BITS "32")
set(SITE_COMPILER_NAME "vs2008")
set(SITE_COMPILER_VERSION "9")
endif()
## Set the following to unique id your computer ##
Finally due to licensing terms of the optionally compiled SZIP library as detailed here: https://www.hdfgroup.org/doc_resource/SZIP/ this needs to be excluded from builds. To do that find the lines:
### disable ext libs building
#set(ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DHDF5_ENABLE_Z_LIB_SUPPORT:BOOL=OFF")
#set(ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DHDF5_ENABLE_SZIP_SUPPORT:BOOL=OFF")
#set(ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DHDF5_ENABLE_SZIP_ENCODING:BOOL=OFF")
and uncomment them:
### disable ext libs building
set(ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DHDF5_ENABLE_Z_LIB_SUPPORT:BOOL=OFF")
set(ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DHDF5_ENABLE_SZIP_SUPPORT:BOOL=OFF")
set(ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DHDF5_ENABLE_SZIP_ENCODING:BOOL=OFF")
The configuration is now good to build. Open a command prompt and change directory to the CMake-hdf5-1.8.16 folder with the amended build script files. Initialise the compilers environment variables by running
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\vcvars32.bat
or
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\vcvars64.bat
according to the platform you are targetting. CMake installed earlier may not be in the executable path in which case you can add it temporarily to your command prompt by entering:
set PATH=%PATH%;C:\Program Files\CMake\bin
Then run the build script by entering:
build-VS2008-32.bat
This should build the library, bundle it into a compressed zip file located in CMake-hdf5-1.8.16\build and named HDF5-1.8.16-win32.zip or HDF5-1.8.16-win64.zip depending on the platform. As with the python bundles these files should be uploaded to http://cci.lbl.gov/cctbx_dependencies so that bootstrap.py may fetch it during the base stage of the build.