Am 09.07.21 um 13:16 schrieb Winter, Graeme (DLSLtd,RAL,LSCI):

With DIALS we use python3 exclusively - have done so for a long while - everything we use from cctbx seems largely OK with this

Python2 is dead to me, and many other dials devs

Oh, that's great to hear, I really hope to have everything Python3.6+ soon as well. Still bootstrap.py seems to default to building a python 2.7.18 interpreter? Is this just for legacy backwards compatibility? All the best Jan

All the best Graeme

...

On 9 Jul 2021, at 11:49, Jan M. Simons wrote:

Cheers,

I'm getting more and more concerned with the state of cctbx and the impending removal of python2 from quite a lot of major linux distributions in the near future.

So far I have not managed to find documentation on how to build the cctbx to then use a python3 interpreter for the cctbx.python script, but I would really like to have this, as I'd have to check/port my code that's using cctbx modules (mostly smtbx and xrax) to python3 as well, so that it does not get obsoleted.

So is it possible to build cctbx as a python3 module by now and if so how would one be able to do this?

Thanks for your support Jan _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Hi Jan, Not an expert on cctbx bootstrap - dials uses it's own. But I think you need to explicitly pass "--python 37", and possibly "--use_conda" to the cctbx bootstrap.py; I don't know if the non-conda bootstrap was ever updated to use python 3. I can't imagine the python 3.7 non-conda build is very widely used at this point. To slightly elaborate on what Graeme said about DIALS support - for DIALS/dxtbx we do support 3.6+, but will be dropping 3.6 for 3.7+ after the next release (mid-august), as we are now following the numpy deprecation schedule: https://numpy.org/neps/nep-0029-deprecation_policy.html#support-table Nick On Fri, Jul 9, 2021 at 12:30 PM Jan M. Simons wrote:

Am 09.07.21 um 13:16 schrieb Winter, Graeme (DLSLtd,RAL,LSCI):

...

With DIALS we use python3 exclusively - have done so for a long while - everything we use from cctbx seems largely OK with this

Python2 is dead to me, and many other dials devs

Oh, that's great to hear, I really hope to have everything Python3.6+ soon as well.

Still bootstrap.py seems to default to building a python 2.7.18 interpreter? Is this just for legacy backwards compatibility?

All the best Jan

...

All the best Graeme

...

On 9 Jul 2021, at 11:49, Jan M. Simons wrote:

Cheers,

I'm getting more and more concerned with the state of cctbx and the impending removal of python2 from quite a lot of major linux distributions in the near future.

So far I have not managed to find documentation on how to build the cctbx to then use a python3 interpreter for the cctbx.python script, but I would really like to have this, as I'd have to check/port my code that's using cctbx modules (mostly smtbx and xrax) to python3 as well, so that it does not get obsoleted.

So is it possible to build cctbx as a python3 module by now and if so how would one be able to do this?

Thanks for your support Jan _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

On Fri, Jul 9, 2021 at 12:40 PM Winter, Graeme (DLSLtd,RAL,LSCI) wrote:

Dials has a custom bootstrap which gives you a recent 3 build. I forget if it’s 3.9 or some other. But it’s certainly up to date and supported.

We base our build environment on conda rather than building everything locally which really speeds things up

All the best Graeme

...

On 9 Jul 2021, at 12:30, Jan M. Simons wrote:

Am 09.07.21 um 13:16 schrieb Winter, Graeme (DLSLtd,RAL,LSCI):

...

With DIALS we use python3 exclusively - have done so for a long while - everything we use from cctbx seems largely OK with this

Python2 is dead to me, and many other dials devs

Oh, that's great to hear, I really hope to have everything Python3.6+ soon as well.

Still bootstrap.py seems to default to building a python 2.7.18 interpreter? Is this just for legacy backwards compatibility?

All the best Jan

...

All the best Graeme

...

...

On 9 Jul 2021, at 11:49, Jan M. Simons wrote:

Cheers,

I'm getting more and more concerned with the state of cctbx and the impending removal of python2 from quite a lot of major linux distributions in the near future.

So far I have not managed to find documentation on how to build the cctbx to then use a python3 interpreter for the cctbx.python script, but I would really like to have this, as I'd have to check/port my code that's using cctbx modules (mostly smtbx and xrax) to python3 as well, so that it does not get obsoleted.

So is it possible to build cctbx as a python3 module by now and if so how would one be able to do this?

Thanks for your support Jan _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Hi Jan, an old DIY method: Once you have all the dependencies in place (using either method): * create a directory where you want to build cctbx and make it current (assuming here it is on the same level as 'modules' dir) * python_of_your_choice ../modules/cctbx_project/libtx/configure.py smtbx fast_linalg anything_else * Linux - 'source setpaths.sh', Windows - 'setpaths.bat' * libtbx.scons -jN, N - number of simultaneous jobs to run To get the clean bundle: libtbs.python ../modules/cctbx_project/libtbx/bundle/copy_all.py cctbx This will create two directories - cctbx_build and cctbx_sources that include only the required files. Cheers, Oleg. ________________________________ From: [email protected] on behalf of Nicholas Devenish Sent: 09 July 2021 12:44 To: cctbx mailing list Subject: Re: [cctbxbb] Is there a schedule for the python3 migration or can cctbx.python already be built to be/use a python3 interpreter? Hi Jan, Not an expert on cctbx bootstrap - dials uses it's own. But I think you need to explicitly pass "--python 37", and possibly "--use_conda" to the cctbx bootstrap.py; I don't know if the non-conda bootstrap was ever updated to use python 3. I can't imagine the python 3.7 non-conda build is very widely used at this point. To slightly elaborate on what Graeme said about DIALS support - for DIALS/dxtbx we do support 3.6+, but will be dropping 3.6 for 3.7+ after the next release (mid-august), as we are now following the numpy deprecation schedule: https://numpy.org/neps/nep-0029-deprecation_policy.html#support-table Nick On Fri, Jul 9, 2021 at 12:30 PM Jan M. Simons wrote:

Am 09.07.21 um 13:16 schrieb Winter, Graeme (DLSLtd,RAL,LSCI):

...

With DIALS we use python3 exclusively - have done so for a long while - everything we use from cctbx seems largely OK with this

Python2 is dead to me, and many other dials devs

Oh, that's great to hear, I really hope to have everything Python3.6+ soon as well.

Still bootstrap.py seems to default to building a python 2.7.18 interpreter? Is this just for legacy backwards compatibility?

All the best Jan

...

All the best Graeme

...

On 9 Jul 2021, at 11:49, Jan M. Simons wrote:

Cheers,

I'm getting more and more concerned with the state of cctbx and the impending removal of python2 from quite a lot of major linux distributions in the near future.

So far I have not managed to find documentation on how to build the cctbx to then use a python3 interpreter for the cctbx.python script, but I would really like to have this, as I'd have to check/port my code that's using cctbx modules (mostly smtbx and xrax) to python3 as well, so that it does not get obsoleted.

So is it possible to build cctbx as a python3 module by now and if so how would one be able to do this?

Thanks for your support Jan _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

On Fri, Jul 9, 2021 at 12:40 PM Winter, Graeme (DLSLtd,RAL,LSCI) wrote:

Dials has a custom bootstrap which gives you a recent 3 build. I forget if it’s 3.9 or some other. But it’s certainly up to date and supported.

We base our build environment on conda rather than building everything locally which really speeds things up

All the best Graeme

...

On 9 Jul 2021, at 12:30, Jan M. Simons wrote:

Am 09.07.21 um 13:16 schrieb Winter, Graeme (DLSLtd,RAL,LSCI):

...

With DIALS we use python3 exclusively - have done so for a long while - everything we use from cctbx seems largely OK with this

Python2 is dead to me, and many other dials devs

Oh, that's great to hear, I really hope to have everything Python3.6+ soon as well.

Still bootstrap.py seems to default to building a python 2.7.18 interpreter? Is this just for legacy backwards compatibility?

All the best Jan

...

All the best Graeme

...

...

On 9 Jul 2021, at 11:49, Jan M. Simons wrote:

Cheers,

I'm getting more and more concerned with the state of cctbx and the impending removal of python2 from quite a lot of major linux distributions in the near future.

So far I have not managed to find documentation on how to build the cctbx to then use a python3 interpreter for the cctbx.python script, but I would really like to have this, as I'd have to check/port my code that's using cctbx modules (mostly smtbx and xrax) to python3 as well, so that it does not get obsoleted.

So is it possible to build cctbx as a python3 module by now and if so how would one be able to do this?

Thanks for your support Jan _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Hi Jan, Do you need to build cctbx from scratch? The cctbx has been available for Python 3 for a while now, but is using conda packages ( https://docs.conda.io/en/latest/) for managing dependencies. Building the dependencies from scratch for multiple versions of Python 3 for multiple platforms was getting too complicated. You can install cctbx as a conda package for Python 3.6 through 3.9 with support for macOS (Intel and Apple Silicon), linux, and Windows. Instructions can be found here ( https://github.com/cctbx/cctbx_project#installation), but essentially it is just conda install -c conda-forge cctbx-base to get it into an existing environment. The smtbx module will be available, but not fast_linalg yet. Note that with a conda environment, you do not need to run any of the "setpaths" scripts to set it up. Just activate the environment and cctbx will be available in python. There are also nightly builds of the conda packages on a separate channel ( https://github.com/cctbx/cctbx_project#nightly-builds). The command above becomes conda install -c cctbx-nightly -c conda-forge cctbx-base You can also build cctbx with Python 3 using conda packages as dependencies (https://github.com/cctbx/cctbx_project#building-a-development-version). The bootstrap.py command becomes python bootstrap.py --use-conda --python 38 This will install miniconda if conda is not already available on your system. You can run python bootstrap.py --help to see more options. There are monthly releases and I will make announcements on this bulletin board from now on to help spread the word. -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Fax: (510) 486-5909 Web: https://phenix-online.org On Fri, Jul 9, 2021 at 4:57 AM Jan M. Simons wrote:

Hi Oleg,

Am 09.07.21 um 13:16 schrieb Oleg Dolomanov:

...

Hi Jan,

which way have you been using to build cctbx?

I've been following https://cctbx.github.io/installation.html

(as https://cci.lbl.gov/cctbx_docs/installation.html seems to be the older/outdated documentation which should perhaps redirect to the newer one instead.)

Is this not the right documentation on how to build it on a bare bones linux distribution? (I'm using sabayon, a gentoo derivative)

So right now I've re-tried to execute an up-to-date bootstrap.py with my system supplied python3.7 and on first attempt it still built python2.7 and then failed with

Installing h5py 2.10.0... Read and write HDF5 files from Python log file is /home/jan/Arbeit/cctbx-dev/base_tmp/h5py_install_log Traceback (most recent call last): File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 1682, in <module> installer(args=sys.argv, log=sys.stdout) File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 256, in __init__ self.build_dependencies(packages=packages) File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 782, in build_dependencies getattr(self, 'build_%s'%i)() File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 1220, in build_h5py confirm_import_module='h5py', extra_options=["--no-binary=h5py"]) File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 694, in build_python_module_pip log=log) File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 359, in call return call(args, log=log, verbose=self.verbose, **kwargs) File

"/home/jan/Arbeit/cctbx-dev/modules/cctbx_project/libtbx/auto_build/installer_utils.py", line 55, in call raise RuntimeError("Call to '%s' failed with exit code %d" % (args, rc)) RuntimeError: Call to '/home/jan/Arbeit/cctbx-dev/base/bin/python -m pip install "h5py==2.10.0" --no-binary=h5py' failed with exit code 1 Process failed with return code 1

Then calling $ python3 ./bootstrap.py again it is still compiling, but from the paths I suppose that it's continuing to only build the python2.7 version (like before).

So, I suppose I'm missing some bit of documentation on how to get it working with python3.

Thanks for your support Jan _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Hi Jan, Good to know about the noexec /tmp error. I'll have to add a $TMPDIR environment variable since it looks like mounting /tmp as noexec is becoming more common. Another note is that the dependencies are updated about every 6 months. The versions and builds of the conda packages used for dependencies are explicitly tracked with their URLs and hashes to ensure reproducibility of the build environment. You can delete the "conda_base" directory and the bootstrap.py command (run in a shell before sourcing any of the "setpaths" files) will recreate it. Let us know if you have any other issues. Thanks! -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Fax: (510) 486-5909 Web: https://phenix-online.org On Fri, Jul 9, 2021 at 3:00 PM Jan M. Simons wrote:

Am 09.07.21 um 15:11 schrieb Billy Poon:

...

Hi Jan,

Do you need to build cctbx from scratch? The cctbx has been available for Python 3 for a while now, but is using conda packages (https://docs.conda.io/en/latest/ https://docs.conda.io/en/latest/) for managing dependencies. Building the dependencies from scratch for multiple versions of Python 3 for multiple platforms was getting too complicated.

You can install cctbx as a conda package for Python 3.6 through 3.9 with support for macOS (Intel and Apple Silicon), linux, and Windows. Instructions can be found here (https://github.com/cctbx/cctbx_project#installation https://github.com/cctbx/cctbx_project#installation), but essentially it is just

conda install -c conda-forge cctbx-base

to get it into an existing environment. The smtbx module will be available, but not fast_linalg yet.

Note that with a conda environment, you do not need to run any of the "setpaths" scripts to set it up. Just activate the environment and cctbx will be available in python.

There are also nightly builds of the conda packages on a separate channel (https://github.com/cctbx/cctbx_project#nightly-builds https://github.com/cctbx/cctbx_project#nightly-builds). The command above becomes

conda install -c cctbx-nightly -c conda-forge cctbx-base

You can also build cctbx with Python 3 using conda packages as dependencies (https://github.com/cctbx/cctbx_project#building-a-development-version https://github.com/cctbx/cctbx_project#building-a-development-version). The bootstrap.py command becomes

python bootstrap.py --use-conda --python 38

This will install miniconda if conda is not already available on your system.

As I did not have conda installed I went for this option and at first I encountered this:

===== Running in .: base Location of conda installation not provided Proceeding with a fresh installation Downloading https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh Downloaded file to /home/jan/Arbeit/cctbx-dev/Miniconda3-latest-Linux-x86_64.sh Installing miniconda to "/home/jan/Arbeit/cctbx-dev/mc3" /home/jan/Arbeit/cctbx-dev/mc3/conda.exe: error while loading shared libraries: libz.so.1: failed to map segment from shared object /home/jan/Arbeit/cctbx-dev/mc3/conda.exe: error while loading shared libraries: libz.so.1: failed to map segment from shared object Traceback (most recent call last): File "modules/cctbx_project/libtbx/auto_build/install_conda.py", line 1093, in <module> sys.exit(run()) File "modules/cctbx_project/libtbx/auto_build/install_conda.py", line 1069, in run verbose=namespace.verbose) File "modules/cctbx_project/libtbx/auto_build/install_conda.py", line 326, in __init__ self.conda_base = self.install_miniconda(prefix=self.root_dir) File "modules/cctbx_project/libtbx/auto_build/install_conda.py", line 543, in install_miniconda output = check_output(command_list, env=self.env) File

"/home/jan/Arbeit/cctbx-dev/modules/cctbx_project/libtbx/auto_build/installer_utils.py", line 68, in check_output raise RuntimeError("Call to '%s' failed with exit code %d" % (popenargs, retcode)) RuntimeError: Call to '(['/bin/sh', '/home/jan/Arbeit/cctbx-dev/Miniconda3-latest-Linux-x86_64.sh', '-b -u -p "/home/jan/Arbeit/cctbx-dev/mc3"'],)' failed with exit code 1 Process failed with return code 1

But after a little research [1] I found the cause to be a noexec /tmp.

Remounting /tmp exec made the bootstrap succeed and now I'm happy with

$ /home/jan/Arbeit/cctbx-dev/build/bin/cctbx.python Python 3.8.6 | packaged by conda-forge | (default, Oct 7 2020, 19:08:05) [GCC 7.5.0] on linux

So on to finally moving my codebase to Python3.

Thank you all for your support and have a nice weekend

Jan

[1]

https://stackoverflow.com/questions/60106630/conda-exe-error-while-loading-s... _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb