Hi Luc, Also, we are exploring the use of conda for installing the dependencies instead of compiling from scratch. There is some preliminary work in https://github.com/cctbx/conda_build for dependencies and the "conda_compiler" branch in cctbx_project for building. The "cctbx_dependencies" conda package already installs mkl and it would be easy to install openblas. This should be ready in a few months once some other details (e.g. building releases, filling in missing dependencies) are worked out. -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org On Wed, Jun 20, 2018 at 9:48 AM Nigel Moriarty wrote:

Luc

I like the idea. I may be able to convince a new collaborator to drop FORTRAN in favour of the cctbx.

Cheers

Nigel

--- Nigel W. Moriarty Building 33R0349, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709 Email : [email protected] Fax : 510-486-5909 Web : CCI.LBL.gov

On Wed, Jun 20, 2018 at 5:13 AM, Luc Bourhis wrote:

...

Hi,

years ago, I had wrote on this discussion list about a project of mine to dramatically accelerate scitbx.lstbx by using optimised BLAS libraries. That work had been dormant on a branch but now we are planning to move forward with the help of Pascal Parlois who will do the actual coding in the coming months. In order for this new code to be exercised by nightly tests, we, the smtbx people, need to write code so that that optimised BLAS library is installed during bootstrap. Right now, as a stopgap, I used conda to install either OpenBLAS and MKL but that won’t do with the cctbx philosophy.

There are basically two choices: either OpenBLAS or MKL. When I started this project, I chose OpenBLAS because MKL was not freely redistributable, and I had got the green light to install it as needed. However now MKL has become completely free, thus becoming a possible choice. MKL is more performant on Intel but OpenBLAS is better on AMD (OpenBLAS actually runs on pretty much any processor out there but we don’t care in the context of cctbx): c.f. Julia people conclusions https://discourse.julialang.org/t/openblas-is-faster-than-intel-mkl-on-amd-h... Now I realise that most of you may not care one way or another!?! But we won’t move forward before we got the green light…

Note that I am taking about installing a BLAS usable from C++ here, not getting a BLAS-accelerated numpy. The latter could be an alternative though. But that would require to modify the bootstrap code to compile a MKL- or OpenBLAS-enabled numpy anyway.

Best wishes,

Luc J Bourhis

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Hi, What is the state of this preliminary work? Is it somehow useable? I am wondering if I should look into including openblas in bootstrap or make use of the conda branch. How much work and how easy would it be to include openblas in bootsrap? Basically I just need to download the archive, run make and make install into base. Regards, Pascal On 20/06/18 18:54, Billy Poon wrote:

Hi Luc,

Also, we are exploring the use of conda for installing the dependencies instead of compiling from scratch. There is some preliminary work in https://github.com/cctbx/conda_build for dependencies and the "conda_compiler" branch in cctbx_project for building. The "cctbx_dependencies" conda package already installs mkl and it would be easy to install openblas.

This should be ready in a few months once some other details (e.g. building releases, filling in missing dependencies) are worked out.

-- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

On Wed, Jun 20, 2018 at 9:48 AM Nigel Moriarty mailto:[email protected]> wrote:

Luc

I like the idea. I may be able to convince a new collaborator to drop FORTRAN in favour of the cctbx.

Cheers

Nigel

--- Nigel W. Moriarty Building 33R0349, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709     Email : [email protected] Fax   : 510-486-5909       Web  : CCI.LBL.gov http://CCI.LBL.gov

On Wed, Jun 20, 2018 at 5:13 AM, Luc Bourhis mailto:[email protected]> wrote:

Hi,

years ago, I had wrote on this discussion list about a project of mine to dramatically accelerate scitbx.lstbx by using optimised BLAS libraries. That work had been dormant on a branch but now we are planning to move forward with the help of Pascal Parlois who will do the actual coding in the coming months. In order for this new code to be exercised by nightly tests, we, the smtbx people, need to write code so that that optimised BLAS library is installed during bootstrap. Right now, as a stopgap, I used conda to install either OpenBLAS and MKL but that won’t do with the cctbx philosophy.

There are basically two choices: either OpenBLAS or MKL. When I started this project, I chose OpenBLAS because MKL was not freely redistributable, and I had got the green light to install it as needed. However now MKL has become completely free, thus becoming a possible choice. MKL is more performant on Intel but OpenBLAS is better on AMD (OpenBLAS actually runs on pretty much any processor out there but we don’t care in the context of cctbx): c.f. Julia people conclusions https://discourse.julialang.org/t/openblas-is-faster-than-intel-mkl-on-amd-h...  Now I realise that most of you may not care one way or another!?! But we won’t move forward before we got the green light…

Note that I am taking about installing a BLAS usable from C++ here, not getting a BLAS-accelerated numpy. The latter could be an alternative though. But that would require to modify the bootstrap code to compile a MKL- or OpenBLAS-enabled numpy anyway.

Best wishes,

Luc J Bourhis

_______________________________________________ cctbxbb mailing list [email protected] mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Hi On 22/06/18 09:29, Billy Poon wrote:

Hi Pascal,

It depends on what you mean by useable. I outlined some steps for building and running DIALS at https://github.com/cctbx/cctbx_project/issues/85, but those steps will work for Phenix and CCTBX as well. Most tests will pass, but there are still missing dependencies and there is no Windows support. So most things should run, but I have not tested on multiple operating systems yet. Also, the workflow for compiling development builds and release installers still needs to be worked out. Thanks. I will have a look

Does openblas need to be installed by default? If so, testing the builds on different operating systems (especially CentOS 5 and 32-bit oses) are the time-consuming steps. It depends what cctbx people want. At the moment there is no dependency on blas/lapack and it can stay that way and having it optional as well as keeping the actual implementation as is. Which means duplicating some algorithms with blas/lapack calls. You still need to test the alternative at some point though.

Openblas can be made compulsory, some functions updated and any new development can rely on it. openblas, especially when dynamic arch is used is taking a huge amount of time to compile. Plus I can predict a long debate on why openblas or which library to use instead. A third option would be to keep third party blas/lapack libraries optional but include the version from netlib has a fall back in cctbx. No need to duplicate implementation. If you just include the functions used in cctbx from netlib it is also lightweight. Tests should be fine against netlib only. Third party libraries are checked on their own. Pascal

Howdy y’all, If we could avoid adding more FORTRAN source code dependencies to this I (as a fruity computer user) would be delighted :-) Cheerio Graeme On 22 Jun 2018, at 13:01, Luc Bourhis mailto:[email protected]> wrote: Hi, On 22 Jun 2018, at 13:27, Pascal mailto:[email protected]> wrote: A third option would be to keep third party blas/lapack libraries optional but include the version from netlib has a fall back in cctbx. No need to duplicate implementation. If you just include the functions used in cctbx from netlib it is also lightweight. Tests should be fine against netlib only. Third party libraries are checked on their own. I like the idea. It is very important the new smtbx code relying on BLAS 3 is part of nightly tests but Pascal is right here. The only problem is that netlib reference BLAS is written in FORTRAN. So is LAPACK. Both have C interfaces but the actual implementation is in FORTRAN. We could of course run fable on it. In case you do not know Pascal, fable is a tool written by Ralf to convert Fortran to C++. Best wishes, Luc _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

Is not cblas written in C or it is just an interface to the F77 version? Otherwise the gsl (gnu scientific library) could be used? Pascal On 22/06/18 13:23, Luc Bourhis wrote:

Once fable would have been run on it, the FORTRAN would be gone: just commit the C++ and forget the origin of it. But this is terribly unrealistic: this is a vast amount of code to translate and trusting blindly fable would be silly whereas testing it all would be far too much work.

Thus I think the only route for getting the new BLAS 3 smtbx part of nightly tests would be to make the installation of the optimised BLAS not optional.

...

On 22 Jun 2018, at 14:07, [email protected] wrote:

Howdy y’all,

If we could avoid adding more FORTRAN source code dependencies to this I (as a fruity computer user) would be delighted :-)

Cheerio Graeme

On 22 Jun 2018, at 13:01, Luc Bourhis mailto:[email protected]> wrote:

Hi,

On 22 Jun 2018, at 13:27, Pascal mailto:[email protected]> wrote:

A third option would be to keep third party blas/lapack libraries optional but include the version from netlib has a fall back in cctbx. No need to duplicate implementation. If you just include the functions used in cctbx from netlib it is also lightweight. Tests should be fine against netlib only. Third party libraries are checked on their own.

I like the idea. It is very important the new smtbx code relying on BLAS 3 is part of nightly tests but Pascal is right here. The only problem is that netlib reference BLAS is written in FORTRAN. So is LAPACK. Both have C interfaces but the actual implementation is in FORTRAN. We could of course run fable on it. In case you do not know Pascal, fable is a tool written by Ralf to convert Fortran to C++.

Best wishes,

Luc

_______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

" Licensing GSL is distributed under the terms of the GNU General Public Licensehttps://www.gnu.org/licenses/gpl.html (GPL). The reasons why the GNU Project uses the GPL are described in the following articles: * Copyleft: Pragmatic Idealismhttps://www.gnu.org/philosophy/pragmatic.html by Richard Stallman * Why you should not use the Lesser GPL for your next libraryhttps://www.gnu.org/licenses/why-not-lgpl.html by Richard Stallman Additional information for researchers is available in the following article: * Releasing Free Software if you work at a Universityhttps://www.gnu.org/philosophy/university.html by Richard Stallman Some answers to common questions about the license: If I write an application which uses GSL, am I forced to distribute that application? No. The license gives you the option to distribute your application if you want to. You do not have to exercise this option in the license. If I wanted to distribute an application which uses GSL, what license would I need to use? The GNU General Public License (GPL). The bottom line for commercial users: GSL can be used internally ("in-house") without restriction, but only redistributed in other software that is under the GNU GPL." tricky for our BSD-licensed DIALS / cctbx / etc. & blocker for Phenix (I guess?) On 22 Jun 2018, at 13:49, [email protected]mailto:[email protected] mailto:[email protected]> wrote: GSL is a licensing nightmare? Hardcore GNU license? On 22 Jun 2018, at 13:48, Pascal mailto:[email protected]> wrote: Is not cblas written in C or it is just an interface to the F77 version? Otherwise the gsl (gnu scientific library) could be used? Pascal On 22/06/18 13:23, Luc Bourhis wrote: Once fable would have been run on it, the FORTRAN would be gone: just commit the C++ and forget the origin of it. But this is terribly unrealistic: this is a vast amount of code to translate and trusting blindly fable would be silly whereas testing it all would be far too much work. Thus I think the only route for getting the new BLAS 3 smtbx part of nightly tests would be to make the installation of the optimised BLAS not optional. On 22 Jun 2018, at 14:07, [email protected]mailto:[email protected] mailto:[email protected]> wrote: Howdy y’all, If we could avoid adding more FORTRAN source code dependencies to this I (as a fruity computer user) would be delighted :-) Cheerio Graeme On 22 Jun 2018, at 13:01, Luc Bourhis mailto:[email protected]mailto:[email protected]> wrote: Hi, On 22 Jun 2018, at 13:27, Pascal mailto:[email protected]mailto:[email protected]> wrote: A third option would be to keep third party blas/lapack libraries optional but include the version from netlib has a fall back in cctbx. No need to duplicate implementation. If you just include the functions used in cctbx from netlib it is also lightweight. Tests should be fine against netlib only. Third party libraries are checked on their own. I like the idea. It is very important the new smtbx code relying on BLAS 3 is part of nightly tests but Pascal is right here. The only problem is that netlib reference BLAS is written in FORTRAN. So is LAPACK. Both have C interfaces but the actual implementation is in FORTRAN. We could of course run fable on it. In case you do not know Pascal, fable is a tool written by Ralf to convert Fortran to C++. Best wishes, Luc _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

cblas is actually written entirely in C indeed. I had always somehow believed it was just an interface. Maybe it was at the beginning. Anyway here is the download link: http://www.netlib.org/blas/blast-forum/cblas.tgz http://www.netlib.org/blas/blast-forum/cblas.tgz Ok, good find. Now making netlib cblas a compulsory part of bootstrap and MKL or OpenBLAS an optional part: is that really saving work?

On 22 Jun 2018, at 14:48, Pascal wrote:

Is not cblas written in C or it is just an interface to the F77 version?

Otherwise the gsl (gnu scientific library) could be used?

Pascal

On 22/06/18 13:23, Luc Bourhis wrote:

...

Once fable would have been run on it, the FORTRAN would be gone: just commit the C++ and forget the origin of it. But this is terribly unrealistic: this is a vast amount of code to translate and trusting blindly fable would be silly whereas testing it all would be far too much work.

Thus I think the only route for getting the new BLAS 3 smtbx part of nightly tests would be to make the installation of the optimised BLAS not optional.

...

On 22 Jun 2018, at 14:07, [email protected] wrote:

Howdy y’all,

If we could avoid adding more FORTRAN source code dependencies to this I (as a fruity computer user) would be delighted :-)

Cheerio Graeme

On 22 Jun 2018, at 13:01, Luc Bourhis mailto:[email protected]> wrote:

Hi,

On 22 Jun 2018, at 13:27, Pascal mailto:[email protected]> wrote:

A third option would be to keep third party blas/lapack libraries optional but include the version from netlib has a fall back in cctbx. No need to duplicate implementation. If you just include the functions used in cctbx from netlib it is also lightweight. Tests should be fine against netlib only. Third party libraries are checked on their own.

I like the idea. It is very important the new smtbx code relying on BLAS 3 is part of nightly tests but Pascal is right here. The only problem is that netlib reference BLAS is written in FORTRAN. So is LAPACK. Both have C interfaces but the actual implementation is in FORTRAN. We could of course run fable on it. In case you do not know Pascal, fable is a tool written by Ralf to convert Fortran to C++.

Best wishes,

Luc

_______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

On 22/06/18 13:59, Luc Bourhis wrote:

...

Ok, good find. Now making netlib cblas a compulsory part of bootstrap and MKL or OpenBLAS an optional part: is that really saving work? Actually, no, it does not work. We also need LAPACK anyway. And that’s only in FORTRAN as far as I know.

Clapack is also in c.

Luc, Please make sure that any new dependencies are optional, and that existing users can continue to use the existing code. LSTBX is a fundamental component of CCTBX, and I wouldn't be keen on suddenly changing all the underlying implementations! Also, we cannot include anything GPL in CCTBX. If need be we can set up a high-level discussion of this also involving Paul Adams. Nick Nicholas K. Sauter, Ph. D. Senior Scientist, Molecular Biophysics & Integrated Bioimaging Division Lawrence Berkeley National Laboratory 1 Cyclotron Rd., Bldg. 33R0345 Berkeley, CA 94720 (510) 486-5713 On Fri, Jun 22, 2018 at 6:16 AM, Luc Bourhis wrote:

...

...

Actually, no, it does not work. We also need LAPACK anyway. And that’s only in FORTRAN as far as I know.

Clapack is also in c.

Ah, yes, Lapack passed through f2c. Forgot about that indeed. Mmmhmmm… Ok, so let’s sum up the alternatives.

1. compile together netlib cblas and clapack, and install in the right place in the cctbx tree

2. copy MKL headers, and shared libs (Linux), dynamic libraries (MacOS), or DLL’s (Windows) to the right place in the cctbx tree

3. compile OpenBLAS and install in the right place in the cctbx tree

I am not convinced option 1 is any simpler, because we need to do the job for two libraries and make sure they work together whereas for option 3 OpenBLAS makefiles take care of everything. Actually, I think option 2 is the simplest one because no compilation is involved.

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Folks, Optional use could be interesting for dials... I find for big rotation sets the refinement can be rate limiting... faster algorithms are of interest but I agree optional better. Also dials scaling... Also with the use of git and branches and stuff we can test this very carefully before merging back Best wishes Graeme ________________________________ From: [email protected] on behalf of Luc Bourhis Sent: 22 June 2018 20:28:19 To: cctbx mailing list Subject: Re: [cctbxbb] Install OpenBLAS or MKL in base Nicholas,

Please make sure that any new dependencies are optional, and that existing users can continue to use the existing code. LSTBX is a fundamental component of CCTBX, and I wouldn't be keen on suddenly changing all the underlying implementations!

Yes, I had noticed it is used by xfel and by a few tests. Only the class non_linear_ls is used though, and that I did not optimise in my current attempt, that can be found on the branch reworked_fast_lstbx. I only optimised the L.S. I needed for smtbx, i.e. where the scale between model and data is unknown and needs to be fitted. The class is non_linear_ls_with_separable_scale_factor. So, right now, there is nothing to worry about as far as the smtbx is not concerned! Unless I missed something… Would that be acceptable then, if we don’t touch the features of scitbx.lstbx used outside of smtbx? The thing is that keeping both BLAS 2 and 3 implementations significantly increased the complexity of it all. Moreover it makes some alternative solutions we are considering, like moving part of the code to Python, more difficult. So we would prefer to have the freedom to break code using non_linear_ls_with_separable_scale_factor.

Also, we cannot include anything GPL in CCTBX.

Yes, sure. The discussion went on a tangent, sorry. Best wishes, Luc _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

On 22/06/18 14:16, Luc Bourhis wrote:

...

...

Actually, no, it does not work. We also need LAPACK anyway. And that’s only in FORTRAN as far as I know. Clapack is also in c. Ah, yes, Lapack passed through f2c. Forgot about that indeed. Mmmhmmm… Ok, so let’s sum up the alternatives.

1. compile together netlib cblas and clapack, and install in the right place in the cctbx tree Hum, maybe not. it needs to be linked with -lf2c

2. copy MKL headers, and shared libs (Linux), dynamic libraries (MacOS), or DLL’s (Windows) to the right place in the cctbx tree Depending on the binary is there any restrictions on the compiler type and version you can use?

Pascal