- cctbxbb - phenix-online.org

testing newer versions of boost
by Robert Oeffner 12 Feb '21

12 Feb '21

Hi, I'm testing newer versions of boost than 1.70 currently distributed with CCTBX. I can't remember if I have to do something special other than unpacking the zip file, say boost_1_74_0.zip, that I download from boost.org and put it into the modules directory for my cctbx build. But I get several errors during build like: C:\Users\Oeffner\OeffnerStuff\Work\PhenixVoyager\modules\boost\libs\filesystem\src\codecvt_error_category.cpp(71): error C2491: 'boost::filesystem::codecvt_error_category': definition of dllimport function not allowed I'm have tried with the VS2015, VS2017 and the VS2019 compiler and they all complain. So I'm wondering if there are some other magic invocations that have to be done before building with a new version of boost. Thanks in advance for any suggestions, Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus The Keith Peters Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234 -- This email has been checked for viruses by AVG. https://www.avg.com

2 2

Find nearest orthonormal matrix
by Robert Oeffner 25 Jan '21

25 Jan '21

Does the CCTBX has a routine for that? I didn't find anything in the scitbx module. But I came across a procedure online here: http://people.csail.mit.edu/bkph/articles/Nearest_Orthonormal_Matrix.pdf If not already in the CCTBX I plan to include a pythonic version of it within the next few days. Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus The Keith Peters Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234 -- This email has been checked for viruses by AVG. https://www.avg.com

3 2

(not really cctbx: PHENIX release bundle content: .o files)
by Winter, Graeme (DLSLtd,RAL,LSCI) 25 Jan '21

25 Jan '21

Hi All, Could not think of a better place to ask this question Installed a PHENIX update this morning and noticed it is quite a substantial download - hard to avoid, in this modern day. While unpacking I noticed it contained a lot of object files - totting it up seems to be ~ 450 MB of .o files which are probably not useful for the average end user. Just wondered, are these worth stripping out before making the release bundles? I note the same is true for DIALS so I am not claiming some moral high ground here & also https://github.com/dials/dials/discussions/1563 … looks like my colleagues decided to pick up on this & happened to agree Just thought worth asking Cheers Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

Space-saving tips
by Tristan Croll 25 Jan '21

25 Jan '21

Hi all, Following on from Graeme's email about .o files, another way to quite dramatically reduce the size of the distribution (with little impact on code complexity or performance) would be to keep all the various text-based data files in .zip format. With modern Python fetching the data from these is little different from working with uncompressed files anyway (1-2 extra lines of code, generally negligible runtime cost). A couple of snippets as an example from ISOLDE (where I keep all MD ligand definition files in a .zip): To get a list of the file contents: def _ligand_db_from_zip(self, ligand_zip): from zipfile import ZipFile import os namelist = [] with ZipFile(ligand_zip) as zf: for fname in zf.namelist(): name, ext = os.path.splitext(fname) if ext.lower() == '.xml': namelist.append(name) return namelist To read a given file from the zip as needed: if ligand_db is not None: zip, namelist = ligand_db if name in namelist: from zipfile import ZipFile logger.info('Loading residue template for {} from internal database'.format(name)) with ZipFile(zip) as zf: with zf.open(name+'.xml') as xf: forcefield.loadFile(xf) Even just doing this for the CaBLAM contour data cuts about 160 MB off the size of the uncompressed distribution. Best regards, Tristan

1 0

Cleaning clutter / pre-commit hooks
by Winter, Graeme (DLSLtd,RAL,LSCI) 25 Jan '21

25 Jan '21

Greetings cctbxbb Was reminded a moment ago about clutter and pre-commit hooks and wondered if there was merit in revisiting this question for cctbx? Not suggesting that pre-commit hooks should be mandatory (though you can git commit -n even if they are) but thinking that it would be a useful way of avoiding doubling up on commits and “failed builds” due to clutter Welcome viewpoints If consensus is not against the use of pre-commit hooks we can try and adapt some from e.g. dials for use here but I would not bother with this if people prefer to manually remove clutter Cheers Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

alternative to scipy.linalg.eig
by Robert Oeffner 10 Dec '20

10 Dec '20

scipy.linalg.eig is useful for diagonalising a 3x3 matrix to find eigenvalues and eigenvectors. Does anybody know if there is code in CCTBX which can also do this? Rob -- This email has been checked for viruses by AVG. https://www.avg.com

3 2

Power disruption (12/1 11 pm to 12/2 2 pm)
by Billy Poon 02 Dec '20

02 Dec '20

Hi everyone, Sorry for the short notice, but due to some electrical work in our building, several of the Berkeley servers will be shut down later tonight at 11 pm Pacific and will return tomorrow afternoon at 2 pm Pacific. This will affect: - The Phenix web site (http://phenix-online.org/) - The CCI web site (http://cci.lbl.gov/) - The Structural Biology Technology Portal (http://technology.sbkb.org/) Bug reports and emails requesting help should be queued and we will receive them once the servers are back online. We apologize for any inconvenience. -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

1 0

Re: [cctbxbb] [git/cctbx_project] master: de-clutter (e9be98075a)
by Robert Oeffner 07 Oct '20

07 Oct '20

Hi Graeme, It's just a simple call to the libtbx.clean_clutter script. But with the path set explicitly so it only works for this clone of the repo. Rob [hook "precommit"] cmdline = C:\\Users\\Oeffner\\Work\\Phenix\\phenix-installer-dev-3981-intel-windows-x86_64\\build\\bin\\libtbx.clean_clutter.bat wait = true show = true On 07-10-2020 14:33, Winter, Graeme (DLSLtd,RAL,LSCI) wrote: > Hi Rob, > > No worries - the deluge of commits last night with the same fricken > message have made my hyper sensitive and I have had a pet hate of > clean clutter for a long while, and your commit happened to come along > while I had it in my head > > I really think universal pre-commit hooks would be _a good thing_ > > if you wanted to circulate the hook in question you may also help > others… though I seem to recall there are some who dislike things like > pre-commit hooks as somehow impinging on their freedom or similar… > > Cheers Graeme > >> On 7 Oct 2020, at 14:23, Robert Oeffner <rdo20(a)cam.ac.uk >> <mailto:[email protected]>> wrote: >> >> Sorry about the noise in the commit history. I have now put a >> pre-commit hook in my git settings that runs clean_clutter on my >> clone of the cctbx repo. >> >> Rob >> >> >> On 07-10-2020 13:36, Winter, Graeme (DLSLtd,RAL,LSCI) wrote: >>> Hi Folks >>> >>> Would anyone be offended if we had a pre-commit hook which ran the >>> clutter / unused imports tests to stop the history fulling up with >>> clean clutter commits? >>> >>> If clutter is important, probably should not let it in in the first >>> place? >>> >>> I’m not suggesting anything extreme like black, like we use for >>> dxtbx and dials, just running the tools that developers are supposed >>> to run before commits >>> >>> Cheers Graeme >>> >>>> Begin forwarded message: >>>> >>>> *From: *CCTBX commit <diamondlightsource.jenkins(a)GMAIL.COM >>>> <mailto:[email protected]>> >>>> *Subject: **[git/cctbx_project] master: de-clutter (e9be98075a)* >>>> *Date: *7 October 2020 at 12:56:50 BST >>>> *To: *CCTBX-COMMIT(a)JISCMAIL.AC.UK <mailto:[email protected]> >>>> *Reply-To: *cctbxbb(a)PHENIX-ONLINE.ORG >>>> <mailto:[email protected]> >>>> >>>> Repository : https://github.com/cctbx/cctbx_project/ >>>> <https://github.com/cctbx/cctbx_project/> >>>> On branch : master >>>> Link : >>>> https://github.com/cctbx/cctbx_project/commit/e9be98075a31f4584f1eca41f9eba… >>>> >>>> ------------------------------------------------------------------------ >>>> >>>> >>>> commit e9be98075a31f4584f1eca41f9eba5e408d0cb8f >>>> Author: Robert Oeffner <rdo20(a)cam.ac.uk <mailto:[email protected]>> >>>> Date: Wed Oct 7 12:56:50 2020 +0100 >>>> >>>> de-clutter >>>> >>>> >>>> ------------------------------------------------------------------------ >>>> >>>> >>>> e9be98075a31f4584f1eca41f9eba5e408d0cb8f >>>> cctbx/miller/__init__.py | 0 >>>> 1 file changed, 0 insertions(+), 0 deletions(-) >>>> >>>> >>>> ------------------------------------------------------------------------ >>>> >>>> To unsubscribe from the CCTBX-COMMIT list, click the following link: >>>> https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCTBX-COMMIT&A=1 >>>> >>> >>> >>> -- >>> >>> This e-mail and any attachments may contain confidential, copyright >>> and or privileged material, and are for the use of the intended >>> addressee only. If you are not the intended addressee or an >>> authorised recipient of the addressee please notify us of receipt by >>> returning the e-mail and do not use, copy, retain, distribute or >>> disclose the information in or attached to the e-mail. >>> Any opinions expressed within this e-mail are those of the >>> individual and not necessarily of Diamond Light Source Ltd. >>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any >>> attachments are free from viruses and we cannot accept liability for >>> any damage which you may sustain as a result of software viruses >>> which may be transmitted in or with the message. >>> Diamond Light Source Limited (company no. 4375679). Registered in >>> England and Wales with its registered office at Diamond House, >>> Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 >>> 0DE, United Kingdom >>> >>> >>> _______________________________________________ >>> cctbxbb mailing list >>> cctbxbb(a)phenix-online.org >>> http://phenix-online.org/mailman/listinfo/cctbxbb >> -- >> Robert Oeffner, Ph.D. >> Research Associate, The Read Group >> Department of Haematology, >> Cambridge Institute for Medical Research >> University of Cambridge >> Cambridge Biomedical Campus >> The Keith Peters Building >> Hills Road >> Cambridge CB2 0XY >> www.cimr.cam.ac.uk/investigators/read/index.html >> tel: +44(0)1223 763234 >> >> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_cam…> >> Virus-free. www.avg.com >> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_cam…> >> >> > > -- > > This e-mail and any attachments may contain confidential, copyright > and or privileged material, and are for the use of the intended > addressee only. If you are not the intended addressee or an authorised > recipient of the addressee please notify us of receipt by returning > the e-mail and do not use, copy, retain, distribute or disclose the > information in or attached to the e-mail. > Any opinions expressed within this e-mail are those of the individual > and not necessarily of Diamond Light Source Ltd. > Diamond Light Source Ltd. cannot guarantee that this e-mail or any > attachments are free from viruses and we cannot accept liability for > any damage which you may sustain as a result of software viruses which > may be transmitted in or with the message. > Diamond Light Source Limited (company no. 4375679). Registered in > England and Wales with its registered office at Diamond House, Harwell > Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United > Kingdom > -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus The Keith Peters Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234 -- This email has been checked for viruses by AVG. https://www.avg.com

1 0

Fwd: [git/cctbx_project] master: de-clutter (e9be98075a)
by Winter, Graeme (DLSLtd,RAL,LSCI) 07 Oct '20

07 Oct '20

Hi Folks Would anyone be offended if we had a pre-commit hook which ran the clutter / unused imports tests to stop the history fulling up with clean clutter commits? If clutter is important, probably should not let it in in the first place? I’m not suggesting anything extreme like black, like we use for dxtbx and dials, just running the tools that developers are supposed to run before commits Cheers Graeme Begin forwarded message: From: CCTBX commit <diamondlightsource.jenkins(a)GMAIL.COM<mailto:[email protected]>> Subject: [git/cctbx_project] master: de-clutter (e9be98075a) Date: 7 October 2020 at 12:56:50 BST To: CCTBX-COMMIT(a)JISCMAIL.AC.UK<mailto:[email protected]> Reply-To: cctbxbb(a)PHENIX-ONLINE.ORG<mailto:[email protected]> Repository : https://github.com/cctbx/cctbx_project/ On branch : master Link : https://github.com/cctbx/cctbx_project/commit/e9be98075a31f4584f1eca41f9eba… ________________________________ commit e9be98075a31f4584f1eca41f9eba5e408d0cb8f Author: Robert Oeffner <rdo20(a)cam.ac.uk<mailto:[email protected]>> Date: Wed Oct 7 12:56:50 2020 +0100 de-clutter ________________________________ e9be98075a31f4584f1eca41f9eba5e408d0cb8f cctbx/miller/__init__.py | 0 1 file changed, 0 insertions(+), 0 deletions(-) ________________________________ To unsubscribe from the CCTBX-COMMIT list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCTBX-COMMIT&A=1 -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

Clutter etc.
by Winter, Graeme (DLSLtd,RAL,LSCI) 03 Oct '20

03 Oct '20

Hi Folks Please could we make less of a big deal about clutter? I have been hacking away for weeks at a PR & errors such as 2020-10-03T09:39:50.9328925Z Generating script. 2020-10-03T09:39:50.9348357Z ========================== Starting Command Output =========================== 2020-10-03T09:39:50.9378363Z [command]/bin/bash --noprofile --norc /home/vsts/work/_temp/2787f38a-d1cc-4c10-b0bd-c9b041b41631.sh 2020-10-03T09:40:32.0565443Z 2020-10-03T09:40:32.0566065Z mmtbx/model/model.py: unused imports=1 2020-10-03T09:40:32.0566273Z 2020-10-03T09:40:32.0566573Z *** To clean up please use: libtbx.find_unused_imports_crude *** 2020-10-03T09:40:32.0566824Z 2020-10-03T09:40:32.0711947Z 2020-10-03T09:40:32.0785277Z ##[error]Bash exited with code '1'. 2020-10-03T09:40:32.0802183Z ##[section]Finishing: Check for clutter 2020-10-03T09:40:32.0831426Z ##[section]Starting: Checkout cctbx/cctbx_project@refs/pull/533/merge to ./modules/cctbx_project 2020-10-03T09:40:32.0841670Z ============================================================================== 2020-10-03T09:40:32.0842063Z Task : Get sources 2020-10-03T09:40:32.0842444Z Description : Get sources from a repository. Supports Git, TfsVC, and SVN repositories. 2020-10-03T09:40:32.0842850Z Version : 1.0.0 2020-10-03T09:40:32.0843106Z Author : Microsoft 2020-10-03T09:40:32.0843503Z Help : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199) 2020-10-03T09:40:32.0843947Z ============================================================================== 2020-10-03T09:40:32.4896538Z Cleaning any cached credential from repository: cctbx/cctbx_project (GitHub) 2020-10-03T09:40:32.4948059Z ##[section]Finishing: Checkout cctbx/cctbx_project@refs/pull/533/merge to ./modules/cctbx_project 2020-10-03T09:40:32.5010291Z ##[section]Starting: Finalize Job 2020-10-03T09:40:32.5043249Z Cleaning up task key 2020-10-03T09:40:32.5044549Z Start cleaning up orphan processes. 2020-10-03T09:40:32.5217555Z ##[section]Finishing: Finalize Job 2020-10-03T09:40:32.5261526Z ##[section]Finishing: syntax python2.7 are making it hard to actually make the tests pass (this change is something I have not touched) Is it really worth breaking builds in PR’s because of some white space / unused imports in unrelated files? Cheers Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 2