- cctbxbb - phenix-online.org

Quick Boost python question
by Nicholas Sauter 21 Feb '17

21 Feb '17

<http://phenix-online.org/mailman/listinfo/cctbxbb>Quick question from James Holton and myself. We want to use a boost python *.add_property() to add a make_getter and make_setter to our wrapper, so we can expose a class variable to the Python layer. However, we want to execute a change of measurement units, i.e., the Python value will always be given in millimetres, and the C++ value always in metres. How can this be done within a simple add_property() paradigm? Nick

1 1

Berkeley Server Downtime (1/13 - 1/14)
by Billy Poon 15 Jan '17

15 Jan '17

Dear all, On Friday, January 13, our servers will be shut down starting at 5:00 pm Pacific for scheduled maintenance in our building. We expect that the servers will be available the next day (1/14) by 6:00 pm Pacific at the latest. This will affect, - The Phenix web site (http://phenix-online.org/) - The DIALS West web site (http://dials.lbl.gov/) - The CCI web site (http://cci.lbl.gov/) - The Structural Biology Technology Portal (http://technology.sbkb.org/) Bug reports and emails requesting help should be queued and we will receive them once the servers are back online. We apologize for any inconvenience. -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

1 3

Re: [cctbxbb] iotbx.merging_statistics test failing: data file removed from phenix_regression
by richard.gildea＠diamond.ac.uk 06 Jan '17

06 Jan '17

Hi Tom, Thanks for the quick response! To run this specific test: $ libtbx.python $(libtbx.find_in_repositories iotbx/regression/tst_merging_statistics.py) The test also uses the file phenix_regression/wizards/unmerged.mtz which I guess was also removed with the same commit? $ grep phenix_regression $(libtbx.find_in_repositories iotbx/regression/tst_merging_statistics.py) if (not libtbx.env.has_module("phenix_regression")) : print "phenix_regression not configured, skipping." relative_path="phenix_regression/wizards/p9_se_w2.sca", relative_path="phenix_regression/harvesting/unmerged.sca", relative_path="phenix_regression/harvesting/refine.mtz", relative_path="phenix_regression/reflection_files/i_anomalous.mtz", relative_path="phenix_regression/wizards/unmerged.mtz", relative_path="phenix_regression/reflection_files/AUTOMATIC_DEFAULT_scaled_unmerged_WAVE1.mtz", $ libtbx.find_in_repositories phenix_regression/wizards/unmerged.mtz Sorry: libtbx.find_in_repositories: cannot locate "phenix_regression/wizards/unmerged.mtz" The other files still seem to be present in phenix_regression. Cheers, Richard Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE ________________________________________ From: Terwilliger, Thomas Charles [terwilliger(a)lanl.gov] Sent: 06 January 2017 15:57 To: Gildea, Richard (DLSLtd,RAL,LSCI); cctbxbb(a)phenix-online.org Cc: Terwilliger, Thomas Charles Subject: Re: iotbx.merging_statistics test failing: data file removed from phenix_regression Hi Richard, Thanks for pointing that out! I'll fix it. Yes, I deleted that file when I moved all the small files in phenix_regression/wizards/ into sub-directories. I made the mistake of assuming that only wizards/ regression tests used files in the wizards/ directory (I should have run all the cctbx regression tests to make sure). All the best, Tom T ________________________________________ From: richard.gildea(a)diamond.ac.uk <richard.gildea(a)diamond.ac.uk> Sent: Friday, January 6, 2017 8:51 AM To: cctbxbb(a)phenix-online.org; Terwilliger, Thomas Charles Subject: iotbx.merging_statistics test failing: data file removed from phenix_regression Hi all, The test for iotbx.merging_statistics has recently started failing as a result of one of the files it uses for testing being deleted, phenix_regression/wizards/p9_se_w2.sca: https://github.com/cctbx/cctbx_project/blob/master/iotbx/regression/tst_mer… Would it be possible to restore this file to phenix_regression so that the test will pass? $ svn log -r r3879 -v ------------------------------------------------------------------------ r3879 | terwill | 2016-12-29 02:48:15 +0000 (Thu, 29 Dec 2016) | 1 line Changed paths: D /trunk/wizards/000README D /trunk/wizards/4c5q.pdb D /trunk/wizards/ATP.pdb D /trunk/wizards/MM-all.tgz D /trunk/wizards/NAD.pdb D /trunk/wizards/PARAMS_1.eff D /trunk/wizards/a2u-data.mtz D /trunk/wizards/a2u-sigmaa.mtz D /trunk/wizards/a2u.sca D /trunk/wizards/a2u_short.pdb D /trunk/wizards/anb.pdb D /trunk/wizards/autosol_unmerged_False.std D /trunk/wizards/autosol_unmerged_True.std D /trunk/wizards/autosol_unmerged_p9.std D /trunk/wizards/bad_model.pdb D /trunk/wizards/check.xyz D /trunk/wizards/check_multiple_input_files_042908.tgz D /trunk/wizards/check_multiple_input_files_062608.tgz D /trunk/wizards/check_multiple_input_files_070508.tgz D /trunk/wizards/check_multiple_input_files_2011-03-03.tgz D /trunk/wizards/check_multiple_input_files_2011-06-29.tgz D /trunk/wizards/check_multiple_input_files_2011-11-04.tgz D /trunk/wizards/clean.com D /trunk/wizards/cocn.cif D /trunk/wizards/coords.pdb D /trunk/wizards/coords1.ali D /trunk/wizards/coords1.mtz D /trunk/wizards/coords1.pdb D /trunk/wizards/coords_A001.pdb D /trunk/wizards/coords_rna.pdb D /trunk/wizards/coords_rna_gap.pdb D /trunk/wizards/data_for_commands_in_doc.tgz D /trunk/wizards/find_all_ligands.eff D /trunk/wizards/find_all_ligands.standard D /trunk/wizards/find_ncs_ncs.standard D /trunk/wizards/find_ncs_resolve.standard D /trunk/wizards/fobs.mtz D /trunk/wizards/fobs_std.mtz D /trunk/wizards/fobs_std_edit.mtz D /trunk/wizards/gene-5.pdb D /trunk/wizards/generate.com D /trunk/wizards/get_cc_mtz_mtz.standard D /trunk/wizards/get_cc_mtz_pdb.standard D /trunk/wizards/ha.pdb D /trunk/wizards/ha_lig.pdb D /trunk/wizards/ha_perfect.pdb D /trunk/wizards/ha_perfect_2.pdb D /trunk/wizards/hipip.pdb D /trunk/wizards/hipip_2.sca D /trunk/wizards/hipip_shortseq.dat D /trunk/wizards/ignore.dat D /trunk/wizards/inf.sca D /trunk/wizards/ki.sca D /trunk/wizards/ligand.pdb D /trunk/wizards/mad.eff D /trunk/wizards/mad.mtz D /trunk/wizards/map_coeffs.mtz D /trunk/wizards/map_coeffs_1.mtz D /trunk/wizards/map_coeffs_2.mtz D /trunk/wizards/map_to_object.standard D /trunk/wizards/maps_only.standard D /trunk/wizards/mir.eff D /trunk/wizards/mlt.mtz D /trunk/wizards/mlt_gap.pdb D /trunk/wizards/mlt_regression.tgz D /trunk/wizards/mol1.pdb D /trunk/wizards/mol2.pdb D /trunk/wizards/mr_rosetta.eff D /trunk/wizards/mult_regr.tgz D /trunk/wizards/multi.pdb D /trunk/wizards/multi.pdb.mtz D /trunk/wizards/my_autobuild.eff D /trunk/wizards/my_automr.eff D /trunk/wizards/my_autosol.eff D /trunk/wizards/my_ligand.pdb D /trunk/wizards/my_ligandfit.eff D /trunk/wizards/my_target.pdb D /trunk/wizards/native.sca D /trunk/wizards/noligand.mtz D /trunk/wizards/omit_lig_results.list D /trunk/wizards/omit_regr.tgz D /trunk/wizards/p.pdb D /trunk/wizards/p21.sym D /trunk/wizards/p9-seq.dat D /trunk/wizards/p9-sites.xyz D /trunk/wizards/p9_ha.pdb D /trunk/wizards/p9_se_w2.sca D /trunk/wizards/part_mlt.pdb D /trunk/wizards/partial.pdb D /trunk/wizards/partial_refine_001.pdb D /trunk/wizards/partial_refine_001_map_coeffs.mtz D /trunk/wizards/peak.sca D /trunk/wizards/perfect.mtz D /trunk/wizards/perfect_rna.mtz D /trunk/wizards/phaser_1_offset.mtz D /trunk/wizards/pt.sca D /trunk/wizards/quick_clashlist.standard D /trunk/wizards/random.pdb D /trunk/wizards/random_0.7_mlt.mtz D /trunk/wizards/resolve_1_offset.mtz D /trunk/wizards/rna.seq D /trunk/wizards/run_multi.eff D /trunk/wizards/run_multi_new.eff D /trunk/wizards/seq.dat D /trunk/wizards/seq1.dat D /trunk/wizards/seq2.dat D /trunk/wizards/seq_mlt.dat D /trunk/wizards/short_seq.standard D /trunk/wizards/side.pdb D /trunk/wizards/sir_sad.eff D /trunk/wizards/sites_mlt.pdb D /trunk/wizards/smiles.dat D /trunk/wizards/solve.setup D /trunk/wizards/start.pdb D /trunk/wizards/starting_map.mtz D /trunk/wizards/test3.gz D /trunk/wizards/test9.gz D /trunk/wizards/test_ampl_int_autobuild.tgz D /trunk/wizards/test_labels_1.std D /trunk/wizards/test_labels_2.std D /trunk/wizards/test_labels_3.std D /trunk/wizards/test_missing.std D /trunk/wizards/test_multi.tgz D /trunk/wizards/test_ncs.standard D /trunk/wizards/test_ncs_find_ncs.standard D /trunk/wizards/test_ncs_lc.pdb D /trunk/wizards/test_resno.std D /trunk/wizards/test_resno.tgz D /trunk/wizards/unmerged.mtz D /trunk/wizards/w1.sca D /trunk/wizards/w1_rna.sca D /trunk/wizards/w2.sca remove unneeded files ------------------------------------------------------------------------ Cheers, Richard Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

iotbx.merging_statistics test failing: data file removed from phenix_regression
by richard.gildea＠diamond.ac.uk 06 Jan '17

06 Jan '17

Hi all, The test for iotbx.merging_statistics has recently started failing as a result of one of the files it uses for testing being deleted, phenix_regression/wizards/p9_se_w2.sca: https://github.com/cctbx/cctbx_project/blob/master/iotbx/regression/tst_mer… Would it be possible to restore this file to phenix_regression so that the test will pass? $ svn log -r r3879 -v ------------------------------------------------------------------------ r3879 | terwill | 2016-12-29 02:48:15 +0000 (Thu, 29 Dec 2016) | 1 line Changed paths: D /trunk/wizards/000README D /trunk/wizards/4c5q.pdb D /trunk/wizards/ATP.pdb D /trunk/wizards/MM-all.tgz D /trunk/wizards/NAD.pdb D /trunk/wizards/PARAMS_1.eff D /trunk/wizards/a2u-data.mtz D /trunk/wizards/a2u-sigmaa.mtz D /trunk/wizards/a2u.sca D /trunk/wizards/a2u_short.pdb D /trunk/wizards/anb.pdb D /trunk/wizards/autosol_unmerged_False.std D /trunk/wizards/autosol_unmerged_True.std D /trunk/wizards/autosol_unmerged_p9.std D /trunk/wizards/bad_model.pdb D /trunk/wizards/check.xyz D /trunk/wizards/check_multiple_input_files_042908.tgz D /trunk/wizards/check_multiple_input_files_062608.tgz D /trunk/wizards/check_multiple_input_files_070508.tgz D /trunk/wizards/check_multiple_input_files_2011-03-03.tgz D /trunk/wizards/check_multiple_input_files_2011-06-29.tgz D /trunk/wizards/check_multiple_input_files_2011-11-04.tgz D /trunk/wizards/clean.com D /trunk/wizards/cocn.cif D /trunk/wizards/coords.pdb D /trunk/wizards/coords1.ali D /trunk/wizards/coords1.mtz D /trunk/wizards/coords1.pdb D /trunk/wizards/coords_A001.pdb D /trunk/wizards/coords_rna.pdb D /trunk/wizards/coords_rna_gap.pdb D /trunk/wizards/data_for_commands_in_doc.tgz D /trunk/wizards/find_all_ligands.eff D /trunk/wizards/find_all_ligands.standard D /trunk/wizards/find_ncs_ncs.standard D /trunk/wizards/find_ncs_resolve.standard D /trunk/wizards/fobs.mtz D /trunk/wizards/fobs_std.mtz D /trunk/wizards/fobs_std_edit.mtz D /trunk/wizards/gene-5.pdb D /trunk/wizards/generate.com D /trunk/wizards/get_cc_mtz_mtz.standard D /trunk/wizards/get_cc_mtz_pdb.standard D /trunk/wizards/ha.pdb D /trunk/wizards/ha_lig.pdb D /trunk/wizards/ha_perfect.pdb D /trunk/wizards/ha_perfect_2.pdb D /trunk/wizards/hipip.pdb D /trunk/wizards/hipip_2.sca D /trunk/wizards/hipip_shortseq.dat D /trunk/wizards/ignore.dat D /trunk/wizards/inf.sca D /trunk/wizards/ki.sca D /trunk/wizards/ligand.pdb D /trunk/wizards/mad.eff D /trunk/wizards/mad.mtz D /trunk/wizards/map_coeffs.mtz D /trunk/wizards/map_coeffs_1.mtz D /trunk/wizards/map_coeffs_2.mtz D /trunk/wizards/map_to_object.standard D /trunk/wizards/maps_only.standard D /trunk/wizards/mir.eff D /trunk/wizards/mlt.mtz D /trunk/wizards/mlt_gap.pdb D /trunk/wizards/mlt_regression.tgz D /trunk/wizards/mol1.pdb D /trunk/wizards/mol2.pdb D /trunk/wizards/mr_rosetta.eff D /trunk/wizards/mult_regr.tgz D /trunk/wizards/multi.pdb D /trunk/wizards/multi.pdb.mtz D /trunk/wizards/my_autobuild.eff D /trunk/wizards/my_automr.eff D /trunk/wizards/my_autosol.eff D /trunk/wizards/my_ligand.pdb D /trunk/wizards/my_ligandfit.eff D /trunk/wizards/my_target.pdb D /trunk/wizards/native.sca D /trunk/wizards/noligand.mtz D /trunk/wizards/omit_lig_results.list D /trunk/wizards/omit_regr.tgz D /trunk/wizards/p.pdb D /trunk/wizards/p21.sym D /trunk/wizards/p9-seq.dat D /trunk/wizards/p9-sites.xyz D /trunk/wizards/p9_ha.pdb D /trunk/wizards/p9_se_w2.sca D /trunk/wizards/part_mlt.pdb D /trunk/wizards/partial.pdb D /trunk/wizards/partial_refine_001.pdb D /trunk/wizards/partial_refine_001_map_coeffs.mtz D /trunk/wizards/peak.sca D /trunk/wizards/perfect.mtz D /trunk/wizards/perfect_rna.mtz D /trunk/wizards/phaser_1_offset.mtz D /trunk/wizards/pt.sca D /trunk/wizards/quick_clashlist.standard D /trunk/wizards/random.pdb D /trunk/wizards/random_0.7_mlt.mtz D /trunk/wizards/resolve_1_offset.mtz D /trunk/wizards/rna.seq D /trunk/wizards/run_multi.eff D /trunk/wizards/run_multi_new.eff D /trunk/wizards/seq.dat D /trunk/wizards/seq1.dat D /trunk/wizards/seq2.dat D /trunk/wizards/seq_mlt.dat D /trunk/wizards/short_seq.standard D /trunk/wizards/side.pdb D /trunk/wizards/sir_sad.eff D /trunk/wizards/sites_mlt.pdb D /trunk/wizards/smiles.dat D /trunk/wizards/solve.setup D /trunk/wizards/start.pdb D /trunk/wizards/starting_map.mtz D /trunk/wizards/test3.gz D /trunk/wizards/test9.gz D /trunk/wizards/test_ampl_int_autobuild.tgz D /trunk/wizards/test_labels_1.std D /trunk/wizards/test_labels_2.std D /trunk/wizards/test_labels_3.std D /trunk/wizards/test_missing.std D /trunk/wizards/test_multi.tgz D /trunk/wizards/test_ncs.standard D /trunk/wizards/test_ncs_find_ncs.standard D /trunk/wizards/test_ncs_lc.pdb D /trunk/wizards/test_resno.std D /trunk/wizards/test_resno.tgz D /trunk/wizards/unmerged.mtz D /trunk/wizards/w1.sca D /trunk/wizards/w1_rna.sca D /trunk/wizards/w2.sca remove unneeded files ------------------------------------------------------------------------ Cheers, Richard Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

easy_mp.multi_core_run function
by R.D. Oeffner 19 Dec '16

19 Dec '16

Hi, I needed a way to do multiprocessing job runs in a simple fashion on a multicore PC. Gabor provided much of the underlying machinery in libtbx for that. But I have often found I'm wasting too much time setting it all up. So I crafted a small function, easy_mp.multi_core_run(), out of his example in Computational Crystallography Newsletter (2013). 4, p21 which does exactly that. Unlike easy_mp.parallel_map() this function continues if one of the jobs crashes catastrophically. It takes as arguments name of function to be parallelised, a list of argument tuples and number of processors to run on. As the jobs completes it returns an iterator of tuples of arguments with corresponding results. Example: def RunMyJob(args1, args2): pass def GatherResults(parmres): pass from libtbx import easy_mp argstuples = [(args1a, args2a), (args1b, args2b), (args1c, args2c) ] for i, parmres in enumerate(easy_mp.multi_core_run( RunMyJob, argstuples, nproc)): GatherResults( parmres ) I'm interested in comments or if this kind of simple function is already available elsewhere. Regards, Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234 mobile: +44(0)7712 887162

1 0

Making branches by accident
by Graeme.Winter＠diamond.ac.uk 19 Dec '16

19 Dec '16

Morning all I am seeing a certain amount of "Merge branch 'master' of github.com:cctbx/cctbx_project" coming through on the commits - this usually means you did not do a git pull -rebase before the git push. This can be set to the default by using the spell Markus sent out git config --global pull.rebase true This will need to be done on each machine you push from, else getting the habit of doing a git pull -rebase before push is a good one. We have had this on and off with DIALS but it tends to pass easily enough. What bad happens? Nothing really but the history becomes confusing... So: may be worth checking that you have the pull.rebase thing set? Cheerio Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

6 11

Re: [cctbxbb] issue tracker on https://github.com/cctbx/cctbx_project
by markus.gerstel＠diamond.ac.uk 10 Dec '16

10 Dec '16

Hi, Yes, we can do that with the tools we have. (Jenkins + Python + Github API) -Markus On 9 Dec 2016 16:13, Graeme.Winter(a)diamond.ac.uk wrote: Hi Nigel, all That could be harder but not beyond our capabilities I expect However the response seems to be a resounding go for it, so I’m sure the issue tracker will be activated RSN Best wishes Graeme From: cctbxbb-bounces(a)phenix-online.org [mailto:[email protected]] On Behalf Of Nigel Moriarty Sent: 09 December 2016 15:43 To: cctbx mailing list Subject: Re: [cctbxbb] issue tracker on https://github.com/cctbx/cctbx_project One feature that would be cool is to set a (say) weekly email to remind developers of the issues assigned to them. In my other collabs, there is a complaints that the issues are not being noticed... Cheers Nigel --- Nigel W. Moriarty Building 33R0349, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709 Email : NWMoriarty(a)LBL.gov<mailto:[email protected]> Fax : 510-486-5909 Web : CCI.LBL.gov<http://CCI.LBL.gov> On Fri, Dec 9, 2016 at 7:40 AM, Aaron Brewster <asbrewster(a)lbl.gov<mailto:[email protected]>> wrote: Also +1. I like how it's possible to reference issues in commits by using #, which then show up as links in the commit emails and website history. One can cite issue numbers, or even interact with them. For example, if your commit includes the phrase "closes #42" or "fixes #42", then issue 42 will be closed when you push your commit. It's Nifty. -Aaron On Fri, Dec 9, 2016 at 7:29 AM, Nigel Moriarty <nwmoriarty(a)lbl.gov<mailto:[email protected]>> wrote: We at Phenix would like to see how the issue/bug tracker works as it's an important feature we'd like to use when we migrate our other repos to GitHub. Cheers Nigel --- Nigel W. Moriarty Building 33R0349, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709<tel:%28510%29%20486-5709> Email : NWMoriarty(a)LBL.gov<mailto:[email protected]> Fax : 510-486-5909<tel:%28510%29%20486-5909> Web : CCI.LBL.gov<http://CCI.LBL.gov> On Fri, Dec 9, 2016 at 6:23 AM, Luc Bourhis <luc_j_bourhis(a)mac.com<mailto:[email protected]>> wrote: Hi Graeme, Does anyone have an objection to opening up the issue tracker on […] Your thoughts? +1 Best wishes, Luc _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org<mailto:[email protected]> http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org<mailto:[email protected]> http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org<mailto:[email protected]> http://phenix-online.org/mailman/listinfo/cctbxbb -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

issue tracker on https://github.com/cctbx/cctbx_project
by Graeme.Winter＠diamond.ac.uk 10 Dec '16

10 Dec '16

Good afternoon all Does anyone have an objection to opening up the issue tracker on https://github.com/cctbx/cctbx_project at the moment we make a lot of use of issue trackers for dials and xia2, and for cctbx this would be helpful - so we could for example host issues in a more appropriate place like https://github.com/dials/dials/issues/265 which would be more "at home" in cctbx Your thoughts? Thanks & best wishes Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

4 6

Moving cctbx to git
by markus.gerstel＠diamond.ac.uk 23 Nov '16

23 Nov '16

Hello everyone, As Aaron requested - here is an update regarding the move to GitHub. At this time if you want to commit code: *keep committing to Sourceforge*. Do not worry about leaving uncommitted changes in your SVN tree. Those can be migrated later. The Sourceforge cctbx repository history has now been migrated to GitHub. The repository lives at https://github.com/cctbx/cctbx. Please do not commit to this repository yet! Over the next few days the bootstrap script will be modified alongside the build servers to point to the GitHub repository. This means there will only be limited testing, and the produced builds will lag behind the development effort. To be listed as a developer on https://github.com/cctbx you need to set your visibility to public at https://github.com/orgs/cctbx/people<%20https:/github.com/orgs/cctbx/people>. You can also view some fancy graphs about the cctbx project history at https://github.com/cctbx/cctbx/graphs/contributors<%20https:/github.com/cctbx/cctbx/graphs/contributors> If you haven't done it before, now is a good time to set up your local installation of git to use your name and mail address with git config --global user.name "John Doe" git config --global user.email your_email(a)example.com<mailto:[email protected]> as well as this setting to avoid merge commits: git config --global pull.rebase true Note that you must set these on every computer you use! You may want to keep a copy of those commands around in case you ever get a new computer. You should use mail addresses registered with your github account, or - if you enable it in your github account - the anonymous address offered by github. You may want to use the time to sort out your ssh access keys with your GitHub account, and play around with git a bit. You can test that everything works using the temporary sandbox repository of cctbx. Here are some commands to get you started: git clone [email protected]:cctbx/cctbx-playground.git cd cctbx-playground echo some text >> some-file git add some-file git commit -m "some commit" git push git log Then go to https://github.com/cctbx/cctbx-playground/commits/master to see how other people will see your commits. The next steps: *Tuesday 8th November 2016, 11:59pm Pacific Time* *= Wednesday 9th November 2016, 07:59am GMT:* Sourceforge: Committing allowed. GitHub: Checkout/Cloning allowed, *Read-only*. We will update our build servers and the bootstrap script to refer to the git repositories. All commits still run into Sourceforge, and will be copied over to GitHub automatically every couple of hours. Since the build servers will only see code on GitHub, testing and deployment will be limited during this period. *Tuesday 22nd November 2016, 11:59pm Pacific Time* *= Wednesday 23rd November 2016, 07:59am GMT:* Sourceforge: *Read-only*. GitHub: Committing allowed We should have achieved a stable build on both Jenkins and Buildbot by this time. Change any website pointers from SF to GitHub. Instructions to migrate local developer repositories will be sent around. All future commits should go to GitHub only. Remaining commits on Sourceforge will be migrated to GitHub. -Markus Dr Markus Gerstel MBCS Postdoctoral Research Associate Tel: +44 1235 778698 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

5 8

Re: [cctbxbb] cctbxbb Digest, Vol 92, Issue 17
by Tijskens Engelbert 22 Nov '16

22 Nov '16

> Message: 2 > Date: Mon, 21 Nov 2016 13:22:20 +0000 > From: Gabor Bunkoczi <gabor.bunkoczi(a)googlemail.com> > To: cctbx mailing list <cctbxbb(a)phenix-online.org> > Subject: Re: [cctbxbb] convert an arbitrary point to one inside the > asymmetric unit > Message-ID: > <CABYxhoWJgnRWL0Tun6M9MzcQAWNfy77YMNRSqEw+L4_b5yFkvA(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Dear Engelbert, > > you can use the asu_mappings objects to compute this (there may be > something more suitable for regular grids). Here is a link to a tutorial > written by Ralf Grosse-Kunstleve, where this is explained: > > http://cci.lbl.gov/publications/download/iucrcompcomm_aug2004.pdf > > BW, Gabor I have looked into this, but if i understand it well, it does exactly the opposite of what i want: it extends the asymmetric unit to a spherical region and makes symmetric copies of some specified points in the (scatterers in the tutorial mentioned). i have points outside the asymmetric unit, and i want to find the corresponding points inside the asymmetric unit from which i can interpolate the electron density. kindest regards and still hoping … bert

2 1

convert an arbitrary point to one inside the asymmetric unit
by Tijskens Engelbert 21 Nov '16

21 Nov '16

Hi, I am an experienced C++ and Python programmer, but new to cctbx. Can any tell me if there is an algorithm in cctbx available to convert an arbritrary point in direct space to the corresponding point in the asymmetric unit of a given space group? i can figure out how to convert it to the corresponding point in the unit cell at the origin, but from there to the asymmetric unit is unclear to me. I found the brick class which gives me the assymmetric unit from cctbx import crystal,sgtbx if __name__=='__main__': crystal_symmetry = crystal.symmetry( unit_cell=(3.5093,3.5093,3.5093,90,90,90) , space_group_symbol="229" # = "I m -3 m" ) sg = crystal_symmetry.space_group() brick = sgtbx.brick( sg.type() ) print brick I also found cctbx::sgtbx::asu and the classes therein, but they are largely undocumented. For the interested, this is the problem i want to solve: I have a (scalar) physical property P(r), depending on spatial position, presented as a 3D array on a regular grid over the unit cell (space group is known). I need to integrate P(r)*f(r), f(r) being an arbitrary function over a region of direct space, which typically comprises several unit cells. In order to reduce the memory footprint of the P array, i would like to restrict it to the space group’s asymmetric unit. All help is greatly appreciated. kindest regards, Dr [Engel]bert Tijskens HPC Analyst/Consultant Flemish Supercomputer Center vscentrum.be<http://vscentrum.be> HPC core facility CalcUA uantwerp.be/calcua<http://uantwerp.be/calcua> Computational mathematics www.uantwerpen.be/en/rg/cma<http://www.uantwerpen.be/en/rg/cma> University of Antwerp Middelheimlaan1 G.309, B-2020 Antwerp, Belgium engelbert.tijskens(a)uantwerpen.be<mailto:[email protected]> ++32 3 265 3879

3 3

Re: [cctbxbb] cctbxbb Digest, Vol 92, Issue 17
by Tijskens Engelbert 21 Nov '16

21 Nov '16

> Message: 3 > Date: Mon, 21 Nov 2016 13:38:21 +0000 > From: Gabor Bunkoczi <gabor.bunkoczi(a)googlemail.com> > To: cctbx mailing list <cctbxbb(a)phenix-online.org> > Subject: Re: [cctbxbb] convert an arbitrary point to one inside the > asymmetric unit > Message-ID: > <CABYxhoUxqFsc0f_TjWwn4oC4hKFk8AEbzwx5=9=_+gwW3xmkpg(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > On second thought, the simplest solution is to use the direct_space_asu > object. This is available for all space groups: > >>>> from cctbx import sgtbx >>>> sgi = sgtbx.space_group_info( "P2" ) >>>> asu = sgi.direct_space_asu() >>>> asu.is_inside( ( 0, 0, 0 ) ) > True >>>> asu.is_inside( (0, 0, 0.51 ) ) > False > > BW, Gabor > Thanks for your quick reply, Gabor. Unfortunately, it does not solve my question. For the example you give above, the second point is not inside the asu, and my question is how i can the corresponding point inside the asu, For a simple space group as P2 the answer is easily figured out by hand. For a space group such as I m -3 m, it becomes complicated. I would like to have a (computationally efficient) method for finding the point inside the asu given an arbitrary point which works for every space group. Any “third” thoughts? kindest regards, bert

2 1

Re: [cctbxbb] Unstable platform-dependent sort order for flex.sort_permutation
by richard.gildea＠diamond.ac.uk 20 Nov '16

20 Nov '16

Hi Oleg, The permutations would lead to the same sequence of miller indices, but in the context of an unmerged reflection array (i.e. miller.array.sort_permutation), then the data associated with those miller indices would not necessarily be in the same order. As the dataset is then split into two half-datasets, this difference in sort order leads to a different value of the calculate correlation coefficient between those two half datasets: https://github.com/cctbx/cctbx_project/blob/master/cctbx/miller/__init__.py… Cheers, Richard Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE ________________________________ From: cctbxbb-bounces(a)phenix-online.org [cctbxbb-bounces(a)phenix-online.org] on behalf of oleg(a)olexsys.org [oleg(a)olexsys.org] Sent: 18 November 2016 15:59 To: cctbx mailing list Subject: Re: [cctbxbb] Unstable platform-dependent sort order for flex.sort_permutation Hi Richard, it is a bug only if the permutations do not lead to the same sequence. Otherwise you cannot expect to get the same sorting permutations for collections with redundant data on different platforms or even between different versions of compilers. Cheers, Oleg. ________________________________ From: cctbxbb-bounces(a)phenix-online.org <cctbxbb-bounces(a)phenix-online.org> on behalf of richard.gildea(a)diamond.ac.uk <richard.gildea(a)diamond.ac.uk> Sent: 18 November 2016 14:12:29 To: cctbxbb(a)phenix-online.org Subject: [cctbxbb] Unstable platform-dependent sort order for flex.sort_permutation Hi, I've been trying to diagnose why the miller array cc_anom calculations seem to be platform dependent. I've narrowed this down to a variation in the sort order returned by flex.sort_permutation which is called from within miller.array.sort("packed_indices"). The following code demonstrates the problem and the different output I get on mac and Linux: Mac: >>> from scitbx.array_family import flex >>> a = flex.size_t([7, 1, 1, 5, 1, 3, 3, 7, 1, 7, 3, 3, 5, 7, 5, 5, 5, 1, 1, 1]) >>> print list(flex.sort_permutation(a)) [1, 2, 4, 8, 17, 18, 19, 5, 6, 10, 11, 3, 12, 14, 15, 16, 0, 7, 9, 13] Linux: >>> from scitbx.array_family import flex >>> a = flex.size_t([7, 1, 1, 5, 1, 3, 3, 7, 1, 7, 3, 3, 5, 7, 5, 5, 5, 1, 1, 1]) >>> print list(flex.sort_permutation(a)) [19, 1, 2, 18, 4, 17, 8, 11, 10, 6, 5, 12, 14, 15, 16, 3, 9, 7, 13, 0] Is it known/expected for the sort order to be platform dependent, or is this a bug? Here is the relevant code for flex.sort_permutation(): https://github.com/cctbx/cctbx_project/blob/master/scitbx/array_family/sort… Cheers, Richard Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org http://phenix-online.org/mailman/listinfo/cctbxbb

5 9

Unstable platform-dependent sort order for flex.sort_permutation
by richard.gildea＠diamond.ac.uk 18 Nov '16

18 Nov '16

Hi, I've been trying to diagnose why the miller array cc_anom calculations seem to be platform dependent. I've narrowed this down to a variation in the sort order returned by flex.sort_permutation which is called from within miller.array.sort("packed_indices"). The following code demonstrates the problem and the different output I get on mac and Linux: Mac: >>> from scitbx.array_family import flex >>> a = flex.size_t([7, 1, 1, 5, 1, 3, 3, 7, 1, 7, 3, 3, 5, 7, 5, 5, 5, 1, 1, 1]) >>> print list(flex.sort_permutation(a)) [1, 2, 4, 8, 17, 18, 19, 5, 6, 10, 11, 3, 12, 14, 15, 16, 0, 7, 9, 13] Linux: >>> from scitbx.array_family import flex >>> a = flex.size_t([7, 1, 1, 5, 1, 3, 3, 7, 1, 7, 3, 3, 5, 7, 5, 5, 5, 1, 1, 1]) >>> print list(flex.sort_permutation(a)) [19, 1, 2, 18, 4, 17, 8, 11, 10, 6, 5, 12, 14, 15, 16, 3, 9, 7, 13, 0] Is it known/expected for the sort order to be platform dependent, or is this a bug? Here is the relevant code for flex.sort_permutation(): https://github.com/cctbx/cctbx_project/blob/master/scitbx/array_family/sort… Cheers, Richard Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

convert an arbitrary point to one inside the asymmetric unit
by Tijskens Engelbert 18 Nov '16

18 Nov '16

Hi, I am an experienced C++ and Python programmer, but new to cctbx. Can any tell me if there is an algorithm in cctbx available to convert an arbritrary point in direct space to the corresponding point in the asymmetric unit of a given space group? i can figure out how to convert it to the corresponding point in the unit cell at the origin, but from there to the asymmetric unit is unclear to me. I found the brick class which gives me the assymmetric unit from cctbx import crystal,sgtbx if __name__=='__main__': crystal_symmetry = crystal.symmetry( unit_cell=(3.5093,3.5093,3.5093,90,90,90) , space_group_symbol="229" # = "I m -3 m" ) sg = crystal_symmetry.space_group() brick = sgtbx.brick( sg.type() ) print brick I also found cctbx::sgtbx::asu and the classes therein, but they are largely undocumented. For the interested, this is the problem i want to solve: I have a (scalar) physical property P(r), depending on spatial position, presented as a 3D array on a regular grid over the unit cell (space group is known). I need to integrate P(r)*f(r), f(r) being an arbitrary function over a region of direct space, which typically comprises several unit cells. In order to reduce the memory footprint of the P array, i would like to restrict it to the space group’s asymmetric unit. All help is greatly appreciated. kindest regards, Dr [Engel]bert Tijskens HPC Analyst/Consultant Flemish Supercomputer Center vscentrum.be<http://vscentrum.be> HPC core facility CalcUA uantwerp.be/calcua<http://uantwerp.be/calcua> Computational mathematics www.uantwerpen.be/en/rg/cma<http://www.uantwerpen.be/en/rg/cma> University of Antwerp Middelheimlaan1 G.309, B-2020 Antwerp, Belgium engelbert.tijskens(a)uantwerpen.be<mailto:[email protected]> ++32 3 265 3879

1 0

who else has troubles to commit to cctbx now?
by Pavel Afonine 16 Nov '16

16 Nov '16

Can't commit anything since yesterday: pva_mbp:/Users/pafonine/phenix_dev/modules/cctbx_project/mmtbx % svn commit -m "test" maps/correlation.py Sending maps/correlation.py Transmitting file data .svn: E000013: Commit failed (details follow): svn: E000013: Can't create directory '/svn/p/cctbx/code/db/transactions/25717-1.txn': Permission denied pva_mbp:/Users/pafonine/phenix_dev/modules/cctbx_project/mmtbx % Nigel has the same issue. I don't see any warnings on sourceforge. Pavel

3 3

command line syntax for multiple scope PHIL
by R.D. Oeffner 09 Nov '16

09 Nov '16

Say a program, myprog, has a master PHIL definition like: minimization.input .multiple = True { file_name = None .type = str nval = None .type = int } and a PHIL input file like: minimization.input { file_name = experiment1.dat nval = 42 } minimization.input { file_name = experiment2.dat nval = 13 } so that the user on the command line can type "myprog phil_input.txt" What would be the equivalent command line syntax of the PHIL input above? Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234 mobile: +44(0)7712 887162

2 2

Failing shelxl test
by Graeme.Winter＠diamond.ac.uk 09 Nov '16

09 Nov '16

Folks, I keep getting this error: libtbx.python "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/development/run_shelx.py" P31 [FAIL] Time: 1.38 Return code: 1 OKs: 0 Standard error: Traceback (most recent call last): File "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/development/run_shelx.py", line 253, in <module> run(args=sys.argv[1:]) File "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/development/run_shelx.py", line 250, in run debug_utils.parse_options_loop_space_groups(args, run_call_back) File "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/development/debug_utils.py", line 120, in parse_options_loop_space_groups show_cpu_times, **kwds) File "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/development/debug_utils.py", line 83, in loop_space_groups call_back_result = call_back(flags, space_group_info, **kwds) File "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/development/run_shelx.py", line 244, in run_call_back verbose=flags.Verbose) File "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/development/run_shelx.py", line 234, in exercise run_shelx(shelx_titl, structure_factors, verbose=verbose) File "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/development/run_shelx.py", line 210, in run_shelx check_anisou(shelx_titl, xray_structure, shelx_pdb, verbose) File "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/development/run_shelx.py", line 163, in check_anisou assert TotalMismatches == 0 AssertionError Please verify these tests manually. it is annoying as I understand it is to do with a change in shelxl - would it be OK to comment the test out to allow builds to pass? I am not sure it is useful at the moment Thank you Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

Re: [cctbxbb] Git status update
by markus.gerstel＠diamond.ac.uk 04 Nov '16

04 Nov '16

Hello everyone, since I have received no complaints and heard nothing to the contrary, so I thought I would let you know on the current state of the git migration: Until *Thursday, 1 September 2016*, 07:59 BST/08:59 CEST/11:59 pm PDT Cctbx development happens on Sourceforge. Note that cctbx development is "frozen" in preparation for the phenix release. If you are actively developing cctbx you should *now* sign up for a GitHub account. If you are not among the 27 people that have already joined the cctbx project at https://github.com/cctbx then please send me a mail with your github account name so I can invite you. In preparation for the move to github you should also install git on your machines. Set up your local installation of git on each machine you are using by running > git config --global pull.rebase true > git config --global user.name "John Doe" > git config --global user.email "your_email(a)example.com" Use an email address you have set up in your GitHub account at https://github.com/settings/emails. Note that this address will be forever publically readable! If you do not want your mail address to be public, you need to verify any (private) mail address with GitHub and then check the /Keep my email address private/ box at the link above. You can then use the dummy address <yourgithubname>@users.noreply.github.com instead. Other useful settings may be > git config --global alias.up "pull --rebase" To enable a *git up* command working much like svn up does. > git config --global alias.unstage "reset HEAD" *git unstage* to easily undo *git add*/*git rm*. > git config --global alias.mkremotebranch "push origin -u" *git mkremotebranch* will copy your current local branch to GitHub. > git config --global alias.rmremotebranch "push origin --delete" *git rmremotebranch <name>* will delete said branch from GitHub. > git config --global color.ui auto This adds color to the git output, which some may find useful for eg. *git status*. On the 1st of September the transitional period starts, and we will begin changing scripts to point to the new code locations. The transition may be delayed somewhat if the phenix release encounters any delays. I will send you further updates closer to the day. You can find and comment on the transition plan at https://docs.google.com/document/d/1bTiGbnvw8lNUrBLbRmhEdnd22McDqBbiKxcpyga… Best wishes -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

6 16

use_internal_variance in iotbx.merging_statistics
by Keitaro Yamashita 01 Nov '16

01 Nov '16

Dear Phenix/CCTBX developers, iotbx/merging_statistics.py is used by phenix.merging_statistics, phenix.table_one, and so on. By upgrading phenix from 1.10.1 to 1.11, merging statistics-related codes were significantly changed. Previously, miller.array.merge_equivalents() was always called with argument use_internal_variance=False, which is consistent with XDS, Aimless and so on. Currently, use_internal_variance=True is default, and cannot be changed by users (see below). These changes were made by @afonine and @rjgildea in rev. 22973 (Sep 26, 2015) and 23961 (Mar 8, 2016). Could anyone explain why these changes were introduced? https://sourceforge.net/p/cctbx/code/22973 https://sourceforge.net/p/cctbx/code/23961 My points are: - We actually cannot control use_internal_variance= parameter because it is not passed to merge_equivalents() in class filter_intensities_by_sigma. - In previous versions, if I gave XDS output to phenix.merging_statistics, <I/sigma> values calculated in the same way (as XDS does) were shown; but not in the current version. - For (for example) phenix.table_one users who expect this behavior, it can give inconsistency. The statistics would not be consistent with the data used in refinement. cf. the related discussion in cctbxbb: http://phenix-online.org/pipermail/cctbxbb/2012-October/000611.html Best regards, Keitaro

4 6

Monthly "how far are we from moving to git" message
by Graeme.Winter＠diamond.ac.uk 21 Oct '16

21 Oct '16

Good afternoon / morning / evening (depending on your locale) I'm just running a bootstrap build and having to check out the whole of cctbx one ... file ... at ... a ... time from svn & it reminded me we were going to move over to git at some point - ISTR was determined by the PHENIX release. Just wondering how this is going & how long it is likely to be until we can get this ball rolling :) Thanks & best wishes Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 6

sginfo and sgtbx
by Pascal 21 Oct '16

21 Oct '16

Hi, Just out of curiosity, sginfo website mentions that it has been superseeded by sgtbx. Is that a brand new implementation? I don't recognize sginfo code at all. Also, anyone can confirm that sgtbx is tightly embedded is cctbx and cannot be used standalone? Regards, Pascal

4 6

flex.int.as_double() performance
by richard.gildea＠diamond.ac.uk 11 Oct '16

11 Oct '16

Hi, Whilst looking into performance of various DIALS routines recently, I noticed that the flex.int.as_double() conversion takes roughly 3x as long as the equivalent functionality in numpy: (the choice of a 2527x2463 array is simply because this is the size of the raw data array for a Pilatus 6M detector) import time from dials.array_family import flex n = 100 a = flex.int(flex.grid(2527, 2463)) t0 = time.time() for i in range(n): b = a.as_double() t1 = time.time() t = t1-t0 print "flex.int.as_double(): %.2fs (%.3fs/call)" %(t, t/n) import numpy as np a = np.random.randint(100, size=a.all()) t0 = time.time() for i in range(n): b = a.astype(np.float64) t1 = time.time() t = t1-t0 print "numpy.astype(np.float64): %.2fs (%.3fs/call)" %(t, t/n) $ libtbx.python time_as_double.py flex.int.as_double(): 5.76s (0.058s/call) numpy.astype(np.float64): 1.86s (0.019s/call) This is the current version of the as_double function: static flex_double as_double(f_t const& a) { shared_plain<double> result(a.size(), init_functor_null<double>()); for(std::size_t i=0;i<a.size();i++) result[i] = a[i]; return flex_double(result, a.accessor()); } Replacing the above code with the following, I see a significant runtime improvement, now flex.int.as_double() is marginally quicker than the equivalent numpy routine: static flex_double as_double(f_t const& a) { shared_plain<double> result(a.begin(), a.end()); return flex_double(result, a.accessor()); } $ libtbx.python time_as_double.py flex.int.as_double(): 1.64s (0.016s/call) numpy.astype(np.float64): 1.82s (0.018s/call) Is there any obvious reason why making this change would be A Bad Thing? In the context of dials spot-finding, this change results in a 22s saving (out of a total 172s with the original code) when running dials.find_spots on a 540 image Pilatus 6M dataset using a single processor, so it is definitely a noticeable speed-up. All timings done on a macbook pro. Cheers, Richard Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 3

"import pickle"
by markus.gerstel＠diamond.ac.uk 23 Sep '16

23 Sep '16

Hi everyone, I note that we are using "import pickle" on some 57 occasions. Is there a particular reason we don't import cPickle? For example: --- libtbx/env_config.py (revision 25451) +++ libtbx/env_config.py (working copy) @@ -7,7 +7,10 @@ from libtbx import adopt_init_args import platform import shutil -import pickle +try: + import cPickle as pickle +except: + import pickle from cStringIO import StringIO import re import sys, os This should still work on all systems without issue. According to python docs the only difference is that cPickle can't be subclassed. If it's alright with you I would go through those imports and update them to use cPickle where possible. -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 2

bootstrap / svn method / say reason not anonymous && http
by Graeme.Winter＠diamond.ac.uk 20 Sep '16

20 Sep '16

Hello folks, In bootstrap.py is there any logical reason why we can’t use %sfmethod for anonymous as well as authenticated access to the cctbx_project sources i.e. in # Core CCTBX repositories # These must all provide anonymous access. class cctbx_module(SourceModule): module = 'cctbx_project' anonymous = ['svn','svn://svn.code.sf.net/p/cctbx/code/trunk'] authenticated = ['svn', '%(sfmethod)s://%(sfuser)[email protected]/p/cctbx/code/trunk'] also respect %(sfmethod) in place of svn as transport layer. Will help people behind firewalls. Appreciate that this will also need changing when we move to git anyhow… but meantimes there are people out there this would help. Not keen to make this change myself without checking first…! Thanks & best wishes Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0