Hi, I invited you. At this point the historical commits are not fixed yet, so I will change them to reflect your username. Best wishes -Markus Dr Markus Gerstel MBCS Postdoctoral Research Associate Tel: +44 1235 778698 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE From: [email protected] [mailto:[email protected]] On Behalf Of Dorothee Liebschner Sent: 17 August 2016 18:55 To: cctbx mailing list Subject: Re: [cctbxbb] Git status update Hi Markus, Could you please invite me for the cctbx project? I just created a github account and my account name is dcliebschner (when I answered your last email, I choose another name, but I guess it is better to have same user name consistently) Best wishes, Dorothee On Wed, Aug 17, 2016 at 7:17 AM, mailto:[email protected]> wrote: Hello everyone, since I have received no complaints and heard nothing to the contrary, so I thought I would let you know on the current state of the git migration: Until *Thursday, 1 September 2016*, 07:59 BST/08:59 CEST/11:59 pm PDT Cctbx development happens on Sourceforge. Note that cctbx development is "frozen" in preparation for the phenix release. If you are actively developing cctbx you should *now* sign up for a GitHub account. If you are not among the 27 people that have already joined the cctbx project at https://github.com/cctbx then please send me a mail with your github account name so I can invite you. In preparation for the move to github you should also install git on your machines. Set up your local installation of git on each machine you are using by running

git config --global pull.rebase true git config --global user.namehttp://user.name "John Doe" git config --global user.email "[email protected]mailto:[email protected]" Use an email address you have set up in your GitHub account at https://github.com/settings/emails. Note that this address will be forever publically readable! If you do not want your mail address to be public, you need to verify any (private) mail address with GitHub and then check the /Keep my email address private/ box at the link above. You can then use the dummy address <yourgithubname>@users.noreply.github.comhttp://users.noreply.github.com instead.

Other useful settings may be

git config --global alias.up "pull --rebase" To enable a *git up* command working much like svn up does. git config --global alias.unstage "reset HEAD" *git unstage* to easily undo *git add*/*git rm*. git config --global alias.mkremotebranch "push origin -u" *git mkremotebranch* will copy your current local branch to GitHub. git config --global alias.rmremotebranch "push origin --delete" *git rmremotebranch <name>* will delete said branch from GitHub. git config --global color.ui auto This adds color to the git output, which some may find useful for eg. *git status*.

On the 1st of September the transitional period starts, and we will begin changing scripts to point to the new code locations. The transition may be delayed somewhat if the phenix release encounters any delays. I will send you further updates closer to the day. You can find and comment on the transition plan at https://docs.google.com/document/d/1bTiGbnvw8lNUrBLbRmhEdnd22McDqBbiKxcpygaZ... Best wishes -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Hi Markus, Billy, Nigel and I worked through the plan you originally sent out (linked at the top of this email thread). We've added a couple questions as comments in the document (expounded on below). Also, could you make the document writeable by Billy, Nigel and I? We want to assign some of the tasks listed. Here are the questions: - How is the repository at https://github.com/cctbx/cctbx being synced with the SVN version? (This is just my own curiosity :) - We think the non-cctbx repositories there that are copies (clipper, boost, annlib, etc.) of the bundles distributed alongside cctbx need to be removed from the git cctbx organization as they are duplications of what we are currently distributing. You say they are there so you can do tagging with the dials releases, but if Billy has updated the versions at LBL, then wouldn't these versions hosted by git already be out of date? Which would affect the dials stable releases? - We need to think about how websites such as dials.lbl.gov or cctbx.sourceforge.net will be affected by moving to git. Probably it's just updating links or moving html files? Thanks, -Aaron On Tue, Aug 23, 2016 at 6:32 AM, wrote:

Hi Marcin.

Currently I would say there is no plan. I do not have the history of the projects, so I could not do anything about that. I also don't use any of those repositories, so can't really comment on their status or future. I created the repositories only so I can tag versions for the DIALS releases, otherwise someone changing the downloadable archive would affect stable releases.

-Markus

Dr Markus Gerstel MBCS Postdoctoral Research Associate Tel: +44 1235 778698

Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE

-----Original Message----- From: [email protected] [mailto:cctbxbb-bounces@ phenix-online.org] On Behalf Of Marcin Wojdyr Sent: 23 August 2016 14:26 To: cctbx mailing list Subject: Re: [cctbxbb] Git status update

Hi Markus, what's the plan for the repositories that are currently imported as "copy of LBL archive" (without history). I guess the plan is to preserve the history - but will they be merged or added as submodule/subtree of the main cctbx repository?

Marcin _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Hi Markus, Indeed, dials.lbl.gov is built off of the same sources. I was more concerned about cctbx.sourceforge.net. We can look into that on our end. As for the repositories, there are licensing issues with the way you have things set up and we would like to examine if the code duplication that is occurring can be avoided. We (Billy and I) will make up a list of dependencies and their licenses and how we think they should be hosted. Should we do a brief skype call over the next couple days? Thanks, -Aaron On Tue, Nov 1, 2016 at 2:04 AM, wrote:

Hi Aaron,

If you could send me your gmail addresses and I’ll share the doc.

Regarding your questions:

1. Syncing the repositories is done using subgit and I run this currently 4 times a day.

2. Yes, the repositories are copies of what we are distributing. No, I do not think we should remove them, rather we should if at all possible extend them by adding their development history and actually point to them. Currently no changes can be made to any of those packages, and – from a distribution point of view, worse – nobody knows if and when those packages do change. What I therefore do for dials releases is a) download all the packages, b) unpack them into the respective repositories and c) see if any changes were made before I d) tag the repository for release. As you can see this has a massive time impact and is part of the reason why releases are currently more painful than they have to be. Therefore if the packages were recently updated this is not a problem since nobody currently refers to them outside the stable releases.

3. Isn’t dials.lbl.gov already running off the https://github.com/dials/dials.github.io repository? I am sure we can have something similar for the cctbx websites. We already generate http://cctbx.github.io/ but I don’t atm know where cctbx.sourceforge.net comes from.

-Markus

Dr Markus Gerstel MBCS

Postdoctoral Research Associate

Tel: +44 1235 778698

Diamond Light Source Ltd.

Diamond House

Harwell Science & Innovation Campus

Didcot

Oxfordshire

OX11 0DE

*From:* [email protected] [mailto:cctbxbb-bounces@ phenix-online.org] *On Behalf Of *Aaron Brewster *Sent:* 28 October 2016 23:48

*To:* cctbx mailing list *Subject:* Re: [cctbxbb] Git status update

Hi Markus,

Billy, Nigel and I worked through the plan you originally sent out (linked at the top of this email thread). We've added a couple questions as comments in the document (expounded on below). Also, could you make the document writeable by Billy, Nigel and I? We want to assign some of the tasks listed.

Here are the questions:

- How is the repository at https://github.com/cctbx/cctbx being synced with the SVN version? (This is just my own curiosity :) - We think the non-cctbx repositories there that are copies (clipper, boost, annlib, etc.) of the bundles distributed alongside cctbx need to be removed from the git cctbx organization as they are duplications of what we are currently distributing. You say they are there so you can do tagging with the dials releases, but if Billy has updated the versions at LBL, then wouldn't these versions hosted by git already be out of date? Which would affect the dials stable releases? - We need to think about how websites such as dials.lbl.gov or cctbx.sourceforge.net will be affected by moving to git. Probably it's just updating links or moving html files?

Thanks,

-Aaron

On Tue, Aug 23, 2016 at 6:32 AM, wrote:

Hi Marcin.

Currently I would say there is no plan. I do not have the history of the projects, so I could not do anything about that. I also don't use any of those repositories, so can't really comment on their status or future. I created the repositories only so I can tag versions for the DIALS releases, otherwise someone changing the downloadable archive would affect stable releases.

-Markus

Dr Markus Gerstel MBCS Postdoctoral Research Associate Tel: +44 1235 778698

Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE

-----Original Message----- From: [email protected] [mailto:cctbxbb-bounces@ phenix-online.org] On Behalf Of Marcin Wojdyr Sent: 23 August 2016 14:26 To: cctbx mailing list Subject: Re: [cctbxbb] Git status update

Hi Markus, what's the plan for the repositories that are currently imported as "copy of LBL archive" (without history). I guess the plan is to preserve the history - but will they be merged or added as submodule/subtree of the main cctbx repository?

Marcin _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Hi folks, we are good to go on our side. The final dates are Nov 8th for this first stage of the transition (updating bootstrap.py, buildbot and Jenkins to all point at git) and Nov 22nd (locking the SVN tree on sourceforge). Markus, now is a great time to send your helpful tip emails that you did for DIALS :) Thanks all! -Aaron On Wed, Nov 2, 2016 at 1:18 AM, wrote:

Hi Aaron,

Yes. Please give a shout when ready.

-Markus

(Side note: If it’s the same to you I would prefer a hangout call because I can’t remember my Skype account details. Otherwise I’ll dig them up or get a new account.)

Dr Markus Gerstel MBCS

Postdoctoral Research Associate

Tel: +44 1235 778698

Diamond Light Source Ltd.

Diamond House

Harwell Science & Innovation Campus

Didcot

Oxfordshire

OX11 0DE

*From:* [email protected] [mailto:cctbxbb-bounces@ phenix-online.org] *On Behalf Of *Aaron Brewster *Sent:* 01 November 2016 17:51

*To:* cctbx mailing list *Subject:* Re: [cctbxbb] Git status update

Hi Markus,

Indeed, dials.lbl.gov is built off of the same sources. I was more concerned about cctbx.sourceforge.net. We can look into that on our end.

As for the repositories, there are licensing issues with the way you have things set up and we would like to examine if the code duplication that is occurring can be avoided. We (Billy and I) will make up a list of dependencies and their licenses and how we think they should be hosted. Should we do a brief skype call over the next couple days?

Thanks,

-Aaron

On Tue, Nov 1, 2016 at 2:04 AM, wrote:

Hi Aaron,

If you could send me your gmail addresses and I’ll share the doc.

Regarding your questions:

1. Syncing the repositories is done using subgit and I run this currently 4 times a day.

2. Yes, the repositories are copies of what we are distributing. No, I do not think we should remove them, rather we should if at all possible extend them by adding their development history and actually point to them. Currently no changes can be made to any of those packages, and – from a distribution point of view, worse – nobody knows if and when those packages do change. What I therefore do for dials releases is a) download all the packages, b) unpack them into the respective repositories and c) see if any changes were made before I d) tag the repository for release. As you can see this has a massive time impact and is part of the reason why releases are currently more painful than they have to be. Therefore if the packages were recently updated this is not a problem since nobody currently refers to them outside the stable releases.

3. Isn’t dials.lbl.gov already running off the https://github.com/dials/dials.github.io repository? I am sure we can have something similar for the cctbx websites. We already generate http://cctbx.github.io/ but I don’t atm know where cctbx.sourceforge.net comes from.

-Markus

Dr Markus Gerstel MBCS

Postdoctoral Research Associate

Tel: +44 1235 778698

Diamond Light Source Ltd.

Diamond House

Harwell Science & Innovation Campus

Didcot

Oxfordshire

OX11 0DE

*From:* [email protected] [mailto:cctbxbb-bounces@ phenix-online.org] *On Behalf Of *Aaron Brewster *Sent:* 28 October 2016 23:48

*To:* cctbx mailing list *Subject:* Re: [cctbxbb] Git status update

Hi Markus,

Billy, Nigel and I worked through the plan you originally sent out (linked at the top of this email thread). We've added a couple questions as comments in the document (expounded on below). Also, could you make the document writeable by Billy, Nigel and I? We want to assign some of the tasks listed.

Here are the questions:

- How is the repository at https://github.com/cctbx/cctbx being synced with the SVN version? (This is just my own curiosity :) - We think the non-cctbx repositories there that are copies (clipper, boost, annlib, etc.) of the bundles distributed alongside cctbx need to be removed from the git cctbx organization as they are duplications of what we are currently distributing. You say they are there so you can do tagging with the dials releases, but if Billy has updated the versions at LBL, then wouldn't these versions hosted by git already be out of date? Which would affect the dials stable releases? - We need to think about how websites such as dials.lbl.gov or cctbx.sourceforge.net will be affected by moving to git. Probably it's just updating links or moving html files?

Thanks,

-Aaron

On Tue, Aug 23, 2016 at 6:32 AM, wrote:

Hi Marcin.

Currently I would say there is no plan. I do not have the history of the projects, so I could not do anything about that. I also don't use any of those repositories, so can't really comment on their status or future. I created the repositories only so I can tag versions for the DIALS releases, otherwise someone changing the downloadable archive would affect stable releases.

-Markus

Dr Markus Gerstel MBCS Postdoctoral Research Associate Tel: +44 1235 778698

Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE

-----Original Message----- From: [email protected] [mailto:cctbxbb-bounces@ phenix-online.org] On Behalf Of Marcin Wojdyr Sent: 23 August 2016 14:26 To: cctbx mailing list Subject: Re: [cctbxbb] Git status update

Hi Markus, what's the plan for the repositories that are currently imported as "copy of LBL archive" (without history). I guess the plan is to preserve the history - but will they be merged or added as submodule/subtree of the main cctbx repository?

Marcin _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

--

This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Hi Marcin, here's a more thorough answer than you were probably asking for, but Billy and I just finished a review of the cctbx dependencies so here's the result. I'm speaking here only of cctbx dependencies, defined exactly by what happens when you run libtbx/auto_build/bootstrap.py --builder=cctbx. First off, the copies of boost, scons, annlib, etc in the cctbx git organization were moved into the git DIALS organization, since they are only being used internally by the DIALS team to manage releases (thanks Markus). As for the cctbx dependencies, they can be divided into four categories: 1. SVN repositories hosted by LBL (clipper*, tntbx, annlib_adaptbx, ccp4io, ccp4io_adaptbx and gui_resources) 2. Repositories hosted elsewhere (cbflib) 3. Hot bundles we maintain at http://cci.lbl.gov/hot (scons, boost and annlib) 4. 'Base' dependencies that get downloaded by install_base_packages.py and are hosted at http://cci.lbl.gov/cctbx_dependencies/ After the cctbx to git transition is complete, Billy and I will look into moving the svn repositories in 1) into the cctbx git organization using subgit, if that seems reasonable all around. Before we do, we want to look into their licensing a bit, probably adding some extremely open and permissive license file to annlib_adaptbx, ccp4io_adaptbx which currently don't have licenses. 2) is fine how it is. 3) and 4) it would be nice to download from the original websites (boost.org, scons.org, etc.), but that's longer term and probably needs to be case by case. Hosting our own version in case of the original shutting down seems important to me. *clipper isn't distributed with the cctbx basic installation. In other words, bootstrap.py --builder=cctbx does check out clipper, but the bundled package built by buildbot isn't including it. -Aaron On Fri, Nov 4, 2016 at 9:48 AM, Marcin Wojdyr wrote:

out of curiosity: what was decided regarding the repositories hosted at LBL (ccp4io, clipper, boost, annlib, etc.)? _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Hi Aaron, first thank you very much to you all for the work you have put into this transition to git. Highly appreciated.

On 4 Nov 2016, at 18:19, Aaron Brewster wrote:

1. SVN repositories hosted by LBL (clipper*, tntbx, annlib_adaptbx, ccp4io, ccp4io_adaptbx and gui_resources) ... After the cctbx to git transition is complete, Billy and I will look into moving the svn repositories in 1) into the cctbx git organization using subgit,

Do you mean using “git subtree”? or “git submodule”? I would have a weak preference for using “subtree”. Best wishes, Luc

I was proposing moving (at least a subset) of the LBL-hosted SVN repositories into git as repositories in the git cctbx organization. I wasn't suggesting using subtrees or submodules as I gather (after some googling) that these are repositories within repositories.

Thanks for the clarification: I misunderstood indeed!