These are basic parameters for FFT based Fcalc calculation. algorithm = fft or direct summation wing_cutoff - is how far away from atomic center you sample density around atom; u_base (or b_base) - is additional smearing B-factor if input atom has too small B so it falls between the grid nodes; quality_factor - determines accuracy of sampled density. For s review, see: Acta Cryst. (2004). A60, 19-32 On a fast calculation of structure factors at a subatomic resolution P. V. Afonine and A. Urzhumtsev and references therein. Pavel On 12/7/12 12:35 PM, Richard Gildea wrote:

Hi Graeme,

I agree that the documentation of much of the cctbx code is somewhat lacking, although I am not sure what is the best solution (especially for the huge volumes of code that is already written). Perhaps some combination of people adding documentation when asking or answering questions on here about specifics would go some way to helping future users of the code.

As to your specific question:

Setting cos_sin_table=True uses interpolation of values from a pre-calculated lookup table for trigonometric function calls in the structure factor calculations in preference to calling the system libraries. Ralf makes some comment on the speed (and variation in speed) of the trigonometric functions in system math libraries in his FABLE paper: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3448510/

However, without digging into the source, I am not sure of the meaning of some of the other keywords to that function (e.g. wing_cutoff), which makes it hard for me to write comprehensive documentation for this class.

Cheers,

Richard

On 7 December 2012 01:59, mailto:[email protected]> wrote:

Hi Folks,

Been thinking some more about the old chestnut of documentation. The lack thereof is a well understood issue, and to be honest one which is not going to change any time soon as no one has the time to maintain a manual -- this is fine and well understood.

As a kind of "patch" for this I was wondering if it would be possible to have a forum where someone could ask for e.g. documentation for a currently undocumented function, which could then be added to the subversion in the code itself. For example:

cctbx_project/cctbx/xray/structure.py

...

def structure_factors(self, anomalous_flag=None, d_min=None,

algorithm=None,

cos_sin_table=False,

quality_factor=None,

u_base=None,

b_base=None,

wing_cutoff=None):

what is cos_sin_table for? with things like this there is the attitude that it is safe to ignore, however if we're writing code to guide people's experiments it would be good to be sure of this. Some simple documentation explaining this would be very useful.

I'm definitely not suggesting (i) that anyone writes a manual or (ii) that all undocumented functions are suddenly documented, only that there is a mechanism set up that someone could ask about this and that someone who knows the answer could commit the changes to cctbx and then reply with the diffs -- perhaps the way that stackoverflow works, or an email, or something really lightweight. Even this mailing list could be used I guess?

I would be happy to help with this, though the above example is not one where I know the answer!

Thoughts?

Best wishes,

Graeme

Dr. Graeme Winter

Senior Software Scientist

Diamond Light Source

+44 1235 778091 tel:%2B44%201235%20778091 (work)

+44 7786 662784 tel:%2B44%207786%20662784 (work mobile)

--

This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

_______________________________________________ cctbxbb mailing list [email protected] mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

To be honest I would say simply adding it to the code would work better as it's easier to maintain and makes it available to libtbx.help. Once you have a wiki you have something extra to do when you make changes to code, to see if the wiki needs updating. That said, a *separate* wiki with more of a how-to feel could be useful, however this has been mooted several times and not happened, I think because the activation energy is not trivial. The main resource - volunteers with useful knowledge and spare time - is in rather short supply. Just MHO, Cheerio, Graeme From: [email protected] [mailto:[email protected]] On Behalf Of Claudia Millán Nebot Sent: 10 December 2012 08:31 To: cctbx mailing list Subject: Re: [cctbxbb] Documentation .... again What about some wiki format? :) Claudia Millán ([email protected]mailto:[email protected]) Institut de Biologia Molecular de Barcelona (IBMB-CSIC) Barcelona, Spain 2012/12/7 Pavel Afonine mailto:[email protected]> These are basic parameters for FFT based Fcalc calculation. algorithm = fft or direct summation wing_cutoff - is how far away from atomic center you sample density around atom; u_base (or b_base) - is additional smearing B-factor if input atom has too small B so it falls between the grid nodes; quality_factor - determines accuracy of sampled density. For s review, see: Acta Cryst. (2004). A60, 19-32 On a fast calculation of structure factors at a subatomic resolution P. V. Afonine and A. Urzhumtsev and references therein. Pavel On 12/7/12 12:35 PM, Richard Gildea wrote: Hi Graeme, I agree that the documentation of much of the cctbx code is somewhat lacking, although I am not sure what is the best solution (especially for the huge volumes of code that is already written). Perhaps some combination of people adding documentation when asking or answering questions on here about specifics would go some way to helping future users of the code. As to your specific question: Setting cos_sin_table=True uses interpolation of values from a pre-calculated lookup table for trigonometric function calls in the structure factor calculations in preference to calling the system libraries. Ralf makes some comment on the speed (and variation in speed) of the trigonometric functions in system math libraries in his FABLE paper: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3448510/ However, without digging into the source, I am not sure of the meaning of some of the other keywords to that function (e.g. wing_cutoff), which makes it hard for me to write comprehensive documentation for this class. Cheers, Richard On 7 December 2012 01:59, mailto:[email protected]> wrote: Hi Folks, Been thinking some more about the old chestnut of documentation. The lack thereof is a well understood issue, and to be honest one which is not going to change any time soon as no one has the time to maintain a manual - this is fine and well understood. As a kind of "patch" for this I was wondering if it would be possible to have a forum where someone could ask for e.g. documentation for a currently undocumented function, which could then be added to the subversion in the code itself. For example: cctbx_project/cctbx/xray/structure.py ... def structure_factors(self, anomalous_flag=None, d_min=None, algorithm=None, cos_sin_table=False, quality_factor=None, u_base=None, b_base=None, wing_cutoff=None): what is cos_sin_table for? with things like this there is the attitude that it is safe to ignore, however if we're writing code to guide people's experiments it would be good to be sure of this. Some simple documentation explaining this would be very useful. I'm definitely not suggesting (i) that anyone writes a manual or (ii) that all undocumented functions are suddenly documented, only that there is a mechanism set up that someone could ask about this and that someone who knows the answer could commit the changes to cctbx and then reply with the diffs - perhaps the way that stackoverflow works, or an email, or something really lightweight. Even this mailing list could be used I guess? I would be happy to help with this, though the above example is not one where I know the answer! Thoughts? Best wishes, Graeme Dr. Graeme Winter Senior Software Scientist Diamond Light Source +44 1235 778091tel:%2B44%201235%20778091 (work) +44 7786 662784tel:%2B44%207786%20662784 (work mobile) -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

On Mon, Dec 10, 2012 at 12:35 AM, wrote:

That said, a *separate* wiki with more of a how-to feel could be useful, however this has been mooted several times and not happened, I think because the activation energy is not trivial. The main resource – volunteers with useful knowledge and spare time – is in rather short supply.

I've started working on a (lengthy) newsletter article that would be a tutorial overview for beginners (with the assumption that they already know Python). At my current rate it will take years to complete, however. I do think it would be helpful if every time there are questions like this - including cases where we have to figure out the answer by trial and error - we add appropriate docstrings to the code. I've been doing this recently (and attempting to document my own code, which is a new experience), and it doesn't actually take very much time. -Nat

On Mon, Dec 10, 2012 at 2:18 PM, Pavel Afonine wrote:

depending on your vision of this, perhaps such tutorial should be a collective effort between those who would like to volunteer to participate. Also, that may help to cut down your time for preparing it.

That was indeed the plan, but it's not like anyone else around here has more free time. I'm happy to share the Google Doc of what I've written so far if anyone would like to contribute. (The plan was actually to write it in at least two parts: first part would cover fundamentals of scitbx and cctbx, part two would cover applications to macromolecules, i.e. iotbx.pdb and mmtbx.) -Nat

Hi Folks, I started on something like these for new starters here at Diamond using cctbx - I wrote a couple of very short tutorials focussed around simple tasks. I would post the link however it seems that source forge is having a bad day. Having these aim to answer a specific use case was helpful, as it did not take so long to write it. I guess we could collate a "cookbook" of these sort of things? That would scale well. I wrote these as latex documents with the code associated with any data - should be trivial to make these into a test case as well. The document describes most of the key bits of the calculation but not necessarily every bit. Happy to post these as examples. Anyhow happy to help with the effort. A comment I would make is it is also helpful to have someone other than the author "debug" the tutorials... Cheerio, Graeme On 10 Dec 2012, at 22:52, Pavel Afonine wrote:

I've been having this plan too for quite some time, and yes, I've never had time to do anything (and most likely wouldn't in near future). However doing it collectively may cut on amount of time and effort, thus making it practical in the end.

I would plan it first. To start, I would say we need:

1) find out who wants to participate; 2) write a framework and agree on it. I see it as a list of bullet points outlining the subjects that we are going to cover. 3) then we see who wants (and can) to cover what and do the first step: everybody fills in his/her contribution. 4) then we collectively review the whole thing, and do a few iterations of that until we are satisfied with what we have.

Additional thoughts:

I know it may make things slightly more complex, but I suggest that we make sure that every example provided in the tutorial is exercised as part of our routine tests. This will ensure that the effort we are going to invest will not be obsoleted shortly after it's complete. Also, this would imply the document may change if a cctbx functionality described in it changed (so it's not a static paper). I did this for command line version of phenix.refine several years ago, and it seems to work well (though I'm not saying that we should do it the same way for this tutorial!).

I have zero experience with Google Docs, so these couple of questions are rather out of my ignorance. Does it allow to 1) insert formulas, 2) put pieces of code that can be extracted automatically for exercising ?

Pavel

On 12/10/12 2:22 PM, Nathaniel Echols wrote:

...

On Mon, Dec 10, 2012 at 2:18 PM, Pavel Afonine wrote:

...

depending on your vision of this, perhaps such tutorial should be a collective effort between those who would like to volunteer to participate. Also, that may help to cut down your time for preparing it. That was indeed the plan, but it's not like anyone else around here has more free time. I'm happy to share the Google Doc of what I've written so far if anyone would like to contribute. (The plan was actually to write it in at least two parts: first part would cover fundamentals of scitbx and cctbx, part two would cover applications to macromolecules, i.e. iotbx.pdb and mmtbx.)

-Nat _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

Hi, I would be happy to contribute in any way that I can, though that might be more reading, testing and commenting on documentation rather than producing new pieces for 'core' cctbx tasks. I use Google Docs/Drive for my own notes, but find it inadequate for any form of polished documentation. The equation editor started off well, being a simple text input field that rendered LaTeX markup, but they scrapped that for a graphical Word style editor that was an enormous step backwards in my opinion. It has improved slightly since then, accepting some LaTeX-like shortcut keys like '_' and '^' for sub- and superscript, but still lacks certain symbols, and the inter-operability with LaTeX that it once had. Also, Google has occasionally changed their Docs engine, requiring me to 'upgrade' my documents to the latest version, which has invariably mangled some of the formatting. For these reasons, much as I like Google Docs for convenience, I would strongly recommend avoiding it for the purpose in mind. Some form of simple and concise markup language seems most appropriate. I use LaTeX, but arguably it fails on both of these requirements. Also, the mess of packages on CTAN, many of which are mutually incompatible, make portability and concatenation of documents a real pain. Some of the obscure documentation for these packages really makes advanced LaTeX use feel like alchemy. I briefly explored reStructuredText a while ago, but concluded that it was great for code blocks, but didn't have rich enough math rendering capabilities. Now, revisiting the documentation I wonder if I was wrong? I would like to be able to do vectors and matrices, equation arrays with control over which lines are numbered, and labels to refer to equations in the text. If this is covered easily by reStructuredText then I nominate it as a serious contender. Cheers -- David On 11 December 2012 09:33, Luc Bourhis wrote:

Hi Pavel,

I agree 100% with everything you said.

I have zero experience with Google Docs, so these couple of questions are rather out of my ignorance. Does it allow to 1) insert formulas, 2) put pieces of code that can be extracted automatically for exercising ?

I would recommend using ReStructureText as we did for the Comp. Comm. news letter and put the file on our Sourceforge repo of course.

Best wishes,

Luc

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

+1 for latex Certainly for DIALS much mathematics will be needed in the documentation. That said, since RST to Latex is a doable transformation there is no need to "choose one." I also like the way Latex deals with images... Cheerio, Graeme -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Nathaniel Echols Sent: 11 December 2012 16:23 To: cctbx mailing list Subject: Re: [cctbxbb] Documentation .... again On Tue, Dec 11, 2012 at 3:02 AM, David Waterman wrote:

I use Google Docs/Drive for my own notes, but find it inadequate for any form of polished documentation. The equation editor started off well, being a simple text input field that rendered LaTeX markup, but they scrapped that for a graphical Word style editor that was an enormous step backwards in my opinion. It has improved slightly since then, accepting some LaTeX-like shortcut keys like '_' and '^' for sub- and superscript, but still lacks certain symbols, and the inter-operability with LaTeX that it once had. Also, Google has occasionally changed their Docs engine, requiring me to 'upgrade' my documents to the latest version, which has invariably mangled some of the formatting. For these reasons, much as I like Google Docs for convenience, I would strongly recommend avoiding it for the purpose in mind.

Point taken. Since everyone who might contribute has SVN write access anyway, we don't really need the added convenience.

I briefly explored reStructuredText a while ago, but concluded that it was great for code blocks, but didn't have rich enough math rendering capabilities. Now, revisiting the documentation I wonder if I was wrong? I would like to be able to do vectors and matrices, equation arrays with control over which lines are numbered, and labels to refer to equations in the text. If this is covered easily by reStructuredText then I nominate it as a serious contender.

Well, it does support embedded LaTeX equations: http://docutils.sourceforge.net/docs/ref/rst/directives.html#math But I suspect it is much less flexible than what you're asking for. So I guess we need to decide whether we want/need to include equations in the tutorial, or just code - if the latter, then RST will suffice, otherwise I have no objection to using LaTeX. (Actually, since we already have a BibTeX bibliography started, that's a very appealing option.) -Nat _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

On Tue, Dec 11, 2012 at 8:44 AM, Pavel Afonine wrote:

I think someone needs to take the lead here. Nat, would you mind to take it? You said you've started doing something anyway...

Okay, I've checked in what I had written so far, converted to LaTeX, in dox/tutorial/. Anyone who wants to expand it, help yourself - don't forget to add yourself to the author list. (Keep in mind that this will probably end up being distributed as a newsletter article at some point, possibly in abridged form.) The initial document is only going to cover scitbx and cctbx for now (plus libtbx.phil, probably), but we can add other chapters as needed.

I've never used LaTeX, and I afraid I don't have extra time and energy to afford learning curve now. I would say a plain text with basic formatting capabilities is the simplest and most robust option. Thus I would vote for RST.

Sorry, I think LaTeX is going to be much more flexible. It's really not very difficult, at least compared to crystallography, and everyone else seems to be comfortable with it. -Nat

Hi Pavel, I think it is fair to say, if you want to add just text / code to the example you could just add it in and someone else could add the \begin{verbatim} etc. needed to make it nicely typeset. Getting the material in is the valuable bit, typesetting less important. I would even volunteer to fix that up. Cheerio, Graeme -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Pavel Afonine Sent: 11 December 2012 22:52 To: [email protected] Subject: Re: [cctbxbb] Documentation .... again On 12/11/12 2:39 PM, Nathaniel Echols wrote:

On Tue, Dec 11, 2012 at 8:44 AM, Pavel Afonine wrote:

...

I've never used LaTeX, and I afraid I don't have extra time and energy to afford learning curve now. I would say a plain text with basic formatting capabilities is the simplest and most robust option. Thus I would vote for RST. Sorry, I think LaTeX is going to be much more flexible. It's really not very difficult, at least compared to crystallography, and everyone else seems to be comfortable with it.

-Nat

no problems. I think I just postpone my participation in this effort. Like I said I can't afford any avoidable learning curve here. Pavel _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

Agreed. LaTeX is reasonably straightforward for simple use and not much more verbose than rST, that is, once someone has kindly provided a template with all the preamble boilerplate and some useful examples. Looks like Nat has just done that ;-) -- David On 12 December 2012 11:35, Luc Bourhis wrote:

Hi Graeme,

I think it is fair to say, if you want to add just text / code to the example you could just add it in and someone else could add the \begin{verbatim} etc. needed to make it nicely typeset.

Definitively. Especially considering that reST can be translated to LaTeX using Sphinx at least, therefore removing most of the drudgery. I would be more than happy to give a hand anyway. So, please Pavel, feel free to submit contributions following reST if that makes your life easier. Eventually, comparing the same text of yours in reST and in LaTeX, you may feel like you could do it in LaTeX in the first place!!

Best wishes,

Luc

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

An added note about reST: I recommend we use Sphinx as the rendering engine. Nicer looking rendering, table of content across several files, etc. Luc