Am Donnerstag 18 April 2013 20:42:39 schrieb James Stroud:

On Apr 18, 2013, at 11:16 AM, Richard Gildea wrote:

...

Hi Jan,

I have attempted to build the sphinx documentation using dox.sphinx/build_docs.sh, and this appears to have run successfully and produced some html which shows the table of contents etc. However, I can't seem to find the documentation for any of the significant modules, e.g. cctbx/xray/structure.py. When I navigate to the link for the structure module within the xray package (dox.sphinx/_build/html/cctbx/cctbx.xray.html#structure-module), I see nothing other than a list of all the modules within the xray package. Am I missing something here?

From my understanding of boost-python, the substantive part of the major modules are all written in C++ wherein the python interface is essentially implicit. I call the capability to handle this implicit relationship as "boost-aware". It's a fairly tall order for sphinx to be boost-aware (i.e. impossibly tall at the moment), because sphinx does not "know" the rules by which boost-python maps python types to C++.

Here, we have a many-to-one mapping that (from my perspective) can only be created manually (i.e. reading code, etc.). For an example of the situation, consider the following C++ signature taken from cctbx.miller.binner:

scitbx::af::shared<int> extract_integers(iotbx::mtz::object {lvalue},scitbx::af::const_ref,char const*)

[ See http://cctbxwiki.bravais.net/Iotbx_mtz_ext.object#extract_integers.28.29 and expand the boost python docstring.]

For this particular declaration, the return type is a flex integer array and the two parameters (after self) are (1) a flex integer array, and (2) a string. The boost generated docstring provides the following help:

extract_integers( (object)arg1, (int)mtz_reflection_indices, (str)column_label) -> int

This help string suggests that the first argument (after self) is an integer, as is the return value. But of course these objects are not integers, they are flex integer arrays. After writing/being in the process of writing a parser for these signatures, I can now recognize when "int" means integer flex array or just "int". But is such knowledge expected of every programmer who might use the cctbx python API?

Even more problematic than flex arrays are complex data structures that the boost help strings refer to as "(object)". Note how the help string represents the iotbx_mtz_ext.object above. As the first argument (self), the python type for "(object)" is obvious. However, in subsequent arguments and in return values, the type of object must be determined manually.

Here is another example:

void __init__(_object*,cctbx::uctbx::unit_cell,scitbx::af::const_ref)

Now, guess which is/are "(object)" and check your answer here: http://cctbxwiki.bravais.net/Cctbx.miller.binning#documentation:_init

In short, Sphinx simply can't decipher the C++ docstrings and/or boost-generated help strings at this time and so will miss the meat of the cctbx library.

I can envision only two routes to intelligible auto-documentation of the API:

1. Every method would have python documentation embedded into the C++ source code (that's not really auto-documentation, though). 2. A specialty tool for boost-python cctbx documentation would have to be made.

I believe the 2nd approach will take the less work, although it is not trivial. I'm writing a tool for my own needs (see http://cctbxwiki.bravais.net/Class_Index for its products). I create type mappings as I need them, so coverage will likely be slow-going and asymptotic.

Perhaps you could also add that special functionality to sphinx as we're already using some kind of special formatting for boost functions (see dox.sphinx/conf.py)

If such boost-python-cctbx auto-documentation is a project that the rest of the cctbx community wants to participate in, I can make what I've made so far available via the SVN sources. Note that at the moment it spits out [the vaguely respectable] wiki text of mediawiki and not [the heinously limited] markdown text of the sourceforge wiki. I could make it spit out [the practically magnificent] reStructuredText recognized by Sphinx, depending on what form is deemed "official" documentation.

If it's very hard to get the functionality directly into shinx I think using rST as a output format would be good, especially as there are parsers from rST into almost any other format. Also rST could be included directly into sphinx so we could then use sphinx to write a html and LaTeX documentation. Cheers, Jan