[cctbxbb] building sphinx documentation

[Jan Marten Simons]

Am Donnerstag 18 April 2013 20:42:39 schrieb James Stroud:
> On Apr 18, 2013, at 11:16 AM, Richard Gildea wrote:
> > Hi Jan,
> > 
> > I have attempted to build the sphinx documentation using
> > dox.sphinx/build_docs.sh, and this appears to have run successfully and
> > produced some html which shows the table of contents etc. However, I
> > can't seem to find the documentation for any of the significant modules,
> > e.g. cctbx/xray/structure.py. When I navigate to the link for the
> > structure module within the xray package
> > (dox.sphinx/_build/html/cctbx/cctbx.xray.html#structure-module), I see
> > nothing other than a list of all the modules within the xray package. Am
> > I missing something here?
> 
> From my understanding of boost-python, the substantive part of the major
> modules are all written in C++ wherein the python interface is essentially
> implicit. I call the capability to handle this implicit relationship as
> "boost-aware".  It's a fairly tall order for sphinx to be boost-aware
> (i.e. impossibly tall at the moment), because sphinx does not "know" the
> rules by which boost-python maps python types to C++.
> 
> Here, we have a many-to-one mapping that (from my perspective) can only be
> created manually (i.e. reading code, etc.). For an example of the
> situation, consider the following C++ signature taken from
> cctbx.miller.binner:
> 
> scitbx::af::shared<int> extract_integers(iotbx::mtz::object
> {lvalue},scitbx::af::const_ref<int, scitbx::af::trivial_accessor>,char
> const*)
> 
> [ See
> http://cctbxwiki.bravais.net/Iotbx_mtz_ext.object#extract_integers.28.29
> and expand the boost python docstring.]
> 
> For this particular declaration, the return type is a flex integer array
> and the two parameters (after self) are (1) a flex integer array, and (2)
> a string. The boost generated docstring provides the following help:
> 
> extract_integers( (object)arg1, (int)mtz_reflection_indices,
> (str)column_label) -> int
> 
> This help string suggests that the first argument (after self) is an
> integer, as is the return value. But of course these objects are not
> integers, they are flex integer arrays. After writing/being in the process
> of writing a parser for these signatures, I can now recognize when "int"
> means integer flex array or just "int". But is such knowledge expected of
> every programmer who might use the cctbx python API?
> 
> Even more problematic than flex arrays are complex data structures that the
> boost help strings refer to as "(object)". Note how the help string
> represents the iotbx_mtz_ext.object above. As the first argument (self),
> the python type for "(object)" is obvious. However, in subsequent
> arguments and in return values, the type of object must be determined
> manually.
> 
> Here is another example:
> 
> void
> __init__(_object*,cctbx::uctbx::unit_cell,scitbx::af::const_ref<double,
> scitbx::af::trivial_accessor>)
> 
> Now, guess which is/are "(object)" and check your answer here:
> http://cctbxwiki.bravais.net/Cctbx.miller.binning#documentation:_init
> 
> In short, Sphinx simply can't decipher the C++ docstrings and/or
> boost-generated help strings at this time and so will miss the meat of the
> cctbx library.
> 
> I can envision only two routes to intelligible auto-documentation of the
> API:
> 
> 1. Every method would have python documentation embedded into the C++
> source code (that's not really auto-documentation, though). 2. A specialty
> tool for boost-python cctbx documentation would have to be made.
> 
> I believe the 2nd approach will take the less work, although it is not
> trivial. I'm writing a tool for my own needs (see
> http://cctbxwiki.bravais.net/Class_Index for its products). I create type
> mappings as I need them, so coverage will likely be slow-going and
> asymptotic.

Perhaps you could also add that special functionality to sphinx as we're 
already using some kind of special formatting for boost functions (see 
dox.sphinx/conf.py)

> If such boost-python-cctbx auto-documentation is a project that the rest of
> the cctbx community wants to participate in, I can make what I've made so
> far available via the SVN sources. Note that at the moment it spits out
> [the vaguely respectable] wiki text of mediawiki and not [the heinously
> limited] markdown text of the sourceforge wiki. I could make it spit out
> [the practically magnificent] reStructuredText recognized by Sphinx,
> depending on what form is deemed "official" documentation.

If it's very hard to get the functionality directly into shinx I think using 
rST as a output format would be good, especially as there are parsers from rST 
into almost any other format. Also rST could be included directly into sphinx 
so we could then use sphinx to write a html and LaTeX documentation.

Cheers,
Jan