thomas.touhey.uk

Adapting Sphinx for Cahute

Cahute is a C library and set of command-line utilities for interacting with CASIO calculators I've been developing for more than a year now (0.1 was released in March 2024). With it comes an extensive documentation, edited and published as static HTML using Sphinx, which comes from the Python ecosystem.

However, in order to make it correspond with my usage of it for Cahute, I've had to make a few modifications to it, represented as extensions and overrides. These modifications are described in the following sections.

Most of the extension stuff is added in a custom, local extension named cahute_extensions; see cahute_extensions.py to view the source code. Some changes require the addition of a custom CSS stylesheet, through the html_css_files = ["custom2.css"] instruction in conf.py; see custom2.css to view its content.

Theme and overrides

This project, as many others of mine before, use a Sphinx theme named Furo by Pradyun Gedam. It's a theme I've learned to use and love during my time at Powens, thanks to Florian Strzelecki, and it's a theme used by many projects in the Python ecosystem, including pip and setuptools.

For most of my use cases with it, fortunately, Furo has a good set of customization options:

For other use cases, I had to dig a little deeper; see the following subsections for more information.

Integrating Mermaid

Cahute's documentation includes diagrams, such as sequence diagrams for communication protocol flows (see CAS40 flows for an example). These diagrams use Mermaid through the sphinxcontrib-mermaid Sphinx extension, which adds the .. mermaid:: reStructuredText directive I can use.

However, Mermaid doesn't play nice with Furo's dark/light theme switch by default, as no Mermaid theme plays well with both, so I needed to add a Javascript snippet to detect dark/light theme (including dynamically, once the page / diagrams have loaded), and update the theme used by Mermaid.

The full code can be found in conf.py, set as the mermaid_init_js attribute that is used by sphinxcontrib-mermaid to replace the default template. A preview of the result is the following:

/_assets/Q-5hQXKK4m8Y/cahute-mermaid.gif

Example usage of theme switch with a Mermaid diagram.

Connecting to Furo's theme changes

Finding out what theme (dark or light) Furo is actually in is a bit tricky, but not impossible; I can compute what the current value of --color-code-foreground is on <body>. Here's a Javascript function to do so:

function isDarkMode() {
    const color = (
        getComputedStyle(document.body)
        .getPropertyValue("--color-code-foreground")
    );

    if (color == "#d0d0d0")
        return true;

    return false;
}

In order to find out when the theme changes, I can use a MutationObserver on the data-theme attribute of <body>, set by Furo. Here's a Javascript snippet to do so:

const observer = new MutationObserver(function(mutations) {
    mutations.forEach(function(mutation) {
        if (
            mutation.type != "attributes"
            || mutation.attributeName != "data-theme"
        )
            return;

        // ...
    });
});

(function (window) {
    // ...
    observer.observe(document.body, {attributes: true});
})(window);

Initializing and updating Mermaid diagrams

This part is trickier, as it requires understanding how Mermaid initializes diagrams in the code.

When initialized on a page, Mermaid looks for HTML <div> elements with the mermaid class. It takes the content as text, assumed to be Mermaid source code, and replaces the content with the rendered diagram. It then adds the data-processed HTML attribute.

This, by default, means that the source code disappears in such cases, and we cannot rerender diagrams by default. So our first step is to do the same exploration as Mermaid before it is run, and save the contents into an attribute of ours, which we call data-original-code.

After saving the initial code, we can run Mermaid as intended. If we need to recompute diagrams with a new theme, we must explore all <div class="mermaid"> nodes again, restore the code into the div's content from its data-original-code attribute, remove the data-processed attribute, and re-run Mermaid.

A snippet do to all of that, using the previously defined isDarkMode to select the theme we want, is the following:

import mermaid from "https://cdn.jsdelivr.net/npm/mermaid/dist/mermaid.esm.min.mjs";

function initializeMermaid(isStart) {
    mermaid.initialize({
        startOnLoad: isStart,
        theme: isDarkMode() ? "dark" : "base",
        darkMode: isDarkMode(),
        securityLevel: "antiscript"
    });
}

function recomputeMermaid() {
    const nodes = document.querySelectorAll(".mermaid");
    nodes.forEach(node => {
        /* Restore the original code before reprocessing. */
        node.innerHTML = node.getAttribute("data-original-code");

        /* Remove the attribute saying data is processed; it is not! */
        if (node.hasAttribute("data-processed"))
            node.removeAttribute("data-processed");
    });

    initializeMermaid(false);
    mermaid.run({nodes: nodes, querySelector: ".mermaid"});
}

Rendering TOC tree sections side-by-side on the index

By default, Sphinx renders table-of-contents (TOC) trees as a simple list, which takes up a lot of space to display less content:

/_assets/wJRL1v-T-7Vm/cahute-ugly-toctree-onecol.png

An example of a section in the Cahute documentation index, with the default TOC tree rendering.

We want to render such TOC trees on the index as two columns, if we have enough space (width).

First, we need a way to target the index page only using CSS. Fortunately, docutils places all of its rendered context within a <section> tag with an id set to a normalized version of what's in the document title. Here, since our section title is Cahute |version| (where |version| is a substitution replaced by the actual version on rendering), the generated identifier for the <section> tag is cahute-version, so we can target #cahute-version.

Next, in order to make two columns, we can use flexboxes. This will result in the following code, placed in custom2.css:

#cahute-version .toctree-wrapper > ul {
    display: flex;
    list-style-type: none;
    flex-wrap: wrap;
    padding-left: 0;
    gap: .5em 1em;
}

#cahute-version .toctree-wrapper > ul > li {
    display: inline-block;
    flex: 1 0 20em;
}

Which renders as the following:

/_assets/Yyibysxe9KKh/cahute-cool-toctree-twocol.png

The same example of a section in the Cahute documentation index, with the new two-column TOC tree rendering.

docutils overrides

Under the hood, Sphinx uses docutils to read reStructuredText formatted documents and render them in many formats, including HTML. I've already written about customizing docutils in Using and adapting docutils, but since this project is not centered around it, my goal is to write to the point code only.

Overriding the HTML translator

Some of the modifications below will require me to implement HTML rendering for some custom elements, or update HTML rendering for some elements. In order to do this, I write my class inheriting from sphinx.writers.html.HTMLTranslator, and set the translator using sphinx.application.Sphinx.set_translator.

The HTML translator is written as a classic docutils node visitor, using visit_<node class name> and depart_<node class name> methods. Visit methods can also raise the following exceptions:

  • SkipNode: the children won't be visited, the departure method won't be called;

  • SkipChildren: the children won't be visited, the departure method will be called;

  • SkipDeparture: the children will be visited, the departure method won't be called.

Adding a cover image to the page title

I commissioned Massena to make an illustration for Cahute, to both include in promotional material for Cahute (forum posts and social media only for now) as well as include as a cover image for page titles in the documentation, to give it a little more panache.

In order to add the illustration as a cover image for the page titles, I need to override the HTML rendering code for docutils' title node, i.e. override visit_title.

The HTML output for a page title, identified by checking we are not in a special context and that our section level is 1 (i.e. if we are not within a container node, even indirectly), is replaced from <h1>...</h1> to something like the following:

<div class="title-container">
    <div class="title-contrast"><hr /></div>
    <h1>...</h1>
</div>

Fixing section titles in the sidebar

Cahute's documentation uses Diátaxis, and I consider this to be a contract between the reader and the writer; therefore, the structure must appear as clear as day, in every place it can.

Until recently, all top-most sections appeared in the sidebar unseparated, their nature (guide, topic or reference) only appearing in their title. However, it is possible in Sphinx to add titles to sections, by adding a :title: property on the corresponding .. toctree:: directive on the index page.

However, when doing this, the title also appeared above the TOC tree, which was repetitive with the title above and undersirable:

/_assets/H6ih5a2CKICl/cahute-ugly-toctree-title.png

Title appearing above the TOC tree by default, highlighted.

In order to remove this, in our custom HTML translator, we need to add another block to visit_title to identify such cases and raise a SkipNode in this case. The correct identification algorithm I've found is the following:

  • If our parent is an sphinx.addnodes.compact_paragraph instance with its toctree attribute set. This corresponds to two cases: the TOC tree in the sidebar, and the TOC tree within the index page; however, we only want to identify the second one.

  • If our grandparent is a docutils.nodes.compound with the toctree-wrapper class, which allows us to only select the second case.

This results in the following block:

if (
    isinstance(node.parent, addnodes.compact_paragraph)
    and node.parent.get("toctree")
    and isinstance(node.parent.parent, nodes.compound)
    and "toctree-wrapper" in node.parent.parent.get("classes", ())
):
    raise nodes.SkipNode()

Adding a feature list presentation

In order to highlight the Cahute features, I wanted to make an element to present them as this:

/_assets/YUPcxM5Cc7kk/cahute-feature-list.png

A list of features listed for Cahute 0.6.

Out of a list resembling the following:

.. feature-list::

    * - |feat-transfer|
      - File transfer between storages
      - With ``p7``, transfer files from and to storages on
        fx-9860G compatible calculators, over USB and serial links!
    * - |feat-program|
      - Program backup
      - With ``CaS``, extract programs from all CASIO calculators since
        1991, over USB and serial links!

.. |feat-transfer| image:: feat-transfer.svg
.. |feat-program| image:: feat-program.svg

In order to do this, I did the following:

  • Create the feature_list, feature, feature_icons, feature_title and feature_detail nodes. The idea was to obtain this in the document obtained from parsing the original document:

    <feature_list>
      <feature>
        <feature_icons>
          <image uri="..."></image>
        </feature_icons>
        <feature_title>
          ...
        </feature_title>
        <feature_detail>
          ...
        </feature_detail>
      </feature>
      <feature>
        ....
      </feature>
    </feature_list>
    
  • Create the FeatureListDirective which obtains this tree from a two-level list passed as the content, and register it using sphinx.application.Sphinx.add_directive;

  • Add the HTML rendering functions in the custom HTML translator, e.g. visit_feature_list, and so on.

Adding a system list presentation

In order to highlight the systems supported by Cahute, as well as present the guides associated with them directly, I wanted to make an element to present them as this:

/_assets/6EPVI4-cu5la/cahute-system-list.png

A list of systems and related guides listed for Cahute 0.6.

Out of a list resembling the following:

.. system-list::

    * - |system-arch|
      - :ref:`feature-topic-system-arch`
      - :ref:`Install from the AUR <install-guide-linux-aur>`
      - :ref:`Install using GiteaPC <install-guide-linux-giteapc>`
      - :ref:`Build natively <build-guide-linux-sh>`
    * - |system-apple|
      - :ref:`feature-topic-system-macos`
      - :ref:`Install using Homebrew <install-guide-macos-homebrew>`
      - :ref:`Build natively <build-guide-macos-sh>`

.. |system-arch| image:: guides/install/arch.svg
.. |system-apple| image:: guides/install/apple.svg

In order to do this, I did the following:

  • Create the system_list, system, system_icon and system_detail nodes. The idea was to obtain this in the document obtained from parsing the original document:

    <system_list>
      <system>
        <system_icon>
          <image uri="..."></image>
        </system_icon>
        <system_detail>
          ...
          <bullet_list>
            <list_item>
              <reference>...</reference>
              <reference>...</reference>
              ...
            </list_item>
          </bullet_list>
        </system_detail>
      </system>
      <system>
        ....
      </system>
    </system_list>
    
  • Create the SystemListDirective which obtains this tree from a two-level list passed as the content, and register it using sphinx.application.Sphinx.add_directive;

  • Add the HTML rendering functions in the custom HTML translator, e.g. visit_system_list, and so on.

Domain-specific directives

Some parts of the documentation are repetitive, because they are a reference with mostly similar examples and structures, with only a few bits changing. While in code, there are principles such as DRY to avoid repeating information and make templates and whatnot, this does not apply to documentation, where in order to get information across, you must adapt to the human psyche and repeat the information in multiple places so that it gets picked up along the way.

In such cases where the presentation is mostly the same, it is sometimes best, in order to avoid typos and make it more maintainable, to make a directive that automates most of the presentation.

Currently in Cahute, there is only one such example.

Protocol 7.00 command directive

In the Known Protocol 7.00 commands by CASIO and Known Protocol 7.00 command extensions in fxRemote pages, you can find a reference of Protocol 7.00 commands, all having the same format with different argument usage. Every command has the following presentation:

  • A title with the command code in ASCII-HEX, and name in quotes;

  • A reference of the D1 to D6 parameters, with description and format;

  • A description of the command;

  • An example of the command, including a checksum (!).

In the related guide, everything but the title is automated using the seven-command directive. For example:

.. seven-command::
    :code: 02
    :d1: Baud rate in decimal format.
    :d1-example: 19200
    :d2: Parity, among "ODD", "EVEN" or "NONE".
    :d2-example: EVEN
    :d3: Stop bits, "1" or "2".
    :d3-example: 2

    If connected to the passive device using a serial link, this command
    signifies a request to communicate using different serial parameters.

    .. note::

        The serial speeds are limited to the following speeds (or baud
        rates): 300, 600, 1200, 2400, 4800, 9600 (*by default*), 19200,
        38400, 57600 and 115200 bauds.

    See :ref:`protocol-topic-seven-update-serial-params` for more
    information.

This example will generate a presentation such as the following:

/_assets/ZMObQfKjIXNo/cahute-seven-command-02.png

Description of command 02 for Protocol 7.00, generated by the seven-command directive (except for the title).

In order to do this, I've created the SevenCommandDirective and registered it using sphinx.application.Sphinx.add_directive.

Redirections

It rarely happens that I move around some elements in the documentation, but still want to keep old links working. In such a case, I needed a way to provide the information to nginx on the host server all of the redirections, so that it can apply them and return HTTP 301 responses in the correct cases.

I ended up adding a redirects property definition in conf.py, resembling the following:

redirects = {
    "/install-guides/": "/guides/install/",
    "/contribution-guides.html": "/guides/contribution.html",
    ...
}

From here, I connect to Sphinx's build-finished hook to create a _redirects.nginx file in the build directory, with rewrite instructions of two types:

On the host server, in the corresponding server block, I add the following:

include /path/to/www/_*.nginx;

Conclusions

Even though it does not have the best documentation or the easiest extension framework, Sphinx is a great framework for writing and organizing docs, even for non-Python projects, and these few tweaks have been done over a year of using the documentation, based on user feedback and my own tastes.

While all modifications described in a single go may seem quite a handful, they are not, and are actually mostly optional and based on my own views of a practical and beautiful documentation. Hopefully it does not refrain you from using Sphinx, or even write docs for your own project; and it gives you ideas on how you can improve upon existing documentation you maintain.

Cahute is a project with many challenges, and the completeness and navigability of its documentation is one of them, especially since it describes undocumented proprietary communication protocols and file formats that were hard to find exhaustive documentation for before. My hope is that these tweaks help bringing out the existing content in the documentation, in order for information regarding these not to be lost!