Automatic Python documentation with Sphinx autodoc and ReadTheDocs

Generating Python documentation for packages/modules can be quite time consuming, but there’s a way to generate it automatically from docstrings. This post is mostly a summary of the fantastic guide by Sam Nicholls found here, but with one important addition (see the section on mocking). We’ll be using the following:

Sphinx – Python package for generating documentation
Sphinx autodoc – Sphinx extension to generate documentation from docstrings
ReadTheDocs – build and host documentation online

Before you start, make sure you’ve written docstrings for your modules/functions/methods. This is the most time consuming part, but you should be commenting and documenting your code anyway! As an example, you might have a module with docstrings that look like this, and after completing this process it will automatically turn into Python documentation that looks like this.

Sphinx Autodoc

First, install the Sphinx package:

pip install Sphinx

Next, create a docs directory at the root of your project directory, cd into that and run sphinx-quickstart

cd /path/to/project
mkdir docs
cd docs/
sphinx-quickstart

This begins the configuration process. The defaults are generally fine, but the only thing you need to do is enable the autodoc extension when asked.

Assuming all of your docstrings have been written, you need to create the stubs for your project in your docs directory (these need to be recreated if new modules are added):

cd docs/
sphinx-apidoc -o source/ ../

Eventually, we’ll be using ReadTheDocs (RTD) to build and host the Python documentation. In order for RTD to find your package files we need to make a change to the Sphinx config. After the quickstart process above, Sphinx should have created a conf.py file in your docs directory. Near the top of that file, you need to add a path to your package contents (or uncomment the lines already in the file):

import os
import sys
sys.path.insert(0, os.path.abspath('../'))

In here we can also change the theme of our documentation page:

html_theme = 'sphinx_rtd_theme'

And add extensions:

extensions = ['sphinx.ext.autodoc',
    'sphinx.ext.coverage',
    'sphinx.ext.napoleon',
    'sphinx.ext.viewcode']

napoleon_google_docstring = False
napoleon_use_param = False
napoleon_use_ivar = True

Now we can try to build the documentation locally. Sphinx includes a make file that we can use for this:

cd docs/
make html

You may need to install the mock and sphinx_rtd_theme modules for a local build to work:

pip install mock
pip install sphinx_rtd_theme

Inside docs/_build/html you should see the automatically generated Python documentation as HTML. Confirm that it correctly picks up your docstrings. We won’t actually be using these files directly as we want RTD to handle the build process for us, but this is just a quick way to make sure everything is working.

ReadTheDocs

RTD imports your project from a GitHub repo and builds the Python documentation directly from your package contents and your docs/ contents. Optionally, you can also automatically trigger a build whenever you commit and push to the repo.

The first thing you need to do is push your package to a Github repo, but we don’t want to include any of our locally built documentation. Do that by adding docs/_build/, docs/_static/, and docs/_templates/ to the repo’s .gitignore file. However, make sure you do commit docs/sources.

Then, create a ReadTheDocs account and import your repo from the list. This should trigger an initial build, which you can see in the Builds tab.

If you want RTD to automatically build every time you push to your GitHub repo, go to the Settings of your repo, click on Integration & Services, and add ReadTheDocs from the list of available services.

At this point you should see that RTD has built the documentation for you. You should be good to go, however, you may run into an issue where RTD doesn’t correctly build the Python module index (i.e. py-modindex.html is missing and gives a 404 error when you try to view the page)…

Mocking

There are a number of reasons why the module index doesn’t get created. The primary reason that I’ve come across is that my package depends on other packages that require C libraries (e.g. numpy). You can read more about this in the RTD FAQ.

The solution is to mock those imported packages and modules.

Identify any dependencies that rely on C libraries, then open conf.py and add the following:

from mock import Mock as MagicMock

class Mock(MagicMock):
    @classmethod
    def __getattr__(cls, name):
        return MagicMock()

MOCK_MODULES = ['numpy', 'scipy', 'scipy.linalg', 'scipy.signal']
sys.modules.update((mod_name, Mock()) for mod_name in MOCK_MODULES)

Make sure to add your dependencies to MOCK_MODULES, and if your package is importing modules from another package, add that specific module to the list as well. For example, if my package specifically requires scipy.signal, then that module should be added to MOCK_MODULES in addition to the scipy package.

Now just commit and push this change to your GitHub repo, fire off a build on RTD and you should see a correctly generated module index.

If the module index still doesn’t work, it could be because RTD is using the wrong Python version to generate the docs. I had this problem with a Python 3.x package I created and the module index wouldn’t show because RTD was generating the virtual environment using Python 2. To change this, go to the RTD Dashboard, Admin > Advanced Settings, and change the Python interpreter near the bottom.

Customization

At this point, the documentation should be ready and we can begin to customize it in various ways. The most obvious place to start is by editing the files in /docs/source to adjust the layout and content of our documentation pages.

Sphinx build options

We can add some more options to conf.py. For a full list of options, see the Sphinx documentation

A few examples:

add_module_names = False so functions aren’t prepended with the name of the package/module
add_function_parentheses = True to ensure that parentheses are added to the end of all function names

Setting the version number

The version number of the documentation can be set “dynamically”. This is especially useful if you need to update the package version number in multiple places (e.g. setup.py for pypi). The easiest way I’ve found to do this is to create a file in the top-level directory of the package called VERSION, then write a function to read from that file in conf.py to extract the version number:

def get_version():
    version_file = open('../VERSION')
    return version_file.read().strip()

version = get_version()
release = version

Include the README file

If we convert out repo README.md to a .rst file (README.rst), we can include it into our main documentation page (index.rst).

Optionally, we can set a marker in README.rst to only include content after a certain point.

Start by adding .. inclusion-marker-main-readme to README.rst. Then in index.rst, add:

.. include:: ../README.rst
   :start-after: inclusion-marker-main-readme

PDF options

If you’ve told RTD to generate a PDF for each build it can sometimes create additional blank pages between chapters. There are 2 ways around this. First, instead of using manual as the document class in conf.py > latex_documents, we can switch to howto to remove chapters altogether. Alternatively, we can set the latex page printing options by changing latex_elements:

latex_elements = {
  'extraclassoptions': ',openany,oneside'
}

Overall, I find that Sphinx autodoc and RTD are a huge time saver. I only need to worry about writing my docstrings and the rest is handled for me automatically.