Using Sphinx clever with GitHub Pages

Thanks to Github Pages, it has never been easier than today to create a website for your documentation. Just create a new git branch gh-pages or a folder docs , drag in an index.html file, and the website is done. Easy, right? Yes definitely, but also efficient? Not really. A website can be set up quickly, bootstrap on it and you’re done, but it gets more and more confusing as the project grows. That’s why there are special tools to make the documentation of your programs efficient. One of them, especially for Python, is Sphinx. But how do you use Sphinx with GitHub Pages?

Sphinx

With Sphinx you can easily create documentation. The individual pages are written in reStructuredText (reST). This is a markup language. Alternatively there is a plugin called recommonmark, which also includes the better known markdown natively.
Sphinx itself is a Python module and can be installed via pip. Also most plugins can be installed via pip..

Example

The start of any Sphinx documentation is the index.rst file. Here is an example of a simple documentation with three subpages:

H1 Headline
***********
An example
.. toctree::
:caption: Contents
module
about/changelog
about/License.md
H2 Headline
===========
.. include:: ../README.md
Generierte Indizes
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

.. _module:
*********************
Module
*********************
:mod:`transfer`
===============
.. automodule:: transfer
:members:
:undoc-members:

.. _changelog:
Changelog
*********
=
0.4
---
- Added better documentation
- Added public function descriptions
0.3
---
- Added (CLI) option to change size, contrast and sharpness
- Added size to ouput name
- Resize goal image automatically to size of base
- Updated Sphinx docs

# License
BSD 3-Clause License
Copyright (c) 2020, Arne Rantzen
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Compiled

Introduction

Headlines

Commands

A reST command is like a class. You create the object with .. , then the name of the class, followed by :: and finally the arguments for the constructor. Special arguments are called with :[name]: .

Table of contents

A table of contents is created by the toctree. (toc = Table of Contents). Here, the indentation, as known from Python, is extremely important, otherwise compilation errors will occur. With recommonmark you can simply name both reST and Markdown files and they will be linked automatically. With Markdown, however, you have to name the extension as well. The toctree service as navigation menu to navigate through the documentation.

Embedding

External files can be easily integrated into a document. In the example above, the GitHub typical Readme.md , located in a folder below, was included in the document using the include command. This is also compiled automatically. This makes it easy to split larger files.

Autodoc

With the plugin autodoc you can automatically put the comments and functions of your modules into a nice overview:

Link

As above with the toctree it is also possible to create links out of the text. You can use :ref:[name] to link to a file, or to a special marker. The marker is set with ... _[name]: . Another special command from autodoc is mod:[name] . It links to the documentation of the automodule [Name].

Working with Sphinx

To work with Sphinx, you have to install it first. To do so, enter pip install sphinx autodoc recommonmark in your terminal and you will get two very handy plugins with it.
When Sphinx is installed, you can navigate to any directory and run sphinx-quickstart. This starts a CLI wizard that builds a basic framework with Makefile. I recommend to separate source and build (answer first question with “y”).

If you now have a finished installation, you can simply call the Makefile with make html and get a compiled website in your build folder. But how do you use Sphinx with GitHub Pages?

Sphinx with GitHub Pages

GitHub Pages is a feature of GitHub that allows you to host a static website for your project. This is done by accessing files in a repository hosted on GitHub.

The problem is that GitHub is not easily compatible with Sphinx. A typical Sphinx installation looks like this:

├───build
│   ├───doctrees
│   └───html
│       ├───_sources
│       └───_static
└───source
├───_static
└───_template

With GitHub you only have two options to host your website: either an index.html in the /docs/ folder, or at the root level. Sphinx creates the index.html in the /build/html/ folder.
It’s of course possible to just copy it to the /docs/ folder, but here’s where the next problem comes in: GitHub uses Jekyll to provide which completely ignores all folders starting with an underscore, like _sources and _static.

From Sphinx it is not possible to rename these folders, but there is a possibility to disable Jekyll in GitHub. To do so, you need to put an empty file named .nojekyll in the root folder of your website.

So we have found a solution: We create the .nojekyll file in the /build/html folder and move the contents of that folder to /docs/.
Possible, but not as elegant. Better to use git magic and hooks!

Git Magic

Our destination is the following folder structure:

├───gh-pages
│   └───docs
└───master
└───src
├───docs
│   ├───build
│   │   ├───doctrees
│   │   └───html
│   │       ├───_sources
│   │       └───_static
│   └───source
│       ├───_static
│       └───_template
└───modules

We need two branches for this: A master and a gh-pages. On gh-pages only the compiled files for our website will come. The master branch is only for the source files. As it should be: A strict separation between source and build.

Create Repository

The following section uses Windows 10 and the Powershell console to build a working GitHub page with Sphinx that follows the above scheme.

Create master repository

First we create a new repository in a new project folder. I call the project once: SphinxPages. Therefore we create a new folder master .

In this our git is initialized and a first file is created. This file is now added to our git and commited.
Once a GitHub repository has been created, it can then be linked to our local git. After that our commit will be uploaded.

Create Branch Repository

Now go back to the root folder and clone the repository again. This is done with the name gh-pages.You switch to this location and create the new gh-pages branch. Then a link will be created so that every time you enter this folder, the branch will be changed. Finally, the .git/index file will be deleted and the branch will be cleaned up.

Create Sphinx folder

The initial structure is ready, now we can take care of the source.
We create the path master/src/docs, change there and start the Sphinx Wizard.
In the first decision we choose “yes”.

Once you have gone through the rest of the steps – it doesn’t matter for this tutorial what you choose here – you have the finished Sphinx structure. Finally the Makefile or make.bat has to be adjusted, so that the html folder in the master is automatically copied into the gp-pages folder.

Automation

For Windows the make.bat file is changed. First we create a new variable DOCS which points to our gh-pages. Since we are in the folder master/src/docs, we have to go 3 levels down.

Then a new query is created to see if “github” was given as command. If this has been done, we want to refer to our new location.

[...]
set BUILDDIR=build
set DOCS=../../../gh-pages
if "%1" == "" goto help
if "%1" == "github" goto github 
[...]
:help
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
goto end
:github
rmdir /S /Q "%BUILDDIR%"
%SPHINXBUILD% -M html %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
copy /Y "%BUILDDIR%/html" "%DOCS%"
echo. 2>"%DOCS%/.nojekyll"
goto end
:end
popd

This new location is then implemented.
This first deletes the build order so that no old artifacts remain, then starts the normal Sphinx build process with the argument html, and finally copies the result to the order gh-pages.

Finally, add a .nojekyll file to gh-pages to make GitHub render the page correctly.

Concluding remark

As soon as the architecture is finished, you can easily create the documentation with make github. The documentation is now automatically copied into the gh-pages folder. So you only have to upload both branches.

Theoretically you can also do this with a git hook, which must look something like this:

echo Update documentation
cd /src/docs && make github
cd ../../../gh-pages
while true; do
read -p "Do you wish to push the updated documenation?" yn
case $yn in
[Yy]* ) 
git add *;    
echo Add a commit message to your updated documentation
read message
git commit -m "$message"
git push
break;;
[Nn]* ) exit;;
* ) echo "Please answer yes or no.";;
esac
done
exit 0

Tl;dr

If you don’t want to understand the background and only want a quick solution, this is the right place. On the last page I provide a script, which does the work and also the changed Makefile. There is also a GitHub template repository, which you can clone as well.

GitHub Template

On the page SphinxPages there is also a ready-made GitHub template which you can easily clone and set up in a few steps.

It has a slightly different structure than in this guide. So the gh-pages folder is integrated into the root repository.

Scripts

Just execute one of the following scripts to get the desired structure:

#!/bin/bash
echo What should the new repository be called?
read name
master="./$name/master"
echo Create path
mkdir -p $master && cd $master
echo Create first entry
git init && touch Readme.md && git add Readme.md && git commit -m "Initial commit"
echo What is the GitHub address?
read github
echo Upload repository to GitHub
git remote add origin $github && git push -u origin master
echo Clone Repository
cd .. && git clone $github gh-pages && cd gh-pages
echo Create gh-branches and automatic change
git branch gh-pages && git symbolic-ref HEAD refs/heads/gh-pages
echo Clear branch
rm .git/index && git clean -fdx
echo Initializing Sphinx
cd .. && mkdir -p ./master/src/docs && cd ./master/src/docs && sphinx-quickstart
exit 0

@echo off
set /p name="What should the new repository be called? "
set master=".\%name%\master"
echo Create path
mkdir %master% && cd %master%
echo Create first entry
git init && type nul >> Readme.md && git add Readme.md && git commit -m "Initial commit"
set /p github="What is the GitHub address? "
echo Upload repository to GitHub
git remote add origin %github% && git push -u origin master
echo Clone Repository
cd .. && git clone %github% gh-pages && cd gh-pages
echo Create gh-branches and automatic change
git branch gh-pages && git symbolic-ref HEAD refs/heads/gh-pages
echo Clear branch
del /q ".git\index" && git clean -fdx
echo Initializing Sphinx
cd .. && mkdir .\master\src\docs && cd .\master\src\docs && sphinx-quickstart
exit 0

Then replace the Makefile and make.bat in the folder master/src/docs/ with the following two files.

# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXOPTS    ?=
SPHINXBUILD   ?= sphinx-build
SOURCEDIR     = source
BUILDDIR      = build
DOCS          = ../../../gh-pages
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
github:
@rm -rf "$(BUILDDIR)"
@make html
@mv "$(BUILDDIR)/html" "$(DOCS)"
@touch "$(DOCS)/.nojekyll"
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

@ECHO OFF
pushd %~dp0
REM Command file for Sphinx documentation
if "%SPHINXBUILD%" == "" (
set SPHINXBUILD=sphinx-build
)
set SOURCEDIR=source
set BUILDDIR=build
set DOCS=..\..\..\gh-pages
if "%1" == "" goto help
if "%1" == "github" goto github
%SPHINXBUILD% >NUL 2>NUL
if errorlevel 9009 (
echo.
echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
echo.installed, then set the SPHINXBUILD environment variable to point
echo.to the full path of the 'sphinx-build' executable. Alternatively you
echo.may add the Sphinx directory to PATH.
echo.
echo.If you don't have Sphinx installed, grab it from
echo.http://sphinx-doc.org/
exit /b 1
)
%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
goto end
:help
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
goto end
:github
rmdir /S /Q "%BUILDDIR%"
%SPHINXBUILD% -M html %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
copy /Y "%BUILDDIR%\html" "%DOCS%"
echo. 2>"%DOCS%\.nojekyll"
goto end
:end
popd

---

I hope this guide on how to use Sphinx with GitHub Pages helped. Please leave me a comment here for your feedback and further questions.