Merging Multiple Package Scribblings

Splitting projects into distinct packages is usually good - especially in Racket where that gives you local Scribble documentation search for free. But how to internally publish a website composed of multiple documentation sets of many packages?

The benefits of splitting a larger projects into distinct, reusable packages are varying. In addition to the aforementioned documentation benefits it allows the programmers to think about the packages and their modules in larger context than the project in question which usually leads to better code with cleaner interfaces. Not all code of a larger project is suitable for splitting into a separate package, however it is only advisable to consider it whenever some parts get too big.

Usually rendering the documentation where the packages are installed is more than enough, yet in our case we also need to publish the documentation internally for non-Racket developers. Publishing the whole catalogue is tempting, but probably a bit too much. Therefore we have decided to create a single multi-page documentation composite that will cover the internal project scribblings as well as scribblings of all the Racket packages that are developed as a part of this project. Whenever some feature is merged into the master branch, we need to ensure that the CI/CD pipeline updates the documentation available to everyone.

In our project the root of the git repository includes the pkgs subdirectory which contains all the packages we are creating. The structure is rather simple:

$ ls pkgs
our-package another-one

Another benefit of Racket packages is the ability to query their properties using their info.rkt file. It can be required like any other module and its #%info-lookup binding can provide the information needed:

(define lookup (dynamic-require "pkgs/our-package/info.rkt" '#%info-lookup))
(define scribblings (lookup 'scribblings))

It almost sounds too easy to be true. As the info.rkt file resides directly in the root directory of particular package and can be loaded using dynamic-require easily -- static require can also be used, but dynamic one is better as we are going to see later.

It has however two problems. One is that it returns a S-expression like ((scribblings/our-package.scrbl) ()) and another one is that if the key is not present, the lookup function raises an exception. Addresing both is fortunately very easy as can be seen from the next code snippet:

(define scribblings
  (with-handlers ((exn? (lambda _ #f)))
    (caar (lookup 'scribblings))))

Secondly we should turn the list of packages in our pkgs subdirectory into a list of their main scribbling files. There are some assumptions here to be made and it is probably apparent how the following code was written incrementally - but it works surprisingly well and it is pretty easy to understand.

(define *pkgs* "pkgs")
(define (get-pkgs-scribblings)
  ;; This is expected to be run from the root directory!
  (define subdirs (sort (directory-list *pkgs*) string<=?))
  (define infos
    (for/list ((dirname (in-list subdirs)))
      (define dir (build-path *pkgs* dirname))
      (define infofname (build-path dir "info.rkt"))
      (if (file-exists? infofname)
          (list dirname infofname)
          #f)))
  (define real-infos (filter identity infos))
  (define scribblings
    (for/list ((infofname (in-list real-infos)))
      (define lookup (dynamic-require (build-path ".." (cadr infofname)) '#%info-lookup))
      (define scribblings
        (with-handlers ((exn? (lambda _ #f)))
          (path->string
           (build-path ".." *pkgs* (car infofname) (caar (lookup 'scribblings))))))
      scribblings))
  (define real-scribblings (filter identity scribblings))
  real-scribblings)

Thirdly we need to ensure that all the scribble modules found are included as individual sections in the main document. As Scribble is built on top of Racket it allows us to use the same macro system as Racket possesses. The algorithm for retrieving the relative paths of all the relevant modules defined above can be shifted to phase 1 by wrapping it in begin-for-syntax and then the get-pkgs-scribblings procedure can be used directly in any define-syntax expression. That being said, materializing the scribblings in another scribble file becomes a task which requires only a few lines of code.

@(define-syntax (materialize-scribblings stx)
  (syntax-parse stx
    ((_)
     #:with (scribbling ...)
            (datum->syntax stx (get-pkgs-scribblings))
     #'(begin
         (include-section scribbling) ...))))

@(materialize-scribblings)

This code probably needs a bit more explaining so let us dive into it. The syntax-parse allows to bind syntax pattern variables like with-syntax does. Therefore we bind scribbling ... as a list of syntaxes that contain literal strings with paths to the modules. And then when this template variable is used in the template (which has the same shape), it expands to a begin form with multiple include-section expressions. As we evaluate the materialize-scribblings in the top level, the begin basically splices its contents into the surrounding environment and all the include-sections become top-level forms.

A bit tricky but in the end pretty straightforward. Only one question remains and that is how to easily render all of this into multiple HTML pages. That is, how to do the same task as raco setup does when installing packages and building scribblings with multi-page option in the info.rkt file. That is where the render-multi-mixin comes handy:

#lang racket/base

(require (prefix-in html: scribble/html-render)
         scribble/render
         setup/xref
         racket/class
         racket/cmdline
         "render-docs.scrbl")

(define dest-dir
  (command-line
   #:args
   (dest-dir) dest-dir))

(define search:render-mixin
  (lambda (%)
    (html:render-multi-mixin
      (class (html:render-mixin %)
        (init [search-box? #f])
        (super-new [search-box? search-box?])))))

(render (list doc)
        (list dest-dir)
        #:render-mixin search:render-mixin
        #:redirect-main "https://docs.racket-lang.org"
        #:xrefs (list (load-collections-xref)))

Although this is assumed to be run from the project root directory, we need at least some ability to configure where to put the resulting HTML pages ad that is exactly what the command-line with no options but one positional formal argument does. With this single expression, we get command-line options help for free - which might be useful in the future:

$ racket tools/render-docs.rkt -h
usage: render-docs.rkt [ <option> ... ] <dest-dir>

<option> is one of

  --help, -h
     Show this help
  --
     Do not treat any remaining argument as a switch (at this level)

 Multiple single-letter switches can be combined after
 one `-`. For example, `-h-` is the same as `-h --`.

Not only we added the render-multi-mixin but also we added the render-mixin which allows us to enable search-box? in the future when we implement some search over the identifiers covered by this conglomerate of packages.

Hope you liked our little venture into the depths of Racket documentation tooling and remember to tune in next time for more!