λ

PAX Manual

Table of Contents

[in package MGL-PAX with nicknames PAX]

λ

1 Introduction

What if documentation really lived in the code?

Docstrings are already there. If some narrative glued them together, we'd be able develop and explore the code along with the documentation due to their physical proximity. The main tool that PAX provides for this is defsection:

(defsection @foo-random-manual (:title "Foo Random manual")
  "Foo Random is a random number generator library."
  (foo-random-state class)
  (uniform-random function)
  (@foo-random-examples section))

Like this one, sections can have docstrings and references to definitions (e.g. (uniform-random function)). These docstrings and references are the glue. To support interactive development, PAX

See Emacs Setup.

Beyond interactive workflows, Generating Documentation from sections and all the referenced items in Markdown or HTML format is also implemented.

With the simplistic tools provided, one may emphasize the narrative as with Literate Programming, but documentation is generated from code, not vice versa, and there is no support for chunking.

Code is first, code must look pretty, documentation is code.

Docstrings

PAX automatically recognizes and marks up code with backticks and links code to their definitions. Take, for instance, SBCL's abort function, whose docstring is written in the usual style, uppercasing names of symbols:

(docstring #'abort)
=> "Transfer control to a restart named ABORT, signalling a CONTROL-ERROR if
   none exists."

Note how in the generated documentation, abort is set with a monospace font, while control-error is autolinked:

In the following transcript, the above output is rendered from the raw markdown:

(document #'abort :format :markdown)
.. - [function] **ABORT** *&OPTIONAL CONDITION*
..
..     Transfer control to a restart named `ABORT`, signalling a [`CONTROL-ERROR`][7c2c] if
..     none exists.
..
..   [7c2c]: http://www.lispworks.com/documentation/HyperSpec/Body/e_contro.htm "CONTROL-ERROR (MGL-PAX:CLHS CONDITION)"
..
A Complete Example

Here is an example of how it all works together:

(mgl-pax:define-package :foo-random
  (:documentation "This package provides various utilities for random.
  See FOO-RANDOM:@FOO-RANDOM-MANUAL.")
  (:use #:common-lisp #:mgl-pax))

(in-package :foo-random)

(defsection @foo-random-manual (:title "Foo Random manual")
  "FOO-RANDOM is a random number generator library inspired by CL:RANDOM.
  Functions such as UNIFORM-RANDOM use *FOO-STATE* and have a
  :RANDOM-STATE keyword arg."
  (foo-random-state class)
  (state (reader foo-random-state))
  "Hey we can also print states!"
  (print-object (method () (foo-random-state t)))
  (*foo-state* variable)
  (gaussian-random function)
  (uniform-random function)
  ;; This is a subsection.
  (@foo-random-examples section))

(defclass foo-random-state ()
  ((state :reader state)))

(defmethod print-object ((object foo-random-state) stream)
  (print-unreadable-object (object stream :type t)))

(defvar *foo-state* (make-instance 'foo-random-state)
  "Much like *RANDOM-STATE* but uses the FOO algorithm.")

(defun uniform-random (limit &key (random-state *foo-state*))
  "Return a random number from the between 0 and LIMIT (exclusive)
  uniform distribution."
  nil)

(defun gaussian-random (stddev &key (random-state *foo-state*))
  "Return a random number from a zero mean normal distribution with
  STDDEV."
  nil)

(defsection @foo-random-examples (:title "Examples")
  "Let's see the transcript of a real session of someone working
  with FOO:

  ```cl-transcript
  (values (princ :hello) (list 1 2))
  .. HELLO
  => :HELLO
  => (1 2)

  (make-instance 'foo-random-state)
  ==> #<FOO-RANDOM-STATE >
  ```")

Note how (variable *foo-state*) in the defsection form both exports *foo-state* and includes its documentation in @foo-random-manual. The symbols variable and function are just two instances of locatives, which are used in defsection to refer to definitions tied to symbols.

(document @foo-random-manual) generates fancy markdown or HTML output with automatic markup and autolinks uppercase words found in docstrings, numbers sections, and creates a table of contents.

One can even generate documentation for different but related libraries at the same time with the output going to different files but with cross-page links being automatically added for symbols mentioned in docstrings. In fact, this is what PAX World does. See Generating Documentation for some convenience functions to cover the most common cases.

The transcript in the code block tagged with cl-transcript is automatically checked for up-to-dateness when documentation is generated.

λ

2 Emacs Setup

Load src/mgl-pax.el in Emacs, and maybe set up some key bindings.

If you installed PAX with Quicklisp, the location of mgl-pax.el may change with updates, and you may want to copy the current version of mgl-pax.el to a stable location:

(mgl-pax:install-pax-elisp "~/quicklisp/")

Then, assuming the Elisp file is in the quicklisp directory, add something like this to your .emacs:

(load "~/quicklisp/mgl-pax.el")
(mgl-pax-hijack-slime-doc-keys)
(global-set-key (kbd "C-.") 'mgl-pax-document)
(global-set-key (kbd "s-x t") 'mgl-pax-transcribe-last-expression)
(global-set-key (kbd "s-x r") 'mgl-pax-retranscribe-region)

For Browsing Live Documentation, mgl-pax-browser-function can be customized in Elisp. To browse within Emacs, choose w3m-browse-url (see w3m), and make sure both the w3m binary and the w3m Emacs package are installed. On Debian, simply install the w3m-el package. With other browser functions, a HUNCHENTOOT web server is started.

See Navigating Sources in Emacs, Generating Documentation and Transcribing with Emacs for how to use the relevant features.

λ

3 Links and Systems

Here is the official repository and the HTML documentation for the latest version.

PAX is built on top of the DRef library (bundled in the same repository). See DRef's HTML documentation

λ

3.1 The mgl-pax ASDF System

λ

3.2 The mgl-pax/full ASDF System

λ

4 Background

As a user, I frequently run into documentation that's incomplete and out of date, so I tend to stay in the editor and explore the code by jumping around with SLIME's m-. (slime-edit-definition). As a library author, I spend a great deal of time polishing code but precious little writing documentation.

In fact, I rarely write anything more comprehensive than docstrings for exported stuff. Writing docstrings feels easier than writing a separate user manual, and they are always close at hand during development. The drawback of this style is that users of the library have to piece the big picture together themselves.

That's easy to solve, I thought, let's just put all the narrative that holds docstrings together in the code and be a bit like a Literate Programmer turned inside out. The original prototype, which did almost everything I wanted, was this:

(defmacro defsection (name docstring)
  `(defun ,name () ,docstring))

Armed with this defsection, I soon found myself organizing code following the flow of user-level documentation and relegated comments to implementation details entirely. However, some parts of defsection docstrings were just listings of all the functions, macros and variables related to the narrative, and this list was repeated in the defpackage form complete with little comments that were like section names. A clear violation of OAOO, one of them had to go, so defsection got a list of symbols to export.

That was great, but soon I found that the listing of symbols is ambiguous if, for example, a function, a compiler macro and a class were named by the same symbol. This did not concern exporting, of course, but it didn't help readability. Distractingly, on such symbols, M-. was popping up selection dialogs. There were two birds to kill, and the symbol got accompanied by a type, which was later generalized into the concept of locatives:

(defsection @introduction ()
  "A single line for one man ..."
  (foo class)
  (bar function))

After a bit of elisp hacking, M-. was smart enough to disambiguate based on the locative found in the vicinity of the symbol, and everything was good for a while.

Then, I realized that sections could refer to other sections if there were a section locative. Going down that path, I soon began to feel the urge to generate pretty documentation as all the necessary information was available in the defsection forms. The design constraint imposed on documentation generation was that following the typical style of upcasing symbols in docstrings, there should be no need to explicitly mark up links: if M-. works, then the documentation generator shall also be able figure out what's being referred to.

I settled on markdown as a reasonably non-intrusive format, and a few thousand lines later PAX was born. Since then, locatives and references were factored out into the DRef library to let PAX focus on M-. and documentation.

λ

5 Basics

Now let's examine the most important pieces.

λ

5.1 Parsing

When Navigating Sources in Emacs or Generating Documentation, references are parsed from the buffer content or docstrings, respectively. In either case, names are extracted from words and then turned into DRef names to form DRef references maybe with locatives found next to the word.

λ

6 PAX Locatives

To the Locative Types defined by DRef, PAX adds a few of its own.

λ

7 Navigating Sources in Emacs

Integration into SLIME's m-. (slime-edit-definition) allows one to visit the source-location of a definition(0 1).

The definition is either determined from the buffer content at point or is prompted. If prompted, then the format is <name> <locative>, where the locative may be omitted to recover stock Slime behaviour.

When determining the definition from the buffer contents, slime-symbol-at-point is parsed as a word, then candidate locatives are looked for before and after that word. Thus, if a locative is the previous or the next expression around the symbol of interest, then M-. will go straight to the definition which corresponds to the locative. If that fails, M-. will try to find the definitions in the normal way, which may involve popping up an xref buffer and letting the user interactively select one of possible definitions.

In the following examples, when the cursor is on one of the characters of foo or just after foo, pressing M-. will visit the definition of function foo:

function foo
foo function
(function foo)
(foo function)

In particular, references in a defsection form are in (name locative) format so M-. will work just fine there.

Just like vanilla M-., this works in comments and docstrings. In the next example, pressing M-. on foo will visit foo's default method:

;; See RESOLVE* (method () (dref)) for how this all works.

With a prefix argument (C-u M-.), one can enter a symbol plus a locative separated by whitespace to preselect one of the possibilities.

The M-. extensions can be enabled by loading src/mgl-pax.el. See Emacs Setup. In addition, the Elisp command mgl-pax-edit-parent-section visits the source location of the section containing the definition with point in it. See Browsing Live Documentation.

λ

7.1 The mgl-pax/navigate ASDF System

λ

8 Generating Documentation

λ

8.1 The document Function

λ

8.1.1 Documentables

λ

8.1.2 Return Values

If pages are nil, then document - like cl:format - returns a string (when stream is nil) else nil.

If pages, then a list of output designators are returned, one for each non-empty page (to which some output has been written), which are determined as follows.

If the default page given by the stream argument of document was written to, then its output designator is the first element of the returned list. The rest of the designators correspond to the non-empty pages in the pages argument of document in that order.

λ

8.1.3 Pages

The pages argument of document is to create multi-page documents by routing some of the generated output to files, strings or streams. pages is a list of page specification elements. A page spec is a property list with keys :objects, :output, :uri-fragment, :source-uri-fn, :header-fn and :footer-fn. objects is a list of objects (references are allowed but not required) whose documentation is to be sent to :output.

Documentation is initially sent to a default stream (the stream argument of document), but output is redirected if the thing being currently documented is the :object of a page-spec.

:output can be a number things:

Note that even if pages is specified, stream acts as a catch all, absorbing the generated documentation for references not claimed by any pages.

:header-fn, if not nil, is a function of a single stream argument, which is called just before the first write to the page. Since :format :html only generates HTML fragments, this makes it possible to print arbitrary headers, typically setting the title, CSS stylesheet, or charset.

:footer-fn is similar to :header-fn, but it's called after the last write to the page. For HTML, it typically just closes the body.

:uri-fragment is a string such as "doc/manual.html" that specifies where the page will be deployed on a webserver. It defines how links between pages will look. If it's not specified and :output refers to a file, then it defaults to the name of the file. If :uri-fragment is nil, then no links will be made to or from that page.

Finally, :source-uri-fn is a function of a single, reference argument. If it returns a value other than nil, then it must be a string representing an URI. If format is :html and *document-mark-up-signatures* is true, then the locative as displayed in the signature will be a link to this uri. See make-git-source-uri-fn.

pages may look something like this:

`((;; The section about SECTIONs and everything below it ...
   :objects (, @sections)
   ;; ... is so boring that it's not worth the disk space, so
   ;; send it to a string.
   :output (nil)
   ;; Explicitly tell other pages not to link to these guys.
   :uri-fragment nil)
  ;; Send the @EXTENSION-API section and everything reachable
  ;; from it ...
  (:objects (, @extension-api)
   ;; ... to build/tmp/pax-extension-api.html.
   :output "build/tmp/pax-extension-api.html"
   ;; However, on the web server html files will be at this
   ;; location relative to some common root, so override the
   ;; default:
   :uri-fragment "doc/dev/pax-extension-api.html"
   ;; Set html page title, stylesheet, charset.
   :header-fn 'write-html-header
   ;; Just close the body.
   :footer-fn 'write-html-footer)
  ;; Catch references that were not reachable from the above. It
  ;; is important for this page spec to be last.
  (:objects (, @pax-manual)
   :output "build/tmp/manual.html"
   ;; Links from the extension api page to the manual page will
   ;; be to ../user/pax-manual#<anchor>, while links going to
   ;; the opposite direction will be to
   ;; ../dev/pax-extension-api.html#<anchor>.
   :uri-fragment "doc/user/pax-manual.html"
   :header-fn 'write-html-header
   :footer-fn 'write-html-footer))

λ

8.1.4 Package and Readtable

While generating documentation, symbols may be read (e.g. from docstrings) and printed. What values of *package* and *readtable* are used is determined separately for each definition being documented.

The values thus determined come into effect after the name itself is printed, for printing of the arglist and the docstring.

CL-USER> (pax:document #'foo)
- [function] FOO <!> X Y &KEY (ERRORP T)

    Do something with X and Y.

In the above, the <!> marks the place where *package* and *readtable* are bound.

λ

Home Section

The home section of an object is a section that contains the object's definition in its section-entries or nil. In the overwhelming majority of cases there should be at most one containing section.

If there are multiple containing sections, the following apply.

For example, (mgl-pax:document function) is an entry in the MGL-PAX::@BASICS section. Unless another section that contains it is defined in the MGL-PAX package, the home section is guaranteed to be MGL-PAX::@BASICS because the symbol-packages of mgl-pax:document and MGL-PAX::@BASICS are the same (hence their common prefix is maximally long).

This scheme would also work, for example, if the home package of document were mgl-pax/impl, and it were reexported from mgl-pax because the only way to externally change the home package would be to define a containing section in a package like mgl-pax/imp.

Thus, relying on the package system makes it possible to find the intended home section of a definition among multiple containing sections with high probability. However, for names which are not symbols, there is no package system to advantage of.

λ

8.2 The mgl-pax/document ASDF System

λ

8.3 Browsing Live Documentation

Documentation can be browsed live in Emacs or with an external browser. HTML documentation, complete with Codification and links, is generated from docstrings of all kinds of Lisp definitions and PAX sections.

If Emacs Setup has been done, the Elisp function mgl-pax-document generates and displays documentation as a single HTML page. For example, to view the documentation of this very section, one can do:

M-x mgl-pax-document
View Documentation of: pax::@browsing-live-documentation

If the empty string is entered, and there is no existing w3m buffer or w3m is not used, then sections registered in PAX World are listed. If there is a w3m buffer, then entering the empty string displays that buffer.

If we enter function instead, then a disambiguation page will be shown with the documentation of the function class and the function locative. One may then follow the links on the page to navigate to a page with the documentation the desired definition. If you are browsing live documentation right now, then the disambiguation page is like this: function(0 1). In offline documentation, multiple links are shown instead as described in Ambiguous Unspecified Locative.

Alternatively, a locative may be entered as part of the argument to mgl-pax-document as in function class, which gives this result. Finally, the definition of defsection in the context of a single-page PAX Manual can be viewed by entering pax::@pax-manual#pax:defsection pax:macro.

In interactive use, mgl-pax-document defaults to documenting slime-symbol-at-point, possibly with a nearby locative the same way as in Navigating Sources in Emacs. The convenience function mgl-pax-document-current-definition documents the definition with point in it.

λ

8.3.1 PAX URLs

A PAX URL consists of a REFERENCE and an optional fragment part:

URL = [REFERENCE] ["#" FRAGMENT]

where REFERENCE names either

λ

8.3.2 Apropos

The Elisp functions mgl-pax-apropos, mgl-pax-apropos-all, and mgl-pax-apropos-package can display the results of dref:dref-apropos in the live documentation browser. These parallel the functionality of slime-apropos, slime-apropos-all, and slime-apropos-package.

dref:dref-apropos itself is similar to cl:apropos-list, but it supports more flexible matching – e.g. filtering by locative types – and returns DRef references.

The returned references are presented in two groups: those with non-symbol and those with symbol names. The non-symbol group is sorted by locative type then by name. The symbol group is sorted by name then by locative type.

λ

8.3.3 Emacs Setup for Browsing

Make sure Emacs Setup has been done. In particular, set mgl-pax-browser-function to choose between browsing documentation with w3m in an Emacs buffer, or with an external browser plus a web server in the Lisp image.

In Emacs Setup, (mgl-pax-hijack-slime-doc-keys) was evaluated, which handles the common case of binding keys. The Elisp definition is reproduced here for its docstring.

(defun mgl-pax-hijack-slime-doc-keys ()
  "Make the following changes to `slime-doc-map' (assuming it's
bound to `C-c C-d').

- `C-c C-d a': `mgl-pax-apropos' (replaces `slime-apropos')
- `C-c C-d z': `mgl-pax-aproposa-all' (replaces `slime-apropos-all')
- `C-c C-d p': `mgl-pax-apropos-package' (replaces `slime-apropos-package')
- `C-c C-d d': `mgl-pax-document' (replaces `slime-describe-symbol')
- `C-c C-d f': `mgl-pax-document' (replaces `slime-describe-function')
- `C-c C-d c': `mgl-pax-current-definition-toggle-view'

Also, regardless of whether `w3m' is available, add this:

- `C-c C-d u': `mgl-pax-edit-parent-section'

In addition, because it can be almost as useful as `M-.', one may
want to give `mgl-pax-document' a more convenient binding such as
`C-.' or `s-.' if you have a Super key. For example, to bind
`C-.' in all Slime buffers:

    (slime-bind-keys slime-parent-map nil '((\"C-.\" mgl-pax-document)))

To bind `C-.' globally:

    (global-set-key (kbd \"C-.\") 'mgl-pax-document)"
  
...)

λ

8.3.4 Browsing with w3m

With w3m's default key bindings, moving the cursor between links involves tab and s-tab (or <up> and <down>). ret and <right> follow a link, while b and <left> go back in history.

In addition, the following PAX-specific key bindings are available:

λ

8.3.5 Browsing with Other Browsers

When the value of the Elisp variable mgl-pax-browser-function is not w3m-browse-url, requests are served via a web server started in the running Lisp, and documentation is most likely displayed in a separate browser window .

By default, mgl-pax-browser-function is nil, which makes PAX use browse-url-browser-function. You may want to customize the related browse-url-new-window-flag or, for Chrome, set browse-url-chrome-arguments to ("--new-window").

In the browser, clicking on the locative on the left of the object (e.g. in - [function] PRINT) will raise and focus the Emacs window (if Emacs is not in text mode, and also subject to window manager focus stealing settings), then go to the corresponding source location. For sections, clicking on the lambda link will do the same (see *document-fancy-html-navigation*).

Finally, note that the urls exposed by the web server are subject to change.

λ

8.4 Markdown Support

The markdown in docstrings is processed with the 3BMD library.

λ

8.4.1 Markdown in Docstrings

Docstrings of definitions which do not have a Home Section and are not sections themselves are assumed to have been written with no knowledge of PAX and to conform to markdown only by accident. These docstrings are thus sanitized more aggressively.

λ

8.4.2 Syntax Highlighting

For syntax highlighting, Github's fenced code blocks markdown extension to mark up code blocks with triple backticks is enabled so all you need to do is write:

```elisp
(defun foo ())
```

to get syntactically marked up HTML output. Copy src/style.css from PAX and you are set. The language tag, elisp in this example, is optional and defaults to common-lisp.

See the documentation of 3BMD and Colorize for the details.

λ

8.4.3 MathJax

Displaying pretty mathematics in TeX format is supported via MathJax. It can be done inline with $ like this:

$\int_0^\infty e^{-x^2} dx=\frac{\sqrt{\pi}}{2}$

which is displayed as $\int_0^\infty e^{-x^2} dx=\frac{\sqrt{\pi}}{2}$, or it can be delimited by $$ like this:

$$\int_0^\infty e^{-x^2} dx=\frac{\sqrt{\pi}}{2}$$

to get: $$\int_0^\infty e^{-x^2} dx=\frac{\sqrt{\pi}}{2}$$

MathJax will leave code blocks (including those inline with backticks) alone. Outside code blocks, escape $ by prefixing it with a backslash to scare MathJax off.

Escaping all those backslashes in TeX fragments embedded in Lisp strings can be a pain. Pythonic String Reader can help with that.

λ

8.5 Codification

λ

8.6 Linking to Code

In this section, we describe all ways of linking to code available when *document-uppercase-is-code* is true.

Note that invoking M-. on the name of any of the following links will disambiguate based the textual context, determining the locative. In a nutshell, if M-. works without popping up a list of choices, then the documentation will contain a single link.

λ

8.6.1 Specified Locative

The following examples all render as document.

The Markdown link definition (i.e. function between the second set of brackets above) needs no backticks to mark it as code.

Here and below, the name (document) is uppercased, and we rely on *document-uppercase-is-code* being true. Alternatively, the name could be explicitly marked up as code with a pair of backticks, and then its character case would likely not matter (subject to readtable-case).

The link text in the above examples is document. To override it, this form may be used:

λ

8.6.2 Unspecified Locative

When only an name is provided without a locative, all definitions of the name are considered as possible link targets. Then, definitions that are not symbol-based (i.e. whose xref-name is not a symbol) are filtered out to prevent unrelated packages and asdf:systems from cluttering the documentation without the control provided by importing symbols.

To further reduce clutter, if the definitions include a generic-function locative, then all references with locative-type method, accessor, reader and writer are removed to avoid linking to a possibly large number of methods.

Furthermore, filter out all references with locative-type locative if there are references with other locative-types.

λ

Unambiguous Unspecified Locative

In the following examples, although no locative is specified, document names a single name being documented, so they all render as document.

To override the title:

λ

Ambiguous Unspecified Locative

These examples all render as section, linking to both definitions of the name section, the class and the locative. Note that the rendered output is a single link to a disambiguation page when Browsing Live Documentation, while multiple, numbered links are generated in offline documentation.

To override the title:

λ

8.6.3 Explicit and Autolinking

The examples in the previous sections are marked with explicit link or autolink. Explicit links are those with a Markdown reference link spelled out explicitly, while autolinks are those without.

λ

8.6.4 Preventing Autolinking

In the common case, when *document-uppercase-is-code* is true, prefixing an uppercase word with a backslash prevents it from being codified and thus also prevents autolinking form kicking in. For example,

\DOCUMENT

renders as DOCUMENT. If it should be marked up as code but not autolinked, the backslash must be within backticks like this:

`\DOCUMENT`

This renders as document. Alternatively, the dislocated or the argument locative may be used as in [DOCUMENT][dislocated].

λ

8.6.5 Unresolvable Links

λ

8.6.6 Suppressed Links

Autolinking of code (i.e. of something like foo) is suppressed if it would create a link that was already made within the same docstring. In the following docstring, only the first foo will be turned into a link.

"`FOO` is safe. `FOO` is great."

However, explicit links (when a locative was specified or found near the name) are never suppressed. In the following, in both docstrings, both occurrences foo produce links.

"`FOO` is safe. [`FOO`][macro] is great."
"`FOO` is safe. Macro `FOO` is great."

As an exception, links with specified and unambiguous locatives to sections and glossary-terms always produce a link to allow their titles to be displayed properly.

Finally, autolinking to t or nil is suppressed (see *document-link-to-hyperspec*).

λ

8.6.7 Local References

To declutter the generated output by reducing the number of links, the so-called local references (e.g. references to the very definition for which documentation is being generated) are treated specially. In the following example, there are local references to the function foo and its arguments, so none of them get turned into links:

(defun foo (arg1 arg2)
  "FOO takes two arguments: ARG1 and ARG2."
  t)

If linking was desired, one could use a Specified Locative (e.g. [FOO][function] or FOO function), which results in a single link. An explicit link with an unspecified locative like [foo][] generates links to all references involving the foo symbol except the local ones.

The exact rules for local references are as follows:

λ

8.7 Linking to the Hyperspec

λ

8.8 Linking to Sections

The following variables control how to generate section numbering, table of contents and navigation links.

λ

8.9 Miscellaneous Variables

λ

8.10 Utilities for Generating Documentation

Two convenience functions are provided to serve the common case of having an ASDF system with some readmes and a directory with for the HTML documentation and the default CSS stylesheet.

λ

8.10.1 HTML Output

See the following variables, which control HTML generation.

λ

8.10.2 Github Workflow

It is generally recommended to commit generated readmes (see update-asdf-system-readmes), so that users have something to read without reading the code and sites like github can display them.

HTML documentation can also be committed, but there is an issue with that: when linking to the sources (see make-git-source-uri-fn), the commit id is in the link. This means that code changes need to be committed first, and only then can HTML documentation be regenerated and committed in a followup commit.

The second issue is that github is not very good at serving HTML files from the repository itself (and http://htmlpreview.github.io chokes on links to the sources).

The recommended workflow is to use gh-pages, which can be made relatively painless with the git worktree command. The gist of it is to make the doc/ directory a checkout of the branch named gh-pages. There is a good description of this general process. Two commits are needed still, but it is somewhat less painful.

This way the HTML documentation will be available at

http://<username>.github.io/<repo-name>

It is probably a good idea to add sections like the Links and Systems section to allow jumping between the repository and the gh-pages site.

λ

8.10.3 PAX World

PAX World is a registry of documents, which can generate cross-linked HTML documentation pages for all the registered documents. There is an official PAX World.

For example, this is how PAX registers itself:

(defun pax-sections ()
  (list @pax-manual))

(defun pax-pages ()
  `((:objects ,(pax-sections)
     :source-uri-fn ,(make-git-source-uri-fn
                      :mgl-pax
                      "https://github.com/melisgl/mgl-pax"))))

(register-doc-in-pax-world :pax 'pax-sections 'pax-pages)

λ

8.11 Overview of Escaping

Let's recap how escaping Codification, downcasing, and Linking to Code works.

In the following examples capital C/D/A letters mark the presence, and a/b/c the absence of codification, downcasing, and autolinking assuming all these features are enabled by *document-uppercase-is-code*, *document-downcase-uppercase-code*, and *document-link-code*.

DOCUMENT                => [`document`][1234]    (CDA)
\DOCUMENT               => DOCUMENT              (cda)
`\DOCUMENT`             => `document`            (CDa)
`\\DOCUMENT`            => `DOCUMENT`            (CdA)
[DOCUMENT][]            => [`document`][1234]    (CDA)
[\DOCUMENT][]           => [DOCUMENT][1234]      (cdA)
[`\DOCUMENT`][]         => [`document`][1234]    (CDA) *
[`\\DOCUMENT`][]        => [`DOCUMENT`][1234]    (CdA)
[DOCUMENT][dislocated]  => `document`            (CDa)

Note that in the example marked with *, the single backslash, that would normally turn autolinking off, is ignored because it is in an explicit link.

λ

8.12 Output Details

By default, drefs are documented in the following format.

- [<locative-type>] <name> <arglist>

    <docstring>

The line with the bullet is printed with documenting-reference. The docstring is processed with document-docstring while Local References established with with-dislocated-names are in effect for all variables locally bound in a definition with arglist, and *package* is bound to the second return value of docstring.

With this default format, PAX supports all locative types, but for some Locative Types defined in DRef and the PAX Locatives, special provisions have been made.

When the printed initform is too long, it is truncated.

λ

8.13 Documentation Generation Implementation Notes

Documentation Generation is supported on ABCL, AllegroCL, CLISP, CCL, CMUCL, ECL and SBCL, but their outputs may differ due to the lack of some introspective capability. SBCL generates complete output. see arglist, docstring and source-location for implementation notes.

In addition, CLISP does not support the ambiguous case of PAX URLs for Browsing Live Documentation because the current implementation relies on Swank to list definitions of symbols (as variable, function, etc), and that simply doesn't work.

λ

9 Transcripts

What are transcripts for? When writing a tutorial, one often wants to include a REPL session with maybe a few defuns and a couple of forms whose output or return values are shown. Also, in a function's docstring an example call with concrete arguments and return values speaks volumes. A transcript is a text that looks like a REPL session, but which has a light markup for printed output and return values, while no markup (i.e. prompt) for Lisp forms. PAX transcripts may include output and return values of all forms, or only selected ones. In either case, the transcript itself can be easily generated from the source code.

The main worry associated with including examples in the documentation is that they tend to get out-of-sync with the code. This is solved by being able to parse back and update transcripts. In fact, this is exactly what happens during documentation generation with PAX. Code sections tagged with cl-transcript are retranscribed and checked for consistency (that is, no difference in output or return values). If the consistency check fails, an error is signalled that includes a reference to the object being documented.

Going beyond documentation, transcript consistency checks can be used for writing simple tests in a very readable form. For example:

(+ 1 2)
=> 3

(values (princ :hello) (list 1 2))
.. HELLO
=> :HELLO
=> (1 2)

All in all, transcripts are a handy tool especially when combined with the Emacs support to regenerate them and with PYTHONIC-STRING-READER's triple-quoted strings, that allow one to work with nested strings with less noise. The triple-quote syntax can be enabled with:

(in-readtable pythonic-string-syntax)

λ

9.1 The mgl-pax/transcribe ASDF System

λ

9.2 Transcribing with Emacs

Typical transcript usage from within Emacs is simple: add a Lisp form to a docstring or comment at any indentation level. Move the cursor right after the end of the form as if you were to evaluate it with C-x C-e. The cursor is marked by #\^:

This is part of a docstring.

```cl-transcript
(values (princ :hello) (list 1 2))^
```

Note that the use of fenced code blocks with the language tag cl-transcript is only to tell PAX to perform consistency checks at documentation generation time.

Now invoke the Elisp function mgl-pax-transcribe where the cursor is, and the fenced code block from the docstring becomes:

(values (princ :hello) (list 1 2))
.. HELLO
=> :HELLO
=> (1 2)
^

Then you change the printed message and add a comment to the second return value:

(values (princ :hello-world) (list 1 2))
.. HELLO
=> :HELLO
=> (1
    ;; This value is arbitrary.
    2)

When generating the documentation you get a transcription-consistency-error because the printed output and the first return value changed, so you regenerate the documentation by marking the region of bounded by #\| and the cursor at #\^ in the example:

|(values (princ :hello-world) (list 1 2))
.. HELLO
=> :HELLO
=> (1
    ;; This value is arbitrary.
    2)
^

then invoke the Elisp function mgl-pax-retranscribe-region to get:

(values (princ :hello-world) (list 1 2))
.. HELLO-WORLD
=> :HELLO-WORLD
=> (1
    ;; This value is arbitrary.
    2)
^

Note how the indentation and the comment of (1 2) were left alone, but the output and the first return value got updated.

Alternatively, C-u 1 mgl-pax-transcribe will emit commented markup:

(values (princ :hello) (list 1 2))
;.. HELLO
;=> :HELLO
;=> (1 2)

C-u 0 mgl-pax-retranscribe-region will turn commented into non-commented markup. In general, the numeric prefix argument is the index of the syntax to be used in *transcribe-syntaxes*. Without a prefix argument, mgl-pax-retranscribe-region will not change the markup style.

Finally, not only do both functions work at any indentation level but in comments too:

;;;; (values (princ :hello) (list 1 2))
;;;; .. HELLO
;;;; => :HELLO
;;;; => (1 2)

The dynamic environment of the transcription is determined by the :dynenv argument of the enclosing cl-transcript code block (see Controlling the Dynamic Environment).

Transcription support in Emacs can be enabled by loading src/mgl-pax.el. See Emacs Setup.

λ

9.3 Transcript API

λ

9.4 Transcript Consistency Checking

The main use case for consistency checking is detecting out-of-date examples in documentation, although using it for writing tests is also a possibility. Here, we focus on the former.

When a markdown code block tagged cl-transcript is processed during Generating Documentation, the code in it is replaced with the output of with (transcribe <code> nil :update-only t :check-consistency t). Suppose we have the following example of the function greet, that prints hello and returns 7.

```cl-transcript
(greet)
.. hello
=> 7
```

Now, if we change greet to print or return something else, a transcription-consistency-error will be signalled during documentation generation. Then we may fix the documentation or continue from the error.

By default, comparisons of previous to current output, readable and unreadable return values are performed with string=, equal, and string=, respectively, which is great in the simple case. Non-determinism aside, exact matching becomes brittle as soon as the notoriously unportable pretty printer is used or when unreadable objects are printed with their #<> syntax, especially when print-unreadable-object is used with :identity t.

λ

9.4.1 Finer-Grained Consistency Checks

To get around this problem, consistency checking of output, readable and unreadable values can be customized individually by supplying transcribe with a check-consistency argument like ((:output <output-check>) (:readable <readable-check>) (:unreadable <unreadable-check>)). In this case, <output-check> may be nil, t, or a function designator.

The case of <readable-check> and <unreadable-check> is similar.

Code blocks tagged cl-transcript can take arguments, which they pass on to transcribe. The following shows how to check only the output.

```cl-transcript (:check-consistency ((:output t)))
(error "Oh, no.")
.. debugger invoked on SIMPLE-ERROR:
..   Oh, no.

(make-condition 'simple-error)
==> #<SIMPLE-ERROR {1008A81533}>

λ

9.4.2 Controlling the Dynamic Environment

The dynamic environment in which forms in the transcript are evaluated can be controlled via the :dynenv argument of cl-transcript.

```cl-transcript (:dynenv my-transcript)
...
```

In this case, instead of calling transcribe directly, the call will be wrapped in a function of no arguments and passed to the function my-transcript, which establishes the desired dynamic environment and calls its argument. The following definition of my-transcript simply packages up oft-used settings to transcribe.

(defun my-transcript (fn)
  (let ((*transcribe-check-consistency*
          '((:output my-transcript-output=)
            (:readable equal)
            (:unreadable nil))))
    (funcall fn)))

(defun my-transcript-output= (string1 string2)
  (string= (my-transcript-normalize-output string1)
           (my-transcript-normalize-output string2)))

(defun my-transcript-normalize-output (string)
  (squeeze-whitespace (delete-trailing-whitespace (delete-comments string))))

A more involved solution could rebind global variables set in transcripts, unintern symbols created or even create a temporary package for evaluation.

λ

9.4.3 Utilities for Consistency Checking

λ

10 Writing Extensions

λ

10.1 Adding New Locatives

Once everything in Adding New Locatives has been done, there are only a couple of PAX generic functions left to extend.

Also note that due to the Home Section logic, especially for locative types with string names, dref-ext:docstring* should probably return a non-nil package.

λ

10.2 Locative Aliases

define-locative-alias can be used to help M-. and autolinking disambiguate references based on the context of a name as described on Parsing and also in Specified Locative.

The following example shows how to make docstrings read more naturally by defining an alias.

(defclass my-string ()
  ())

(defgeneric my-string (obj)
  (:documentation "Convert OBJ to MY-STRING."))

;;; This version of FOO has a harder to read docstring because
;;; it needs to disambiguate the MY-STRING reference.
(defun foo (x)
  "FOO takes and argument X, a [MY-STRING][class] object.")

;;; Define OBJECT as an alias for the CLASS locative.
(define-locative-alias object class)

;;; Note how no explicit link is needed anymore.
(defun foo (x)
  "FOO takes an argument X, a MY-CLASS object.")

Similary, defining the indefinite articles as aliases of the class locative can reduce the need for explicit linking.

(define-locative-alias a class)
(define-locative-alias an class)

Since these are unlikely to be universally helpful, make sure not to export the symbols a and an.

λ

10.3 Extending document

For all definitions that it encounters, document calls document-object* to generate documentation. The following utilities are for writing new document-object* methods, which emit markdown.

λ

10.4 Sections

section objects rarely need to be dissected since defsection and document cover most needs. However, it is plausible that one wants to subclass them and maybe redefine how they are presented.

λ

10.5 Glossary Terms

glossary-term objects rarely need to be dissected since define-glossary-term and document cover most needs. However, it is plausible that one wants to subclass them and maybe redefine how they are presented.