→ ↺ λ

PAX Manual

Table of Contents

[in package MGL-PAX with nicknames PAX]

← ↑ → ↺ λ

1 mgl-pax ASDF System

← ↑ → ↺ λ

2 mgl-pax/full ASDF System

← ↑ → ↺ λ

3 Links

Here is the official repository and the HTML documentation for the latest version.

← ↑ → ↺ λ

4 Background

As a user, I frequently run into documentation that's incomplete and out of date, so I tend to stay in the editor and explore the code by jumping around with SLIME's M-.. As a library author, I spend a great deal of time polishing code but precious little writing documentation.

In fact, I rarely write anything more comprehensive than docstrings for exported stuff. Writing docstrings feels easier than writing a separate user manual, and they are always close at hand during development. The drawback of this style is that users of the library have to piece the big picture together themselves.

That's easy to solve, I thought, let's just put all the narrative that holds docstrings together in the code and be a bit like a Literate Programmer turned inside out. The original prototype, which did almost everything I wanted, was this:

(defmacro defsection (name docstring)
  `(defun ,name () ,docstring))

Armed with this defsection, I soon found myself organizing code following the flow of user-level documentation and relegated comments to implementation details entirely. However, some parts of defsection docstrings were just listings of all the functions, macros and variables related to the narrative, and this list was repeated in the defpackage form complete with little comments that were like section names. A clear violation of OAOO, one of them had to go, so defsection got a list of symbols to export.

That was great, but soon I found that the listing of symbols is ambiguous if, for example, a function, a compiler macro and a class are named by the same symbol. This did not concern exporting, of course, but it didn't help readability. Distractingly, on such symbols, M-. was popping up selection dialogs. There were two birds to kill, and the symbol got accompanied by a type, which was later generalized into the concept of locatives:

(defsection @introduction ()
  "A single line for one man ..."
  (foo class)
  (bar function))

After a bit of elisp hacking, M-. was smart enough to disambiguate based on the locative found in the vicinity of the symbol, and everything was good for a while.

Then I realized that sections could refer to other sections if there were a section locative. Going down that path, I soon began to feel the urge to generate pretty documentation as all the necessary information was manifest in the defsection forms. The design constraint imposed on documentation generation was that following the typical style of upcasing symbols in docstrings there should be no need to explicitly mark up links: if M-. works, then the documentation generator shall also be able find out what's being referred to.

I settled on Markdown as a reasonably non-intrusive format, and a few thousand lines later PAX was born.

← ↑ → ↺ λ

5 Tutorial

PAX provides an extremely poor man's Explorable Programming environment. Narrative primarily lives in so called sections that mix markdown docstrings with references to functions, variables, etc, all of which should probably have their own docstrings.

The primary focus is on making code easily explorable by using SLIME's M-. (slime-edit-definition). See how to enable some fanciness in Navigating Sources in Emacs. Generating Documentation from sections and all the referenced items in Markdown or HTML format is also implemented.

With the simplistic tools provided, one may accomplish similar effects as with Literate Programming, but documentation is generated from code, not vice versa and there is no support for chunking. Code is first, code must look pretty, documentation is code.

Docstrings

PAX's automatically recognizes and marks up code with backticks and links code to their definitions.

(document "&KEY arguments such as :IF-EXISTS are common.")
=> ("`&KEY` arguments such as `:IF-EXISTS` are common.
")

(document "AND denotes a macro and a type specifier.
          Here we focus on the macro AND.")
=> ("`AND`([`0`][4954] [`1`][330f]) denotes a macro and a type specifier.
Here we focus on the macro [`AND`][4954].

  [330f]: http://www.lispworks.com/documentation/HyperSpec/Body/t_and.htm \"AND TYPE\"
  [4954]: http://www.lispworks.com/documentation/HyperSpec/Body/m_and.htm \"AND MGL-PAX:MACRO\"
")

These features are designed to handle a common style of docstrings with minimal additional markup. The following is the output of (mgl-pax:document #'abort). Note that the docstring of the abort function was not written with PAX in mind.

A Complete Example

Here is an example of how it all works together:

(mgl-pax:define-package :foo-random
  (:documentation "This package provides various utilities for random.
  See FOO-RANDOM:@FOO-RANDOM-MANUAL.")
  (:use #:common-lisp #:mgl-pax))

(in-package :foo-random)

(defsection @foo-random-manual (:title "Foo Random manual")
  "Here you describe what's common to all the referenced (and
  exported) functions that follow. They work with *FOO-STATE*, and
  have a :RANDOM-STATE keyword arg. Also explain when to choose
  which."
  (foo-random-state class)
  (state (reader foo-random-state))
  "Hey we can also print states!"
  (print-object (method () (foo-random-state t)))
  (*foo-state* variable)
  (gaussian-random function)
  (uniform-random function)
  ;; This is a subsection.
  (@foo-random-examples section))

(defclass foo-random-state ()
  ((state :reader state)))

(defmethod print-object ((object foo-random-state) stream)
  (print-unreadable-object (object stream :type t)))

(defvar *foo-state* (make-instance 'foo-random-state)
  "Much like *RANDOM-STATE* but uses the FOO algorithm.")

(defun uniform-random (limit &key (random-state *foo-state*))
  "Return a random number from the between 0 and LIMIT (exclusive)
  uniform distribution."
  nil)

(defun gaussian-random (stddev &key (random-state *foo-state*))
  "Return a random number from a zero mean normal distribution with
  STDDEV."
  nil)

(defsection @foo-random-examples (:title "Examples")
  "Let's see the transcript of a real session of someone working
  with FOO:

  ```cl-transcript
  (values (princ :hello) (list 1 2))
  .. HELLO
  => :HELLO
  => (1 2)

  (make-instance 'foo-random-state)
  ==> #<FOO-RANDOM-STATE >
  ```")

Note how (variable *foo-state*) in the defsection form both exports *foo-state* and includes its documentation in @foo-random-manual. The symbols variable and function are just two instances of locatives, which are used in defsection to refer to definitions tied to symbols.

(document @foo-random-manual) generates fancy markdown or HTML output with automatic markup and autolinks uppercase words found in docstrings, numbers sections, and creates a table of contents.

One can even generate documentation for different but related libraries at the same time with the output going to different files but with cross-page links being automatically added for symbols mentioned in docstrings. See Generating Documentation for some convenience functions to cover the most common cases.

The transcript in the code block tagged with cl-transcript is automatically checked for up-to-dateness when documentation is generated.

← ↑ → ↺ λ

6 Basics

Now let's examine the most important pieces.

← ↑ → ↺ λ

6.1 Locatives and References

To navigate with M-. and to generate documentation we need to refer to things such as the foo type or the foo function.

(deftype foo ()
  "type doc"
  '(or integer real).

(defun foo ()
  "function doc"
  7)

The docstring is available via (cl:documentation 'foo 'type), where type - called doc-type - is what tells cl:documentation that we want the docstring of the type named foo. This design supports disambiguation and working with things that are not first-class, such as types.

PAX generalizes doc-type to the concept of locatives, which may also take arguments. An object and a locative together are called a reference, and they identify a definition. references are actual objects, but often they appear as an (object locative) list (see defsection) or as "OBJECT LOCATIVE" in docstrings (see Linking to Code for the various forms possible).

(defsection @foos ()
  "We discuss the FOO type and the FOO function."
  (foo type)
  (foo function))

← ↑ → ↺ λ

6.1.1 Locatives and References API

(make-reference 'foo 'variable) constructs a reference that captures the path to take from an object (the symbol foo) to an entity of interest (for example, the documentation of the variable). The path is called the locative. A locative can be applied to an object like this:

(locate 'foo 'variable)

which will return the same reference as (make-reference 'foo 'variable). Operations need to know how to deal with references, which we will see in the Writing Extensions.

Naturally, (locate 'foo 'function) will simply return #'foo, no need to muck with references when there is a perfectly good object.

← ↑ → ↺ λ

6.2 Parsing

← ↑ → ↺ λ

7 Locative Types

As we have already briefly seen in defsection and Locatives and References, locatives allow us to refer to, document, and find the source location of various definitions beyond what standard Common Lisp offers. See Writing Extensions for a more detailed treatment. The following are the locatives types supported out of the box. As all locative types, they are named by symbols, which should make it obvious what kind of things they refer to. Unless otherwise noted, locatives take no arguments.

When there is a corresponding CL type, a locative can be resolved to a unique object as is the case in (locate 'foo 'class) returning #<class foo>. Even if there is no such CL type, the source location and the docstring of the defining form is recorded (see locate-and-find-source, locate-and-document in the Writing Extensions), which makes navigating the sources with M-. (see Navigating Sources in Emacs) and Generating Documentation possible.

← ↑ → ↺ λ

7.1 Locatives for Variables

← ↑ → ↺ λ

7.2 Locatives for Macros

← ↑ → ↺ λ

7.3 Locatives for Functions

← ↑ → ↺ λ

7.4 Locatives for Types and Declarations

← ↑ → ↺ λ

7.5 Condition System Locatives

← ↑ → ↺ λ

7.6 Locatives for Packages and Readtables

← ↑ → ↺ λ

7.7 Locatives for PAX Constructs

← ↑ → ↺ λ

7.8 External Locatives

← ↑ → ↺ λ

8 Navigating Sources in Emacs

Integration into SLIME's M-. (slime-edit-definition) allows one to visit the source location of the definition that's identified by slime-symbol-at-point parsed as a word and the locative before or after the symbol in a buffer. With this extension, if a locative is the previous or the next expression around the symbol of interest, then M-. will go straight to the definition which corresponds to the locative. If that fails, M-. will try to find the definitions in the normal way, which may involve popping up an xref buffer and letting the user interactively select one of possible definitions.

In the following examples, when the cursor is on one of the characters of foo or just after foo, pressing M-. will visit the definition of function foo:

function foo
foo function
(function foo)
(foo function)

In particular, references in a defsection form are in (symbol locative) format so M-. will work just fine there.

Just like vanilla M-., this works in comments and docstrings. In the next example, pressing M-. on foo will visit foo's default method:

;;;; See FOO `(method () (t t t))` for how this all works.
;;;; But if the locative has semicolons inside: FOO `(method
;;;; () (t t t))`, then it won't, so be wary of line breaks
;;;; in comments.

With a prefix argument (C-u M-.), one can enter a symbol plus a locative separated by whitespace to preselect one of the possibilities.

The M-. extensions can be enabled by loading src/pax.el.

← ↑ → ↺ λ

8.1 mgl-pax/navigate ASDF System

← ↑ → ↺ λ

9 Generating Documentation

← ↑ → ↺ λ

9.1 mgl-pax/document ASDF System

← ↑ → ↺ λ

9.2 Markdown Support

The Markdown in docstrings is processed with the 3BMD library.

← ↑ → ↺ λ

9.2.1 Indentation

Docstrings can be indented in any of the usual styles. PAX normalizes indentation by converting:

(defun foo ()
  "This is
  indented
  differently")

to

(defun foo ()
  "This is
indented
differently")

See document-object for the details.

← ↑ → ↺ λ

9.2.2 Syntax Highlighting

For syntax highlighting, github's fenced code blocks markdown extension to mark up code blocks with triple backticks is enabled so all you need to do is write:

```elisp
(defun foo ())
```

to get syntactically marked up HTML output. Copy src/style.css from PAX and you are set. The language tag, elisp in this example, is optional and defaults to common-lisp.

See the documentation of 3BMD and colorize for the details.

← ↑ → ↺ λ

9.2.3 MathJax

Displaying pretty mathematics in TeX format is supported via MathJax. It can be done inline with $ like this:

$\int_0^\infty e^{-x^2} dx=\frac{\sqrt{\pi}}{2}$

which is diplayed as $\int_0^\infty e^{-x^2} dx=\frac{\sqrt{\pi}}{2}$, or it can be delimited by $$ like this:

$$\int_0^\infty e^{-x^2} dx=\frac{\sqrt{\pi}}{2}$$

to get: $$\int_0^\infty e^{-x^2} dx=\frac{\sqrt{\pi}}{2}$$

MathJax will leave code blocks (including those inline with backticks) alone. Outside code blocks, escape $ by prefixing it with a backslash to scare MathJax off.

Escaping all those backslashes in TeX fragments embedded in Lisp strings can be a pain. Pythonic String Reader can help with that.

← ↑ → ↺ λ

9.3 Codification

← ↑ → ↺ λ

9.4 Linking to Code

In this section, we describe all ways of linking to code available when *document-uppercase-is-code* is true.

Note that invoking M-. on the object of any of the following links will disambiguate based the textual context, determining the locative. In a nutshell, if M-. works without popping up a list of choices, then the documentation will contain a single link.

← ↑ → ↺ λ

9.4.1 Specified Locative

The following examples all render as document.

The Markdown link definition (i.e. function between the second set of brackets above) needs no backticks to mark it as code.

Here and below, the object (document) is uppercased, and we rely on *document-uppercase-is-code* being true. Alternatively, the object could be explicitly marked up as code with a pair of backticks, and then its character case would likely not matter (subject to readtable-case).

The link text in the above examples is document. To override it, this form may be used:

← ↑ → ↺ λ

9.4.2 Unambiguous Unspecified Locative

In the following examples, although no locative is specified, document names a single object being documented, so they all render as document.

To override the title:

← ↑ → ↺ λ

9.4.3 Ambiguous Unspecified Locative

These examples all render as section(0 1), linking to both definitions of the object section, the class and the locative.

To override the title:

← ↑ → ↺ λ

9.4.4 Explicit and Autolinking

The examples in the previous sections are marked with explicit link or autolink. Explicit links are those with a Markdown reference link spelled out explicitly, while autolinks are those without.

← ↑ → ↺ λ

9.4.5 Preventing Autolinking

In the common case, when *document-uppercase-is-code* is true, prefixing the uppercase word with a backslash prevents it from being codified and thus also prevents autolinking form kicking in. For example,

\DOCUMENT

renders as DOCUMENT. If it should be marked up as code but not autolinked, the backslash must be within backticks like this:

`\DOCUMENT`

This renders as document. Alternatively, the dislocated or the argument locative may be used as in [DOCUMENT][dislocated].

← ↑ → ↺ λ

9.4.6 Unresolvable Links

← ↑ → ↺ λ

9.4.7 Suppressed Links

Within the same docstring, autolinking of code (i.e. of something like foo) is suppressed if the same object was already linked to in any way. In the following docstring, only the first foo will be turned into a link.

"`FOO` is safe. `FOO` is great."

However if a locative was specified or found near the object, then a link is always made. In the following, in both docstrings, both occurrences foo produce links.

"`FOO` is safe. [`FOO`][macro] is great."
"`FOO` is safe. Macro `FOO` is great."

As an exception, links with specified and unambiguous locatives to section(0 1)s and glossary-term(0 1)s always produce a link to allow their titles to be displayed properly.

Finally, autolinking to t or nil is suppressed (see *document-link-to-hyperspec*).

← ↑ → ↺ λ

9.4.8 Filtering Ambiguous References

When there are multiple references to link to - as seen in Ambiguous Unspecified Locative - some references are removed by the following rules.

← ↑ → ↺ λ

9.4.9 Local References

To unclutter the generated output by reducing the number of links, the so-called 'local' references (e.g. references to the very definition for which documentation is being generated) are treated specially. In the following example, there are local references to the function foo and its arguments, so none of them get turned into links:

(defun foo (arg1 arg2)
  "FOO takes two arguments: ARG1 and ARG2."
  t)

If linking was desired, one could use a Specified Locative (e.g. [FOO][function] or FOO function), which results in a single link. An explicit link with an unspecified locative like [foo][] generates links to all references involving the foo symbol except the local ones.

The exact rules for local references are as follows:

← ↑ → ↺ λ

9.5 Linking to the Hyperspec

← ↑ → ↺ λ

9.6 Linking to Sections

The following variables control how to generate section numbering, table of contents and navigation links.

← ↑ → ↺ λ

9.7 Miscellaneous Variables

← ↑ → ↺ λ

9.8 Utilities for Generating Documentation

Two convenience functions are provided to serve the common case of having an ASDF system with some readmes and a directory with for the HTML documentation and the default css stylesheet.

← ↑ → ↺ λ

9.8.1 Github Workflow

It is generally recommended to commit generated readmes (see update-asdf-system-readmes), so that users have something to read without reading the code and sites like github can display them.

HTML documentation can also be committed, but there is an issue with that: when linking to the sources (see make-git-source-uri-fn), the commit id is in the link. This means that code changes need to be committed first, and only then can HTML documentation be regenerated and committed in a followup commit.

The second issue is that github is not very good at serving HTMLs files from the repository itself (and http://htmlpreview.github.io chokes on links to the sources).

The recommended workflow is to use gh-pages, which can be made relatively painless with the git worktree command. The gist of it is to make the doc/ directory a checkout of the branch named gh-pages. A good description of this process is http://sangsoonam.github.io/2019/02/08/using-git-worktree-to-deploy-github-pages.html. Two commits needed still, but it is somewhat less painful.

This way the HTML documentation will be available at http://<username>.github.io/<repo-name>. It is probably a good idea to add sections like the Links section to allow jumping between the repository and the gh-pages site.

← ↑ → ↺ λ

9.8.2 PAX World

PAX World is a registry of documents, which can generate cross-linked HTML documentation pages for all the registered documents.

For example, this is how PAX registers itself:

(defun pax-sections ()
  (list @pax-manual))
(defun pax-pages ()
  `((:objects
     (, @pax-manual)
     :source-uri-fn ,(make-git-source-uri-fn
                      :mgl-pax
                      "https://github.com/melisgl/mgl-pax"))))
(register-doc-in-pax-world :pax (pax-sections) (pax-pages))

← ↑ → ↺ λ

9.9 Overview of Escaping

Let's recap how escaping Codification, downcasing, and Linking to Code works.

In the following examples capital C/D/A letters mark the presence, and a/b/c the absence of codification, downcasing, and autolinking assuming all these features are enabled by *document-uppercase-is-code*. *document-downcase-uppercase-code*, and *document-link-code*.

DOCUMENT                => [`document`][1234]    (CDA)
\DOCUMENT               => DOCUMENT              (cda)
`\DOCUMENT`             => `document`            (CDa)
`\\DOCUMENT`            => `DOCUMENT`            (CdA)
[DOCUMENT][]            => [`document`][1234]    (CDA)
[\DOCUMENT][]           => [DOCUMENT][1234]      (cdA)
[`\DOCUMENT`][]         => [`document`][1234]    (CDA) *
[`\\DOCUMENT`][]        => [`DOCUMENT`][1234]    (CdA)
[DOCUMENT][dislocated]  => `document`            (CDa)

Note that in the example marked with *, the single backslash, that would normally turn autolinking off, is ignored because it is in an explicit link.

← ↑ → ↺ λ

9.10 Document Generation Implementation Notes

Documentation Generation is supported on ABCL, AllegroCL, CLISP, CCL, CMUCL, ECL and SBCL, but their outputs may differ due to the lack of some introspective capability. SBCL generates complete output. Compared to that, the following are not supported:

← ↑ → ↺ λ

10 Transcripts

What are transcripts for? When writing a tutorial, one often wants to include a REPL session with maybe a few defuns and a couple of forms whose output or return values are shown. Also, in a function's docstring an example call with concrete arguments and return values speaks volumes. A transcript is a text that looks like a repl session, but which has a light markup for printed output and return values, while no markup (i.e. prompt) for Lisp forms. PAX transcripts may include output and return values of all forms, or only selected ones. In either case, the transcript itself can be easily generated from the source code.

The main worry associated with including examples in the documentation is that they tend to get out-of-sync with the code. This is solved by being able to parse back and update transcripts. In fact, this is exactly what happens during documentation generation with PAX. Code sections tagged cl-transcript are retranscribed and checked for inconsistency (that is, any difference in output or return values). If the consistency check fails, an error is signalled that includes a reference to the object being documented.

Going beyond documentation, transcript consistency checks can be used for writing simple tests in a very readable form. For example:

(+ 1 2)
=> 3

(values (princ :hello) (list 1 2))
.. HELLO
=> :HELLO
=> (1 2)

All in all, transcripts are a handy tool especially when combined with the Emacs support to regenerate them and with PYTHONIC-STRING-READER's triple-quoted strings, that allow one to work with nested strings with less noise. The triple-quote syntax can be enabled with:

(in-readtable pythonic-string-syntax)

← ↑ → ↺ λ

10.1 mgl-pax/transcribe ASDF System

← ↑ → ↺ λ

10.2 Transcribing with Emacs

Typical transcript usage from within Emacs is simple: add a lisp form to a docstring or comment at any indentation level. Move the cursor right after the end of the form as if you were to evaluate it with C-x C-e. The cursor is marked by #\^:

This is part of a docstring.

```cl-transcript
(values (princ :hello) (list 1 2))^
```

Note that the use of fenced code blocks with the language tag cl-transcript is only to tell PAX to perform consistency checks at documentation generation time.

Now invoke the elisp function mgl-pax-transcribe where the cursor is and the fenced code block from the docstring becomes:

(values (princ :hello) (list 1 2))
.. HELLO
=> :HELLO
=> (1 2)
^

Then you change the printed message and add a comment to the second return value:

(values (princ :hello-world) (list 1 2))
.. HELLO
=> :HELLO
=> (1
    ;; This value is arbitrary.
    2)

When generating the documentation you get a transcription-consistency-error because the printed output and the first return value changed so you regenerate the documentation by marking the region of bounded by #\| and the cursor at #\^ in the example:

|(values (princ :hello-world) (list 1 2))
.. HELLO
=> :HELLO
=> (1
    ;; This value is arbitrary.
    2)
^

then invoke the elisp function mgl-pax-retranscribe-region to get:

(values (princ :hello-world) (list 1 2))
.. HELLO-WORLD
=> :HELLO-WORLD
=> (1
    ;; This value is arbitrary.
    2)
^

Note how the indentation and the comment of (1 2) was left alone but the output and the first return value got updated.

Alternatively, C-u 1 mgl-pax-transcribe will emit commented markup:

(values (princ :hello) (list 1 2))
;.. HELLO
;=> :HELLO
;=> (1 2)

C-u 0 mgl-pax-retranscribe-region will turn commented into non-commented markup. In general, the numeric prefix argument is the index of the syntax to be used in mgl-pax:*transcribe-syntaxes*. Without a prefix argument mgl-pax-retranscribe-region will not change the markup style.

Finally, not only do both functions work at any indentation level, but in comments too:

;;;; (values (princ :hello) (list 1 2))
;;;; .. HELLO
;;;; => :HELLO
;;;; => (1 2)

Transcription support in emacs can be enabled by loading src/transcribe.el.

← ↑ → ↺ λ

10.3 Transcript API

← ↑ → ↺ λ

10.4 Transcript Consistency Checking

The main use case for consistency checking is detecting out-of-date examples in documentation, although using it for writing tests is also a possiblity. Here, we focus on the former.

When a markdown code block tagged cl-transcript is processed during Generating Documentation, the code in it is replaced with the output of with (transcribe <code> nil :update-only t :check-consistency t). Suppose we have the following example of the function greet, that prints hello and returns 7.

```cl-transcript
(greet)
.. hello
=> 7
```

Now, if we change greet to print or return something else, a transcription-consistency-error will be signalled during documentation generation. Then we may fix the documentation or continue from the error.

By default, comparisons of previous to current ouput, readable and unreadable return values are performed with string=, equal, and string=, respectively, which is great in the simple case. Non-determinism aside, exact matching becomes brittle as soon as the notoriously unportable pretty printer is used or when unreadable objects are printed with their #<> syntax, especially when print-unreadable-object is used with :identity t.

← ↑ → ↺ λ

10.4.1 Finer-grained Consistency Checks

To get around this problem, consistency checking of output, readable and unreadable values can be customized individually by supplying transcribe with a check-consistency argument like ((:output <output-check>) (:readable <readable-check>) (:unreadable <unreadable-check>)). In this case, <output-check> may be nil, t, or a function designator.

The case of <readable-check> and <unreadable-check> is similar.

Code blocks tagged cl-transcript can take arguments, which they pass on to transcribe. The following shows how to check only the output.

```cl-transcript (:check-consistency ((:output t)))
(error "Oh, no.")
.. debugger invoked on SIMPLE-ERROR:
..   Oh, no.

(make-condition 'simple-error)
==> #<SIMPLE-ERROR {1008A81533}>

← ↑ → ↺ λ

10.4.2 Controlling the Dynamic Environment

The dynamic enviroment in which forms in the transcript are evaluated can be controlled via the :dynenv argument of cl-transcript.

```cl-transcript (:dynenv my-transcript)
...
```

In this case, instead of calling transcribe directly, the call will be wrapped in a function of no arguments and passed to the function my-transcript, which establishes the desired dynamic environment and calls its argument. The following definition of my-transcript simply packages up oft-used settings to transcribe.

(defun my-transcript (fn)
  (let ((*transcribe-check-consistency*
          '((:output my-transcript-output=)
            (:readable equal)
            (:unreadable nil))))
    (funcall fn)))

(defun my-transcript-output= (string1 string2)
  (string= (my-transcript-normalize-output string1)
           (my-transcript-normalize-output string2)))

(defun my-transcript-normalize-output (string)
  (squeeze-whitespace (delete-trailing-whitespace (delete-comments string))))

A more involved solution could rebind global variables set in transcripts, unintern symbols created or even create a temporary package for evaluation.

← ↑ → ↺ λ

10.4.3 Utilities for Consistency Checking

← ↑ → ↺ λ

11 Writing Extensions

← ↑ → ↺ λ

11.1 Adding New Object Types

One may wish to make the document function and M-. navigation work with new object types. document can be extended by defining a document-object method specialized on that type. To allow these objects to be referenced from defsection, locate-object method is to be defined. If there are multiple equivalent references possible for the same thing, then canonical-reference must be specialized. For the docstring locative to work on the new type, a docstring method is needed. For M-. find-source can be specialized. Finally, exportable-locative-type-p may be overridden if exporting does not makes sense. Here is how all this is done for asdf:system:

(define-locative-type asdf:system ()
  "Refers to an asdf system. The generated documentation will include
  meta information extracted from the system definition. This also
  serves as an example of a symbol that's not accessible in the
  current package and consequently is not exported.

  ASDF:SYSTEM is not EXPORTABLE-LOCATIVE-TYPE-P.")

(defmethod locate-object (name (locative-type (eql 'asdf:system))
                          locative-args)
  (or (and (endp locative-args)
           ;; ASDF:FIND-SYSTEM is slow as hell.
           (asdf:find-system (string-downcase (string name)) nil))
      (locate-error "~S does not name an asdf system." name)))

(defmethod canonical-reference ((system asdf:system))
  (make-reference (character-string (slot-value system 'asdf::name))
                  'asdf:system))

;;; For testing
(defvar *omit-asdf-slots* nil)

(defmethod document-object ((system asdf:system) stream)
  (with-heading (stream system
                        (format nil "~A \\ASDF System"
                                (string-upcase
                                 (slot-value system 'asdf::name))))
    (flet ((foo (name fn &key type)
             (let ((value (funcall fn system)))
               (when (and value (not (equal value "")))
                 (case type
                   ((:link)
                    (format stream "- ~A: [~A](~A)~%" name value value))
                   ((:mailto)
                    (format stream "- ~A: [~A](mailto:~A)~%"
                            name value value))
                   ((:source-control)
                    (format stream "- ~A: [~A](~A)"
                            name (first value) (second value)))
                   ((:docstring)
                    (format stream "- ~A: " name)
                    (document-docstring value stream
                                        :indentation "  "
                                        :exclude-first-line-p t
                                        :paragraphp nil)
                    (terpri stream))
                   ((nil)
                    (format stream "- ~A: ~A~%" name value)))))))
      (unless *omit-asdf-slots*
        (foo "Version" 'asdf/component:component-version)
        (foo "Description" 'asdf/system:system-description :type :docstring)
        (foo "Long Description" 'asdf/system:system-long-description
             :type :docstring)
        (foo "Licence" 'asdf/system:system-licence)
        (foo "Author" 'asdf/system:system-author)
        (foo "Maintainer" 'asdf/system:system-maintainer)
        (foo "Mailto" 'asdf/system:system-mailto :type :mailto)
        (foo "Homepage" 'asdf/system:system-homepage :type :link)
        (foo "Bug tracker" 'asdf/system:system-bug-tracker :type :link)
        (foo "Source control" 'asdf/system:system-source-control
             :type :source-control)
        (terpri stream)))))

(defmethod docstring ((system asdf:system))
  nil)

(defmethod find-source ((system asdf:system))
  `(:location
    (:file ,(namestring (asdf/system:system-source-file system)))
    (:position 1)
    (:snippet "")))

(add-locative-to-source-search-list 'asdf:system)

← ↑ → ↺ λ

11.2 Reference Based Extensions

Let's see how to extend document and M-. navigation if there is no first-class object to represent the definition of interest. Recall that locate returns a reference object in this case. The generic functions that we have specialized in Adding New Object Types have reference delegates, which can be specialized based on locative-type. Here is how the variable locative is defined:

(define-locative-type variable (&optional initform)
  """Refers to a global special variable. INITFORM, or if not specified,
  the global value of the variable is included in the documentation.

  ```
  ;;; A REFERENCE is returned because there is no such type as VARIABLE.
  (locate '*FORMAT* 'variable)
  ==> #<REFERENCE *FORMAT* VARIABLE>
  ```

  For the output of `(DOCUMENT (MAKE-REFERENCE '*FORMAT* 'VARIABLE))`,
  see *FORMAT*. Note that *FORMAT* is unbound. If the variable is
  BOUNDP, then its _current_ value is included in the documentation.
  See *DOCUMENT-LINK-CODE* for an example output. To override the
  current value, `INITFORM` may be provided. This is particulary
  useful if the value of the variable is something undesirable such as
  `\\#<MY-CLASS {100171ED93}>`.""")

(defmethod locate-object (symbol (locative-type (eql 'variable)) locative-args)
  (unless (<= (length locative-args) 1)
    (locate-error "The lambda list of the VARIABLE locative is ~
                   (&OPTIONAL INITFORM)."))
  (make-reference symbol (cons locative-type locative-args)))

(defmethod locate-and-document (symbol (locative-type (eql 'variable))
                                locative-args stream)
  (destructuring-bind (&optional (initform nil initformp)) locative-args
    (let ((arglist (multiple-value-bind (value unboundp)
                       (symbol-global-value symbol)
                     (when (or initformp (not unboundp))
                       (let ((*print-pretty* t))
                         (prin1-to-markdown (if initformp
                                                initform
                                                value)))))))
      (documenting-reference (stream :arglist arglist)
        (document-docstring (documentation* symbol 'variable) stream)))))

(defmethod locate-docstring (symbol (locative-type (eql 'variable))
                             locative-args)
  (declare (ignore locative-args))
  (documentation* symbol 'variable))

(defmethod locate-and-find-source (symbol (locative-type (eql 'variable))
                                   locative-args)
  (declare (ignore locative-args))
  (find-definition symbol 'variable))

We have covered the basic building blocks of reference based extensions. Now let's see how the obscure define-symbol-locative-type and define-definer-for-symbol-locative-type macros work together to simplify the common task of associating definition and documentation with symbols in a certain context.

← ↑ → ↺ λ

11.3 Extending document

The following utilities are for writing new document-object and locate-and-document methods, which emit markdown.

← ↑ → ↺ λ

11.4 Extending find-source

The following utilities are for writing new find-source and locate-and-find-source methods. Their locative arguments are translated to Swank dspecs, and it is an error if there is no translation. In general, Swank supports Common Lisp definitions (hence the variable and function(0 1 2) locatives, for example) but not PAX- and user-defined additions (e.g. section(0 1), asdf:system).

← ↑ → ↺ λ

11.5 Sections

section objects rarely need to be dissected since defsection and document cover most needs. However, it is plausible that one wants to subclass them and maybe redefine how they are presented.

← ↑ ↺ λ

11.6 Glossary Terms

glossary-term objects rarely need to be dissected since define-glossary-term and document cover most needs. However, it is plausible that one wants to subclass them and maybe redefine how they are presented.