λ

PAX Manual

Table of Contents

[in package MGL-PAX with nicknames PAX]

λ

1 Introduction

What if documentation really lived in the code?

Docstrings are already there. If some narrative glued them together, we'd be able develop and explore the code along with the documentation due to their physical proximity. The main tool that PAX provides for this is defsection:

(defsection @foo-random-manual (:title "Foo Random manual")
  "Foo Random is a random number generator library."
  (foo-random-state class)
  (uniform-random function)
  (@foo-random-examples section))

Like this one, sections can have docstrings and references to definitions (e.g. (uniform-random function)). These docstrings and references are the glue. To support interactive development, PAX

See Emacs Setup.

Beyond interactive workflows, Generating Documentation from sections and all the referenced items in Markdown or HTML format is also implemented.

With the simplistic tools provided, one may emphasize the narrative as with Literate Programming, but documentation is generated from code, not vice versa, and there is no support for chunking.

Code is first, code must look pretty, documentation is code.

Docstrings

PAX automatically recognizes and marks up code with backticks and links names in code to their definitions. Take, for instance, SBCL's abort function, whose docstring is written in the usual style, uppercasing names of symbols:

(docstring #'abort)
=> "Transfer control to a restart named ABORT, signalling a CONTROL-ERROR if
   none exists."

Note how in the generated documentation, abort is set with a monospace font, while control-error is Autolinked:

The following transcript shows the raw Markdown for the previous example.

(document #'abort :format :markdown)
.. - [function] **ABORT** *&OPTIONAL CONDITION*
..
..     Transfer control to a restart named `ABORT`, signalling a [`CONTROL-ERROR`][7c2c] if
..     none exists.
..
..   [7c2c]: http://www.lispworks.com/documentation/HyperSpec/Body/e_contro.htm "CONTROL-ERROR (MGL-PAX:CLHS CONDITION)"
..
A Complete Example

Here is an example of how it all works together:

(mgl-pax:define-package :foo-random
  (:documentation "This package provides various utilities for random.
  See FOO-RANDOM:@FOO-RANDOM-MANUAL.")
  (:use #:common-lisp #:mgl-pax))

(in-package :foo-random)

(defsection @foo-random-manual (:title "Foo Random manual")
  "FOO-RANDOM is a random number generator library inspired by CL:RANDOM.
  Functions such as UNIFORM-RANDOM use *FOO-STATE* and have a
  :RANDOM-STATE keyword arg."
  (foo-random-state class)
  (state (reader foo-random-state))
  "Hey we can also print states!"
  (print-object (method () (foo-random-state t)))
  (*foo-state* variable)
  (gaussian-random function)
  (uniform-random function)
  ;; This is a subsection.
  (@foo-random-examples section))

(defclass foo-random-state ()
  ((state :reader state)))

(defmethod print-object ((object foo-random-state) stream)
  (print-unreadable-object (object stream :type t)))

(defvar *foo-state* (make-instance 'foo-random-state)
  "Much like *RANDOM-STATE* but uses the FOO algorithm.")

(defun uniform-random (limit &key (random-state *foo-state*))
  "Return a random number from the between 0 and LIMIT (exclusive)
  uniform distribution."
  nil)

(defun gaussian-random (stddev &key (random-state *foo-state*))
  "Return a random number from a zero mean normal distribution with
  STDDEV."
  nil)

(defsection @foo-random-examples (:title "Examples")
  "Let's see the transcript of a real session of someone working
  with FOO:

  ```cl-transcript
  (values (princ :hello) (list 1 2))
  .. HELLO
  => :HELLO
  => (1 2)

  (make-instance 'foo-random-state)
  ==> #<FOO-RANDOM-STATE >
  ```")

Note how (variable *foo-state*) in the defsection form both exports *foo-state* and includes its documentation in @foo-random-manual. The symbols variable and function are just two instances of locatives, which are used in defsection to refer to definitions tied to symbols.

(document @foo-random-manual) generates fancy Markdown or HTML output with automatic markup and Autolinks uppercase words found in docstrings, numbers sections, and creates a table of contents.

One can even generate documentation for different but related libraries at the same time with the output going to different files but with cross-page links being automatically added for symbols mentioned in docstrings. In fact, this is what PAX World does. See Generating Documentation for some convenience functions to cover the most common cases.

The transcript in the code block tagged with cl-transcript is automatically checked for up-to-dateness when documentation is generated.

λ

2 Emacs Setup

Here is a quick recipe for setting up PAX for use via SLIME to take advantage of the conveniences on offer. Conversely, there is no need to do any of this just to use defsection, write docstrings and for Generating Documentation.

If PAX was installed from Quicklisp, then evaluate this in CL to copy the Elisp code to a stable location:

(mgl-pax:install-pax-elisp "~/quicklisp/")

Assuming the Elisp file is in the ~/quicklisp/ directory, add something like this to your .emacs:

(add-to-list 'load-path "~/quicklisp/")
(require 'mgl-pax)
(global-set-key (kbd "C-.") 'mgl-pax-document)
(global-set-key (kbd "s-x t") 'mgl-pax-transcribe-last-expression)
(global-set-key (kbd "s-x r") 'mgl-pax-retranscribe-region)

λ

2.1 Functionality Provided

λ

2.2 Installing from Quicklisp

If you installed PAX with Quicklisp, the location of mgl-pax.el may change with updates, and you may want to copy the current version of mgl-pax.el to a stable location by evaluating this in CL:

(mgl-pax:install-pax-elisp "~/quicklisp/")

If working from, say, a git checkout, there is no need for this step.

λ

2.3 Loading PAX

Assuming the Elisp file is in the ~/quicklisp/ directory, add something like this to your .emacs:

(add-to-list 'load-path "~/quicklisp/")
(require 'mgl-pax)

If the Lisp variable mgl-pax-autoload is true (the default), then MGL-PAX will be loaded in the connected Lisp on-demand via SLIME.

If loading fails, mgl-pax will be unloaded from Emacs and any overridden Slime key bindings restored.

λ

2.4 Setting up Keys

The recommended key bindings are this:

(global-set-key (kbd "C-.") 'mgl-pax-document)
(global-set-key (kbd "s-x t") 'mgl-pax-transcribe-last-expression)
(global-set-key (kbd "s-x r") 'mgl-pax-retranscribe-region)

The global key bindings above are global because their commands work in any mode. If that's not desired, one may bind C-. locally in all Slime related modes like this:

(slime-bind-keys slime-parent-map nil '(("C-." mgl-pax-document)))

If the customizable variable mgl-pax-hijack-slime-doc-keys is true, then upon loading mgl-pax', the following changes are made to slime-doc-map(assuming it's bound toC-c C-d`):

Calling mgl-pax-unhijack-slime-doc-keys reverts these changes.

λ

3 Links and Systems

Here is the official repository and the HTML documentation for the latest version.

PAX is built on top of the DRef library (bundled in the same repository).

λ

4 Background

As a user, I frequently run into documentation that's incomplete and out of date, so I tend to stay in the editor and explore the code by jumping around with SLIME's M-. (slime-edit-definition). As a library author, I spend a great deal of time polishing code but precious little writing documentation.

In fact, I rarely write anything more comprehensive than docstrings for exported stuff. Writing docstrings feels easier than writing a separate user manual, and they are always close at hand during development. The drawback of this style is that users of the library have to piece the big picture together themselves.

That's easy to solve, I thought, let's just put all the narrative that holds docstrings together in the code and be a bit like a Literate Programmer turned inside out. The original prototype, which did almost everything I wanted, was this:

(defmacro defsection (name docstring)
  `(defun ,name () ,docstring))

Armed with this defsection, I soon found myself organizing code following the flow of user-level documentation and relegated comments to implementation details entirely. However, some parts of defsection docstrings were just listings of all the functions, macros and variables related to the narrative, and this list was repeated in the defpackage form complete with little comments that were like section names. A clear violation of OAOO, one of them had to go, so defsection got a list of symbols to export.

That was great, but soon I found that the listing of symbols is ambiguous if, for example, a function, a compiler macro and a class were named by the same symbol. This did not concern exporting, of course, but it didn't help readability. Distractingly, on such symbols, M-. was popping up selection dialogs. There were two birds to kill, and the symbol got accompanied by a type, which was later generalized into the concept of locatives:

(defsection @introduction ()
  "A single line for one man ..."
  (foo class)
  (bar function))

After a bit of elisp hacking, M-. was smart enough to disambiguate based on the locative found in the vicinity of the symbol, and everything was good for a while.

Then, I realized that sections could refer to other sections if there were a section locative. Going down that path, I soon began to feel the urge to generate pretty documentation as all the necessary information was available in the defsection forms. The design constraint imposed on documentation generation was that following the typical style of upcasing symbols in docstrings, there should be no need to explicitly mark up links: if M-. works, then the documentation generator shall also be able figure out what's being referred to.

I settled on Markdown as a reasonably non-intrusive format, and a few thousand lines later PAX was born. Since then, locatives and references were factored out into the DRef library to let PAX focus on M-. and documentation.

λ

5 Basics

Now let's examine the most important pieces.

λ

6 Parsing

When encountering a word such as CLASSes, PAX needs to find the name in it that makes sense in the context. Codification, for example, looks for interesting names, Navigating Sources in Emacs for names with Lisp definitions, and Linking for names with any kind of definition.

This is not as straightforward as it sounds because it needs to handle cases like nonreadable, classes, all the various forms of Linking in docstrings as well as in comments, and the (name locative) syntax in defsection.

Depending on the context, trimming and depluralization may be enabled (see Raw Names in Words), while the possible names may be restricted to symbols (see Names in Raw Names).

For a word, a number of raw names is generated by trimming delimiter characters and plural markers, and for each raw name a number of names are considered until one is found suitable in the context. The following subsections describe the details of the parsing algorithm.

λ

6.1 Raw Names in Words

From words, raw names are parsed by trimming some prefixes and suffixes. For a given word, multiple raw names are considered in the following order.

  1. The entire word.

  2. Trimming the following characters from the left of the word:

    #<{;"'`
    
  3. Trimming the following characters from the right of the word:

    ,;:.>}"'`
    
  4. Trimming both of the previous two at the same time.

  5. From the result of 4., If a word ends with what looks like a plural marker (case-insensitive), then a name is created by removing it. For example, from the word buses the plural marker es is removed to produce the name bus. The list of plural markers considered is ses (e.g. gasses), es (e.g. buses), s (e.g. cars), zes (e.g. fezzes), and ren (e.g. children).

  6. From the result of 4., removing the prefix before the first, and the suffix after the last uppercase character if it contains at least one lowercase character.

λ

6.2 Names in Raw Names

For each raw name from Raw Names in Words, various names may be considered until one is found suitable in the context.

The following examples list the names considered for a given raw name, assuming that readtable-case is :upcase as well as that foo and |Foo| are interned.

The rules are:

  1. If the raw name is not mixed case (i.e. it doesn't have both upper- and lowercase characters) and it names an interned symbol (subject to the current Package and Readtable), then that symbol is considered as a name.

  2. The raw name itself (a string) is considered a name.

  3. The raw name upcased or downcased according to readtable-case (subject to the current readtable) but still as a string. This is to allow [dref][package] to refer to the "DREF" package regardless of whether the symbol dref is interned in the current package.

  4. If the raw name is explicitly a symbol (it starts with #\|), and it names an interned symbol (subject to the current Package and Readtable), then that symbol is considered as a name and nothing else.

  5. If the raw name has an embedded string (it starts with #\") and read-from-string can read the embedded string from it, then that string is considered as a name and nothing else.


For example, when M-. is pressed while point is over nonREADable., the last word of the sentence It may be nonREADable., the following raw names are considered until one is found with a definition:

  1. The entire word, "nonREADable.".

  2. Trimming left does not produce a new raw name.

  3. Trimming right removes the dot and gives "nonREADable".

  4. Trimming both is the same as trimming right.

  5. No plural markers are found.

  6. The lowercase prefix and suffix is removed around the uppercase core, giving "READ". This names an interned symbol which has a definition, so M-. will visit it.

When Generating Documentation, Autolinking behaves similarly.

λ

7 PAX Locatives

To the Basic Locative Types defined by DRef, PAX adds a few of its own.

λ

8 Navigating Sources in Emacs

Integration into SLIME's M-. (slime-edit-definition) allows one to visit the source-location of a definition. PAX extends standard Slime functionality by

The definition is either determined from the buffer content at point or is prompted for. At the prompt, TAB-completion is available for both names and locatives. With a prefix argument (C-u M-.), the buffer contents are not consulted, and M-. always prompts.

The M-. extensions can be enabled by loading src/mgl-pax.el. See Emacs Setup. In addition, the Elisp command mgl-pax-edit-parent-section visits the source location of the section containing the definition with point in it.

A close relative of M-. is C-. for Browsing Live Documentation.

λ

8.1 M-. Defaulting

When M-. is invoked, it first tries to find a name in the current buffer at point. If no name is found, then it prompts.

First, (slime-sexp-at-point) is taken as a word, from which the name will be parsed. Then, candidate locatives are looked for before and after the word. Thus, if a locative is the previous or the next expression, then M-. will go straight to the definition which corresponds to the locative. If that fails, M-. will try to find the definitions in the normal way, which may involve popping up an xref buffer and letting the user interactively select one of possible definitions.

M-. works on parenthesized references, such as those in defsection:

(defsection @foo ()
  (cos function))

Here, when the cursor is on one of the characters of cos or just after cos, pressing M-. will visit the definition of the function cos.

To play nice with Generating Documentation, forms suitable for Autolinking are recognized:

function cos
cos function

... as well as Reflinks:

[cos][function]
[see this][cos function]

... and Markdown inline code:

cos `function`
`cos` function
`cos` `function`

Everything works the same way in comments and docstrings as in code. In the next example, pressing M-. on resolve* will visit its denoted method:

;;; See RESOLVE* (method () (function-dref)) for how this all works.

λ

8.2 M-. Prompting

λ

8.2.1 M-. Minibuffer Syntax

At the minibuffer prompt, the definitions to edit can be specified as follows.

In all of the above name is a raw name, meaning that print will be recognized as print and pax as "PAX".

The package in which symbols are read is the Elisp slime-current-package. In Lisp buffers, this is the buffer's package, else it's the package of the Slime repl buffer.

λ

8.2.2 M-. Completion

When M-. prompts for the definition to edit, TAB-completion is available in the minibuffer for both names and locatives. To reduce clutter, string names are completed only if they are typed explicitly with an opening quotation mark, and they are case-sensitive. Examples:

For more powerful search, see Apropos.

λ

9 Generating Documentation

λ

9.1 The document Function

λ

9.1.1 documentable

The documentable argument of document may be a single object (e.g. #'print'), a definition such as (dref 'print 'function), a string, or a nested list of these. More precisely, documentable is one of the following:

λ

9.1.2 Return Values

If pages are nil, then document - like cl:format - returns a string (when stream is nil) else nil.

If pages, then a list of output designators are returned, one for each non-empty page (to which some output has been written), which are determined as follows.

If the default page given by the stream argument of document was written to, then its output designator is the first element of the returned list. The rest of the designators correspond to the non-empty pages in the pages argument of document in that order.

λ

9.1.3 pages

The pages argument of document is to create multi-page documents by routing some of the generated output to files, strings or streams. pages is a list of page specification elements. A page spec is a property list with keys :objects, :output, :uri-fragment, :source-uri-fn, :header-fn and :footer-fn. objects is a list of objects (references are allowed but not required) whose documentation is to be sent to :output.

pages may look something like this:

`((;; The section about SECTIONs and everything below it ...
   :objects (, @sections)
   ;; ... is so boring that it's not worth the disk space, so
   ;; send it to a string.
   :output (nil)
   ;; Explicitly tell other pages not to link to these guys.
   :uri-fragment nil)
  ;; Send the @EXTENSION-API section and everything reachable
  ;; from it ...
  (:objects (, @extension-api)
   ;; ... to build/tmp/pax-extension-api.html.
   :output "build/tmp/pax-extension-api.html"
   ;; However, on the web server html files will be at this
   ;; location relative to some common root, so override the
   ;; default:
   :uri-fragment "doc/dev/pax-extension-api.html"
   ;; Set html page title, stylesheet, charset.
   :header-fn 'write-html-header
   ;; Just close the body.
   :footer-fn 'write-html-footer)
  ;; Catch references that were not reachable from the above. It
  ;; is important for this page spec to be last.
  (:objects (, @pax-manual)
   :output "build/tmp/manual.html"
   ;; Links from the extension api page to the manual page will
   ;; be to ../user/pax-manual#<anchor>, while links going to
   ;; the opposite direction will be to
   ;; ../dev/pax-extension-api.html#<anchor>.
   :uri-fragment "doc/user/pax-manual.html"
   :header-fn 'write-html-header
   :footer-fn 'write-html-footer))

Documentation is initially sent to a default stream (the stream argument of document), but output is redirected if the thing being currently documented is the :object of a page-spec.

λ

9.1.4 Package and Readtable

While generating documentation, symbols may be read (e.g. from docstrings) and printed. What values of *package* and *readtable* are used is determined separately for each definition being documented.

The values thus determined come into effect after the name itself is printed, for printing of the arglist and the docstring.

CL-USER> (pax:document #'foo)
- [function] FOO <!> X Y &KEY (ERRORP T)

    Do something with X and Y.

In the above, the <!> marks the place where *package* and *readtable* are bound.

λ

Home Section

The home section of an object is a section that contains the object's definition in its section-entries or nil. In the overwhelming majority of cases there should be at most one containing section.

If there are multiple containing sections, the following apply.

For example, (mgl-pax:document function) is an entry in the MGL-PAX::@BASICS section. Unless another section that contains it is defined in the MGL-PAX package, the home section is guaranteed to be MGL-PAX::@BASICS because the symbol-packages of mgl-pax:document and MGL-PAX::@BASICS are the same (hence their common prefix is maximally long).

This scheme would also work, for example, if the home package of document were mgl-pax/impl, and it were reexported from mgl-pax because the only way to externally change the home package would be to define a containing section in a package like mgl-pax/imp.

Thus, relying on the package system makes it possible to find the intended home section of a definition among multiple containing sections with high probability. However, for names which are not symbols, there is no package system to advantage of.

λ

9.2 Browsing Live Documentation

Documentation for definitions in the running Lisp can be browsed directly without generating documentation in the offline manner. HTML documentation, complete with Codification and Linking, is generated from docstrings of all kinds of definitions and PAX sections in the running Lisp on the fly. This allows ad-hoc exploration of the Lisp, much like describe-function, apropos-command and other online help commands in Emacs, for which direct parallels are provided.

Still, even without Emacs and SLIME, limited functionality can be accessed through PAX Live Home Page by starting the live documentation web server manually.

If Emacs Setup has been done, the Elisp function mgl-pax-document (maybe bound to C-.) generates and displays documentation as a single HTML page. If necessary, a disambiguation page is generated with the documentation of all matching definitions. For example, to view the documentation of this very section, one can do:

M-x mgl-pax-document
View Documentation of: pax::@browsing-live-documentation

Alternatively, pressing C-. with point over the text pax::@browsing-live-documentation in a buffer achieves the same effect.

In interactive use, mgl-pax-document behaves similarly to M-. except:

The convenience function mgl-pax-current-definition-toggle-view (C-c C-d c) documents the definition with point in it.

λ

9.2.1 Browsing with w3m

When the value of the Elisp variable mgl-pax-browser-function is w3m-browse-url (see Emacs Setup), the Emacs w3m browser is used without the need for a web server, and also offering somewhat tighter integration than Browsing with Other Browsers.

With w3m's default key bindings, moving the cursor between links involves tab and s-tab (or <up> and <down>). ret and <right> follow a link, while b and <left> go back in history.

In addition, the following PAX-specific key bindings are available:

λ

9.2.2 Browsing with Other Browsers

When the value of the Elisp variable mgl-pax-browser-function is not w3m-browse-url (see Emacs Setup), requests are served via a web server started in the running Lisp, and documentation is most likely displayed in a separate browser window.

By default, mgl-pax-browser-function is nil, which makes PAX use browse-url-browser-function. You may want to customize the related browse-url-new-window-flag or, for Chrome, set browse-url-chrome-arguments to ("--new-window").

By default, mgl-pax-web-server-port is nil, and PAX will pick a free port automatically.

In the browser, clicking on the locative on the left of the name (e.g. in - [function] PRINT) will raise and focus the Emacs window (if Emacs is not in text mode, and also subject to window manager focus stealing settings), then go to the corresponding source location. For sections, clicking on the lambda link will do the same (see *document-fancy-html-navigation*).

Finally, note that the urls exposed by the web server are subject to change, and even the port used may vary by session if the Elisp variable mgl-pax-web-server-port is nil.

λ

9.2.3 Apropos

The Elisp functions mgl-pax-apropos, mgl-pax-apropos-all, and mgl-pax-apropos-package can display the results of dref-apropos in the live documentation browser. These extend the functionality of slime-apropos, slime-apropos-all and slime-apropos-package to support more kinds of definitions in an extensible way. The correspondence is so close that the PAX versions might take over the Slime key bindings.

Note that apropos functionality is also exposed via the PAX Live Home Page.

More concretely, the PAX versions supports the following extensions:

On the PAX Live Home Page, one may Browse by Locative Types, which gives access to some of the apropos functionality via the browser without involving Emacs.

On the result page:

With mgl-pax-apropos-all and mgl-pax-apropos-package being simple convenience functions on top of mgl-pax-apropos, we only discuss the latter in detail here. For the others, see the Elisp docstrings.

λ

The string Argument of mgl-pax-apropos

The string argument consists of a name pattern and a dtype.

The name pattern has the following forms.

After the name pattern, string may contain a dtype that the definitions must match.

λ

The package Argument of mgl-pax-apropos

When mgl-pax-apropos is invoked with a prefix argument, it prompts for a package pattern among other things. The pattern may be like the following examples.

The above examples assume case-insensitive matching.

λ

9.2.4 PAX Live Home Page

When Browsing Live Documentation, the home page provides quick access to documentation of the definitions in the system. In Emacs, when mgl-pax-document is invoked with the empty string, it visits the home page.

The home page may also be accessed directly by going to the root page of the web server (if one is started). Here, unless the home page is viewed with w3m, one may directly look up documentation and access Apropos via the input boxes provided.

λ

Top-level PAX Sections

The PAX Live Home Page lists the top-level PAX sections: those that have no other sections referencing them (see defsection).

λ

asdf:systems and Related packages

The PAX Live Home Page lists all asdf:systems and packages in the Lisp. For easier overview, the they are grouped based on their source-locations. Two systems are in the same group if the directory of one (i.e. the directory of the .asd file in which it was defined) is the same or is below the other's.

A package presented under a group of systems, if the source-location of the package is below the the top-most directory among the systems in the group.

λ

Systemless Packages

The PAX Live Home Page lists packages unrelated to any asdf:system as systemless.

λ

Browse by Locative Types

The PAX Live Home Page provides quick links to Apropos result pages for all Basic Locative Types which may have definitions.

λ

9.3 Markdown Support

The Markdown in docstrings is processed with the 3BMD library.

λ

9.3.1 Markdown in Docstrings

Docstrings of definitions which do not have a Home Section and are not sections themselves are assumed to have been written with no knowledge of PAX and to conform to markdown only by accident. These docstrings are thus sanitized more aggressively.

λ

9.3.2 Syntax Highlighting

For syntax highlighting, GitHub's fenced code blocks markdown extension to mark up code blocks with triple backticks is enabled so all you need to do is write:

```elisp
(defun foo ())
```

to get syntactically marked up HTML output. Copy src/style.css from PAX and you are set. The language tag, elisp in this example, is optional and defaults to common-lisp.

See the documentation of 3BMD and Colorize for the details.

λ

9.3.3 MathJax

Displaying pretty mathematics in TeX format is supported via MathJax. It can be done inline with $ like this:

$\int_0^\infty e^{-x^2} dx=\frac{\sqrt{\pi}}{2}$

which is displayed as $\int_0^\infty e^{-x^2} dx=\frac{\sqrt{\pi}}{2}$, or it can be delimited by $$ like this:

$$\int_0^\infty e^{-x^2} dx=\frac{\sqrt{\pi}}{2}$$

to get: $$\int_0^\infty e^{-x^2} dx=\frac{\sqrt{\pi}}{2}$$

MathJax will leave code blocks (including those inline with backticks) alone. Outside code blocks, escape $ by prefixing it with a backslash to scare MathJax off.

Escaping all those backslashes in TeX fragments embedded in Lisp strings can be a pain. Pythonic String Reader can help with that.

λ

9.4 Codification

λ

9.5 Linking

PAX supports linking to definitions either with explicit Reflinks or with Autolinks.

When generating offline documentation, only the definitions in documentable may be linkable, but when Browsing Live Documentation, everything is linkable as documentation is generated on-demand.

Many examples in this section link to standard Common Lisp definitions. In the offline case, these will link to external URLs, while in the live case to disambiguation pages that list the definition in the running Lisp and in the HyperSpec.

Invoking M-. on word or name in any of the following examples will disambiguate based on the textual context, determining the locative. This is because navigation and linking use the same Parsing algorithm, although linking is a bit more strict about trimming, depluralization, and it performs Filtering Links. On the other hand, M-. cannot visit the clhs references because there are no associated source locations.

λ

9.5.1 Reflink

The Markdown reference link syntax [label][id] is repurposed for linking to definitions. In the following, we discuss the various forms of reflinks.

λ

Specific Reflink

Format: [word][locative]

The first name in word (with depluralization) that forms a valid dref with locative is determined, and that definition is linked to. If there is no such dref, then an unresolvable-reflink warning is signalled.

Examples:

The Markdown link definition (i.e. type above) needs no backticks to mark it as code, but here and below, the second example relies on *document-uppercase-is-code* being true.

λ

Specific Reflink with Text

Format: [link text][name locative]

If name and locative form a valid dref, then that definition is linked to with link text link text. If there is no such dref, then an unresolvable-reflink warning is signalled.

In this form, if name starts with #\", then it's read as a string, else as a symbol.

Examples:

λ

Unspecific Reflink

Format: [word][]

The first name in word (with depluralization, symbols only) that has some definitions is determined, and those definitions are linked to. If no name with any definition is found, then an unresolvable-reflink warning is signalled.

Examples:

λ

Unspecific Reflink with Text

Format: [link text][name]

The definitions of name are determined, and those definitions are linked to. If name has no definitions, then an unresolvable-reflink warning is signalled.

In this form, if name starts with #\", then it's read as a string, else as as symbol.

Examples:

λ

Markdown Reflink

Format: [label][id]

This is a normal Markdown reference link if id is not a valid locative.

Use urls with define-glossary-term as a better alternative to Markdown reference links (see Markdown in Docstrings).

λ

Unresolvable Links

λ

9.5.2 Autolink

Markdown inline code automatically links to the corresponding definitions without having to use Reflinks. This works especially well in conjunction with Codification. The following examples assume that *document-uppercase-is-code* is true. If that's not the case, explicit backticks are required on word (but not on locative).

λ

Specific Autolink

Format: word locative or locative word

The first name in word (with depluralization) that forms a valid dref with locative is determined, and that definition is linked to. If no such name is found, then Unspecific Autolink is attempted.

Examples:

If locative has spaces, then it needs to be marked up as code, too. For example,

DREF-NAME `(reader dref)`

renders as dref-name (reader dref).

λ

Unspecific Autolink

Format: word

The first name in word (with depluralization, symbols only) that has some definitions is determined, and those definitions are linked to. If no such name is found or the autolink to this name is suppressed (see below), then word is left unchanged. If a locative is found before or after word, then Specific Autolink is tried first.

Examples:

Unspecific Autolinking is suppressed if the name found has a Local Definition or was linked to before in the same docstring:

Unspecific Autolinking to t and nil is also suppressed (see *document-link-to-hyperspec*):

As an exception, a single link (be it either a Specific Link or an unambiguous Unspecific Link) to a section or glossary-term is not suppressed to allow their titles to be displayed properly:

λ

Escaping Autolinking

In the common case, when *document-uppercase-is-code* is true, prefixing an uppercase word with a backslash prevents it from being codified and thus also prevents Autolinking form kicking in. For example,

\DOCUMENT

renders as DOCUMENT. If it should be marked up as code but not autolinked, the backslash must be within backticks like this:

`\DOCUMENT`

This renders as document. Alternatively, the dislocated or the argument locative may be used as in [DOCUMENT][dislocated].

λ

9.5.3 Linking to the HyperSpec

λ

9.5.4 Linking to Sections

The following variables control how to generate section numbering, table of contents and navigation links.

λ

9.5.5 Filtering Links

λ

Specific Link

Specific links are those Reflinks and Autolinks that have a single locative and therefore at most a single matching definition. These are Specific Reflink, Specific Reflink with Text and Specific Autolink.

A specific link to a linkable definition produces a link in the output. If the definition is not linkable, then the output will contain only what would otherwise be the link text.

λ

Unspecific Link

Unspecific links are those Reflinks and Autolinks that do not specify a locative and match all definitions with a name. These are Unspecific Reflink, Unspecific Reflink with Text and Unspecific Autolink.

To make the links predictable and manageable in number, the following steps are taken.

  1. Definitions that are not symbol-based (i.e. whose dref-name is not a symbol) are filtered out to prevent unrelated packages, asdf:systems and clhs sections from cluttering the documentation without the control provided by importing symbols.

  2. All references with locative-type locative are filtered out.

  3. Non-linkable definitions are removed.

  4. If the definitions include a generic-function, then all definitions with locative-type method, accessor, reader and writer are removed to avoid linking to a possibly large number of methods.

If at most a single definition remains, then the output is the same as with a Specific Link. If multiple definitions remain, then the link text is output followed by a number of numbered links, one to each definition.

λ

9.5.6 Link Format

The following variables control various aspects of links and urls.

λ

9.6 Local Definition

While documentation is generated for a definition, that definition is considered local. Other local definitions may also be established. Local definitions inform Codification through interesting names and affect Unspecific Autolinking.

(defun foo (x)
  "FOO adds one to X."
  (1+ x)

In this example, while the docstring of foo is being processed, the global definition (dref 'foo 'function) is also considered local, which suppresses linking foo in the foo's docstring back to its definition. If foo has other definitions, Unspecific Autolinking to those is also suppressed.

Furthermore, the purely local definition (dref 'x 'argument) is established, causing the argument name x to be codified because x is now interesting.

See documenting-reference and with-dislocated-names in Extending document.

λ

9.7 Overview of Escaping

Let's recap how escaping Codification, downcasing, and Linking works.

In the following examples capital C/D/A letters mark the presence, and a/b/c the absence of codification, downcasing, and autolinking assuming all these features are enabled by *document-uppercase-is-code*, *document-downcase-uppercase-code*, and *document-link-code*.

DOCUMENT                => [`document`][1234]    (CDA)
\DOCUMENT               => DOCUMENT              (cda)
`\DOCUMENT`             => `document`            (CDa)
`\\DOCUMENT`            => `DOCUMENT`            (CdA)
[DOCUMENT][]            => [`document`][1234]    (CDA)
[\DOCUMENT][]           => [DOCUMENT][1234]      (cdA)
[`\DOCUMENT`][]         => [`document`][1234]    (CDA) *
[`\\DOCUMENT`][]        => [`DOCUMENT`][1234]    (CdA)
[DOCUMENT][dislocated]  => `document`            (CDa)

Note that in the example marked with *, the single backslash, that would normally turn autolinking off, is ignored because it is in an explicit link.

λ

9.8 Output Details

By default, drefs are documented in the following format.

- [<locative-type>] <name> <arglist>

    <docstring>

The line with the bullet is printed with documenting-reference. The docstring is processed with document-docstring while Local Definitions established with with-dislocated-names are in effect for all variables locally bound in a definition with arglist, and *package* is bound to the second return value of docstring.

With this default format, PAX supports all locative types, but for some Basic Locative Types defined in DRef and the PAX Locatives, special provisions have been made.

When the printed initform is too long, it is truncated.

λ

9.9 Documentation Generation Implementation Notes

Documentation Generation is supported on ABCL, AllegroCL, CLISP, CCL, CMUCL, ECL and SBCL, but their outputs may differ due to the lack of some introspective capability. SBCL generates complete output. see arglist, docstring and source-location for implementation notes.

In addition, CLISP does not support the ambiguous case of Browsing Live Documentation because the current implementation relies on Swank to list definitions of symbols (as variable, function, etc), and that simply doesn't work.

λ

9.10 Utilities for Generating Documentation

Two convenience functions are provided to serve the common case of having an ASDF system with some readmes and a directory with for the HTML documentation and the default CSS stylesheet.

λ

9.10.1 HTML Output

See the following variables, which control HTML generation.

λ

9.10.2 GitHub Workflow

It is generally recommended to commit generated readmes (see update-asdf-system-readmes), so that users have something to read without reading the code and sites like GitHub can display them.

HTML documentation can also be committed, but there is an issue with that: when linking to the sources (see make-git-source-uri-fn), the commit id is in the link. This means that code changes need to be committed first, and only then can HTML documentation be regenerated and committed in a followup commit.

The second issue is that GitHub is not very good at serving HTML files from the repository itself (and http://htmlpreview.github.io chokes on links to the sources).

The recommended workflow is to use gh-pages, which can be made relatively painless with the git worktree command. The gist of it is to make the doc/ directory a checkout of the branch named gh-pages. There is a good description of this general process. Two commits are needed still, but it is somewhat less painful.

This way the HTML documentation will be available at

http://<username>.github.io/<repo-name>

It is probably a good idea to add sections like the Links and Systems section to allow jumping between the repository and the gh-pages site.

λ

9.10.3 PAX World

PAX World is a registry of documents, which can generate cross-linked HTML documentation pages for all the registered documents. There is an official PAX World.

For example, this is how PAX registers itself:

(defun pax-sections ()
  (list @pax-manual))

(defun pax-pages ()
  `((:objects ,(pax-sections)
     :source-uri-fn ,(make-git-source-uri-fn
                      :mgl-pax
                      "https://github.com/melisgl/mgl-pax"))))

(register-doc-in-pax-world :pax 'pax-sections 'pax-pages)

λ

10 Transcripts

What are transcripts for? When writing a tutorial, one often wants to include a REPL session with maybe a few defuns and a couple of forms whose output or return values are shown. Also, in a function's docstring an example call with concrete arguments and return values speaks volumes. A transcript is a text that looks like a REPL session, but which has a light markup for printed output and return values, while no markup (i.e. prompt) for Lisp forms. PAX transcripts may include output and return values of all forms, or only selected ones. In either case, the transcript itself can be easily generated from the source code.

The main worry associated with including examples in the documentation is that they tend to get out-of-sync with the code. This is solved by being able to parse back and update transcripts. In fact, this is exactly what happens during documentation generation with PAX. Code sections tagged with cl-transcript are retranscribed and checked for consistency (that is, no difference in output or return values). If the consistency check fails, an error is signalled that includes a reference to the object being documented.

Going beyond documentation, transcript consistency checks can be used for writing simple tests in a very readable form. For example:

(+ 1 2)
=> 3

(values (princ :hello) (list 1 2))
.. HELLO
=> :HELLO
=> (1 2)

All in all, transcripts are a handy tool especially when combined with the Emacs support to regenerate them and with PYTHONIC-STRING-READER's triple-quoted strings, that allow one to work with nested strings with less noise. The triple-quote syntax can be enabled with:

(in-readtable pythonic-string-syntax)

λ

10.1 Transcribing with Emacs

Typical transcript usage from within Emacs is simple: add a Lisp form to a docstring or comment at any indentation level. Move the cursor right after the end of the form as if you were to evaluate it with C-x C-e. The cursor is marked by #\^:

This is part of a docstring.

```cl-transcript
(values (princ :hello) (list 1 2))^
```

Note that the use of fenced code blocks with the language tag cl-transcript is only to tell PAX to perform consistency checks at documentation generation time.

Now invoke the Elisp function mgl-pax-transcribe where the cursor is, and the fenced code block from the docstring becomes:

(values (princ :hello) (list 1 2))
.. HELLO
=> :HELLO
=> (1 2)
^

Then you change the printed message and add a comment to the second return value:

(values (princ :hello-world) (list 1 2))
.. HELLO
=> :HELLO
=> (1
    ;; This value is arbitrary.
    2)

When generating the documentation you get a transcription-consistency-error because the printed output and the first return value changed, so you regenerate the documentation by marking the region bounded by #\| and the cursor at #\^ in the example:

|(values (princ :hello-world) (list 1 2))
.. HELLO
=> :HELLO
=> (1
    ;; This value is arbitrary.
    2)
^

then invoke the Elisp function mgl-pax-retranscribe-region to get:

(values (princ :hello-world) (list 1 2))
.. HELLO-WORLD
=> :HELLO-WORLD
=> (1
    ;; This value is arbitrary.
    2)
^

Note how the indentation and the comment of (1 2) were left alone, but the output and the first return value got updated.

Alternatively, C-u 1 mgl-pax-transcribe will emit commented markup:

(values (princ :hello) (list 1 2))
;.. HELLO
;=> :HELLO
;=> (1 2)

C-u 0 mgl-pax-retranscribe-region will turn commented into non-commented markup. In general, the numeric prefix argument is the index of the syntax to be used in *transcribe-syntaxes*. Without a prefix argument, mgl-pax-retranscribe-region will not change the markup style.

Finally, not only do both functions work at any indentation level but in comments too:

;;;; (values (princ :hello) (list 1 2))
;;;; .. HELLO
;;;; => :HELLO
;;;; => (1 2)

The dynamic environment of the transcription is determined by the :dynenv argument of the enclosing cl-transcript code block (see Controlling the Dynamic Environment).

Transcription support in Emacs can be enabled by loading src/mgl-pax.el. See Emacs Setup.

λ

10.2 Transcript API

λ

10.3 Transcript Consistency Checking

The main use case for consistency checking is detecting out-of-date examples in documentation, although using it for writing tests is also a possibility. Here, we focus on the former.

When a markdown code block tagged cl-transcript is processed during Generating Documentation, the code in it is replaced with the output of with (transcribe <code> nil :update-only t :check-consistency t). Suppose we have the following example of the function greet, that prints hello and returns 7.

```cl-transcript
(greet)
.. hello
=> 7
```

Now, if we change greet to print or return something else, a transcription-consistency-error will be signalled during documentation generation. Then we may fix the documentation or continue from the error.

By default, comparisons of previous to current output, readable and unreadable return values are performed with string=, equal, and string=, respectively, which is great in the simple case. Non-determinism aside, exact matching becomes brittle as soon as the notoriously unportable pretty printer is used or when unreadable objects are printed with their #<> syntax, especially when print-unreadable-object is used with :identity t.

λ

10.3.1 Finer-Grained Consistency Checks

To get around this problem, consistency checking of output, readable and unreadable values can be customized individually by supplying transcribe with a check-consistency argument like ((:output <output-check>) (:readable <readable-check>) (:unreadable <unreadable-check>)). In this case, <output-check> may be nil, t, or a function designator.

The case of <readable-check> and <unreadable-check> is similar.

Code blocks tagged cl-transcript can take arguments, which they pass on to transcribe. The following shows how to check only the output.

```cl-transcript (:check-consistency ((:output t)))
(error "Oh, no.")
.. debugger invoked on SIMPLE-ERROR:
..   Oh, no.

(make-condition 'simple-error)
==> #<SIMPLE-ERROR {1008A81533}>
```

λ

10.3.2 Controlling the Dynamic Environment

The dynamic environment in which forms in the transcript are evaluated can be controlled via the :dynenv argument of cl-transcript.

```cl-transcript (:dynenv my-transcript)
...
```

In this case, instead of calling transcribe directly, the call will be wrapped in a function of no arguments and passed to the function my-transcript, which establishes the desired dynamic environment and calls its argument. The following definition of my-transcript simply packages up oft-used settings to transcribe.

(defun my-transcript (fn)
  (let ((*transcribe-check-consistency*
          '((:output my-transcript-output=)
            (:readable equal)
            (:unreadable nil))))
    (funcall fn)))

(defun my-transcript-output= (string1 string2)
  (string= (my-transcript-normalize-output string1)
           (my-transcript-normalize-output string2)))

(defun my-transcript-normalize-output (string)
  (squeeze-whitespace (delete-trailing-whitespace (delete-comments string))))

A more involved solution could rebind global variables set in transcripts, unintern symbols created or even create a temporary package for evaluation.

λ

10.3.3 Utilities for Consistency Checking

λ

11 Writing Extensions

λ

11.1 Adding New Locatives

Once everything in Extending DRef has been done, there are only a couple of PAX generic functions left to extend.

Also note that due to the Home Section logic, especially for locative types with string names, dref-ext:docstring* should probably return a non-nil package.

λ

11.2 Locative Aliases

define-locative-alias can be used to help M-. and Specific Autolinks disambiguate references based on the context of a name as described on Parsing.

The following example shows how to make docstrings read more naturally by defining an alias.

(defclass my-string ()
  ())

(defgeneric my-string (obj)
  (:documentation "Convert OBJ to MY-STRING."))

;;; This version of FOO has a harder to read docstring because
;;; it needs to disambiguate the MY-STRING reference.
(defun foo (x)
  "FOO takes and argument X, a [MY-STRING][class] object.")

;;; Define OBJECT as an alias for the CLASS locative.
(define-locative-alias object class)

;;; Note how no explicit link is needed anymore.
(defun foo (x)
  "FOO takes an argument X, a MY-CLASS object.")

Similarly, defining the indefinite articles as aliases of the class locative can reduce the need for explicit linking.

(define-locative-alias a class)
(define-locative-alias an class)

Since these are unlikely to be universally helpful, make sure not to export the symbols a and an.

λ

11.3 Extending document

For all definitions that it encounters, document calls document-object* to generate documentation. The following utilities are for writing new document-object* methods, which emit markdown.

λ

11.4 Sections

section objects rarely need to be dissected since defsection and document cover most needs. However, it is plausible that one wants to subclass them and maybe redefine how they are presented.

λ

11.5 Glossary Terms

glossary-term objects rarely need to be dissected since define-glossary-term and document cover most needs. However, it is plausible that one wants to subclass them and maybe redefine how they are presented.