λ

PAX Manual

Table of Contents

[in package MGL-PAX]

λ

1 mgl-pax ASDF System Details

λ

2 Links

Here is the official repository and the HTML documentation for the latest version.

λ

3 Background

As a user, I frequently run into documentation that's incomplete and out of date, so I tend to stay in the editor and explore the code by jumping around with SLIME's M-.. As a library author, I spend a great deal of time polishing code, but precious little writing documentation.

In fact, I rarely write anything more comprehensive than docstrings for exported stuff. Writing docstrings feels easier than writing a separate user manual and they are always close at hand during development. The drawback of this style is that users of the library have to piece the big picture together themselves.

That's easy to solve, I thought, let's just put all the narrative that holds docstrings together in the code and be a bit like a Literate Programming weenie turned inside out. The original prototype which did almost everything I wanted was this:

(defmacro defsection (name docstring)
  `(defun ,name () ,docstring))

Armed with DEFSECTION, I soon found myself organizing code following the flow of user level documentation and relegated comments to implementational details entirely. However, some portions of DEFSECTION docstrings were just listings of all the functions, macros and variables related to the narrative, and this list was effectively repeated in the DEFPACKAGE form complete with little comments that were like section names. A clear violation of OAOO, one of them had to go, so DEFSECTION got a list of symbols to export.

That was great, but soon I found that the listing of symbols is ambiguous if, for example, a function, a compiler macro and a class are named by the same symbol. This did not concern exporting, of course, but it didn't help readability. Distractingly, on such symbols, M-. was popping up selection dialogs. There were two birds to kill, and the symbol got accompanied by a type which was later generalized into the concept of locatives:

(defsection @mgl-pax-introduction ()
  "A single line for one man ..."
  (foo class)
  (bar function))

After a bit of elisp hacking, M-. was smart enough to disambiguate based on the locative found in the vicinity of the symbol and everything was good for a while.

Then I realized that sections could refer to other sections if there were a SECTION locative. Going down that path, I soon began to feel the urge to generate pretty documentation as all the necessary information was manifest in the DEFSECTION forms. The design constraint imposed on documentation generation was that following the typical style of upcasing symbols in docstrings there should be no need to explicitly mark up links: if M-. works, then the documentation generator shall also be able find out what's being referred to.

I settled on Markdown as a reasonably non-intrusive format, and a few thousand lines later PAX was born.

λ

4 Tutorial

PAX provides an extremely poor man's Explorable Programming environment. Narrative primarily lives in so called sections that mix markdown docstrings with references to functions, variables, etc, all of which should probably have their own docstrings.

The primary focus is on making code easily explorable by using SLIME's M-. (slime-edit-definition). See how to enable some fanciness in Emacs Integration. Generating documentation from sections and all the referenced items in Markdown or HTML format is also implemented.

With the simplistic tools provided, one may accomplish similar effects as with Literate Programming, but documentation is generated from code, not vice versa and there is no support for chunking yet. Code is first, code must look pretty, documentation is code.

In typical use, PAX packages have no :EXPORT's defined. Instead the DEFINE-PACKAGE form gets a docstring which may mention section names (defined with DEFSECTION). When the code is loaded into the lisp, pressing M-. in SLIME on the name of the section will take you there. Sections can also refer to other sections, packages, functions, etc and you can keep exploring.

Here is an example of how it all works together:

(mgl-pax:define-package :foo-random
  (:documentation "This package provides various utilities for
  random. See FOO-RANDOM:@FOO-RANDOM-MANUAL.")
  (:use #:common-lisp #:mgl-pax))

(in-package :foo-random)

(defsection @foo-random-manual (:title "Foo Random manual")
  "Here you describe what's common to all the referenced (and
  exported) functions that follow. They work with *FOO-STATE*,
  and have a :RANDOM-STATE keyword arg. Also explain when to
  choose which."
  (foo-random-state class)
  (state (reader foo-random-state))
  "Hey we can also print states!"
  (print-object (method () (foo-random-state t)))
  (*foo-state* variable)
  (gaussian-random function)
  (uniform-random function)
  ;; this is a subsection
  (@foo-random-examples section))

(defclass foo-random-state ()
  ((state :reader state)))

(defmethod print-object ((object foo-random-state) stream)
  (print-unreadable-object (object stream :type t)))

(defvar *foo-state* (make-instance 'foo-random-state)
  "Much like *RANDOM-STATE* but uses the FOO algorithm.")

(defun uniform-random (limit &key (random-state *foo-state*))
  "Return a random number from the between 0 and LIMIT (exclusive)
  uniform distribution."
  nil)

(defun gaussian-random (stddev &key (random-state *foo-state*))
  "Return a random number from a zero mean normal distribution with
  STDDEV."
  nil)

(defsection @foo-random-examples (:title "Examples")
  "Let's see the transcript of a real session of someone working
  with FOO:

  ```cl-transcript
  (values (princ :hello) (list 1 2))
  .. HELLO
  => :HELLO
  => (1 2)

  (make-instance 'foo-random-state)
  ==> #<FOO-RANDOM-STATE >
  ```")

Generating documentation in a very stripped down markdown format is easy:

(describe @foo-random-manual)

For this example, the generated markdown would look like this:

# Foo Random manual

###### \[in package FOO-RANDOM\]
Here you describe what's common to all the referenced (and
exported) functions that follow. They work with *FOO-STATE*,
and have a :RANDOM-STATE keyword arg. Also explain when to
choose which.

- [class] FOO-RANDOM-STATE

- [reader] STATE FOO-RANDOM-STATE

Hey we can also print states!

- [method] PRINT-OBJECT (OBJECT FOO-RANDOM-STATE) STREAM

- [variable] *FOO-STATE* #<FOO-RANDOM-STATE >

    Much like *RANDOM-STATE* but uses the FOO algorithm.

- [function] GAUSSIAN-RANDOM STDDEV &KEY (RANDOM-STATE *FOO-STATE*)

    Return a random number from a zero mean normal distribution with
    STDDEV.

- [function] UNIFORM-RANDOM LIMIT &KEY (RANDOM-STATE *FOO-STATE*)

    Return a random number from the between 0 and LIMIT (exclusive)
    uniform distribution.

## Examples

Let's see the transcript of a real session of someone working
with FOO:

```cl-transcript
(values (princ :hello) (list 1 2))
.. HELLO
=> :HELLO
=> (1 2)

(make-instance 'foo-random-state)
==> #<FOO-RANDOM-STATE >

```

More fancy markdown or HTML output with automatic markup and linking of uppercase symbol names found in docstrings, section numbering, table of contents, etc is possible by calling the DOCUMENT function.

One can even generate documentation for different, but related libraries at the same time with the output going to different files, but with cross-page links being automatically added for symbols mentioned in docstrings. For a complete example of how to generate HTML with multiple pages, see src/doc.lisp.

Note how (VARIABLE *FOO-STATE*) in the DEFSECTION form both exports *FOO-STATE* and includes its documentation in @FOO-RANDOM-MANUAL. The symbols VARIABLE and FUNCTION are just two instances of 'locatives' which are used in DEFSECTION to refer to definitions tied to symbols. See Locative Types.

The transcript in the code block tagged with cl-transcript is automatically checked for up-to-dateness. See Transcripts.

λ

5 Emacs Integration

Integration into SLIME's M-. (slime-edit-definition) allows one to visit the source location of the thing that's identified by a symbol and the locative before or after the symbol in a buffer. With this extension, if a locative is the previous or the next expression around the symbol of interest, then M-. will go straight to the definition which corresponds to the locative. If that fails, M-. will try to find the definitions in the normal way which may involve popping up an xref buffer and letting the user interactively select one of possible definitions.

Note that the this feature is implemented in terms of SWANK-BACKEND:FIND-SOURCE-LOCATION and SWANK-BACKEND:FIND-DEFINITIONS whose support varies across the Lisp implementations.

In the following examples, pressing M-. when the cursor is on one of the characters of FOO or just after FOO, will visit the definition of function FOO:

function foo
foo function
(function foo)
(foo function)

In particular, references in a DEFSECTION form are in (SYMBOL LOCATIVE) format so M-. will work just fine there.

Just like vanilla M-., this works in comments and docstrings. In this example pressing M-. on FOO will visit FOO's default method:

;;;; See FOO `(method () (t t t))` for how this all works.
;;;; But if the locative has semicolons inside: FOO `(method
;;;; () (t t t))`, then it won't, so be wary of line breaks
;;;; in comments.

With a prefix argument (C-u M-.), one can enter a symbol plus a locative separated by whitespace to preselect one of the possibilities.

The M-. extensions can be enabled by adding this to your Emacs initialization file (or loading src/pax.el):

;;; MGL-PAX M-. integration

(defun slime-edit-locative-definition (name &optional where)
  (or (slime-locate-definition name (slime-locative-before))
      (slime-locate-definition name (slime-locative-after))
      (slime-locate-definition name (slime-locative-after-in-brackets))
      ;; support "foo function" and "function foo" syntax in
      ;; interactive use
      (let ((pos (cl-position ?\s name)))
        (when pos
          (or (slime-locate-definition (cl-subseq name 0 pos)
                                       (cl-subseq name (1+ pos)))
              (slime-locate-definition (cl-subseq name (1+ pos))
                                       (cl-subseq name 0 pos)))))))

(defun slime-locative-before ()
  (ignore-errors (save-excursion
                   (slime-beginning-of-symbol)
                   (slime-last-expression))))

(defun slime-locative-after ()
  (ignore-errors (save-excursion
                   (slime-end-of-symbol)
                   (slime-forward-sexp)
                   (slime-last-expression))))

(defun slime-locative-after-in-brackets ()
  (ignore-errors (save-excursion
                   (slime-end-of-symbol)
                   (skip-chars-forward "`" (+ (point) 1))
                   (when (and (= 1 (skip-chars-forward "\\]" (+ (point) 1)))
                              (= 1 (skip-chars-forward "\\[" (+ (point) 1))))
                     (buffer-substring-no-properties
                      (point)
                      (progn (search-forward "]" nil (+ (point) 1000))
                             (1- (point))))))))

(defun slime-locate-definition (name locative)
  (when locative
    (let ((location
           (slime-eval
            ;; Silently fail if mgl-pax is not loaded.
            `(cl:when (cl:find-package :mgl-pax)
                      (cl:funcall
                       (cl:find-symbol
                        (cl:symbol-name :locate-definition-for-emacs) :mgl-pax)
                       ,name ,locative)))))
      (when (and (consp location)
                 (not (eq (car location) :error)))
        (slime-edit-definition-cont
         (list (make-slime-xref :dspec `(,name)
                                :location location))
         "dummy name"
         where)))))

(add-hook 'slime-edit-definition-hooks 'slime-edit-locative-definition)

λ

6 Basics

Now let's examine the most important pieces in detail.

λ

7 Markdown Support

The Markdown in docstrings is processed with the 3BMD library.

λ

7.1 Indentation

Docstrings can be indented in any of the usual styles. PAX normalizes indentation by converting:

(defun foo ()
  "This is
  indented
  differently")

to

(defun foo ()
  "This is
indented
differently")

See DOCUMENT-OBJECT for the details.

λ

7.2 Syntax highlighting

For syntax highlighting, github's fenced code blocks markdown extension to mark up code blocks with triple backticks is enabled so all you need to do is write:

```elisp
(defun foo ())
```

to get syntactically marked up HTML output. Copy doc/style.css from PAX and you are set. The language tag, elisp in this example, is optional and defaults to common-lisp.

See the documentation of 3BMD and colorize for the details.

λ

7.3 MathJax

Displaying pretty mathematics in TeX format is supported via MathJax. It can be done inline with $ like this:

$\int_0^\infty e^{-x^2} dx=\frac{\sqrt{\pi}}{2}$

which is diplayed as $\int_0^\infty e^{-x^2} dx=\frac{\sqrt{\pi}}{2}$, or it can be delimited by $$ like this:

$$\int_0^\infty e^{-x^2} dx=\frac{\sqrt{\pi}}{2}$$

to get: $$\int_0^\infty e^{-x^2} dx=\frac{\sqrt{\pi}}{2}$$

MathJax will leave code blocks (including those inline with backticks) alone. Outside code blocks, escape $ by prefixing it with a backslash to scare MathJax off.

Escaping all those backslashes in TeX fragments embedded in Lisp strings can be pain. Pythonic String Reader can help with that.

λ

8 Documentation Printer Variables

Docstrings are assumed to be in markdown format and they are pretty much copied verbatim to the documentation subject to a few knobs described below.

λ

9 Locative Types

These are the locatives type supported out of the box. As all locative types, they are symbols and their names should make it obvious what kind of things they refer to. Unless otherwise noted, locatives take no arguments.

λ

10 Extension API

λ

10.1 Locatives and References

While Common Lisp has rather good introspective abilities, not everything is first class. For example, there is no object representing the variable defined with (DEFVAR FOO). (MAKE-REFERENCE 'FOO 'VARIABLE) constructs a REFERENCE that captures the path to take from an object (the symbol FOO) to an entity of interest (for example, the documentation of the variable). The path is called the locative. A locative can be applied to an object like this:

(locate 'foo 'variable)

which will return the same reference as (MAKE-REFERENCE 'FOO 'VARIABLE). Operations need to know how to deal with references which we will see in LOCATE-AND-COLLECT-REACHABLE-OBJECTS, LOCATE-AND-DOCUMENT and LOCATE-AND-FIND-SOURCE.

Naturally, (LOCATE 'FOO 'FUNCTION) will simply return #'FOO, no need to muck with references when there is a perfectly good object.

λ

10.2 Adding New Object Types

One may wish to make the DOCUMENT function and M-. navigation work with new object types. Extending DOCUMENT can be done by defining a DOCUMENT-OBJECT method. To allow these objects to be referenced from DEFSECTION a LOCATE-OBJECT method is to be defined. Finally, for M-. FIND-SOURCE can be specialized. Finally, EXPORTABLE-LOCATIVE-TYPE-P may be overridden if exporting does not makes sense. Here is a stripped down example of how all this is done for ASDF:SYSTEM:

(define-locative-type asdf:system ()
  "Refers to an asdf system. The generated documentation will include
  meta information extracted from the system definition. This also
  serves as an example of a symbol that's not accessible in the
  current package and consequently is not exported.")

(defmethod locate-object (symbol (locative-type (eql 'asdf:system))
                          locative-args)
  (assert (endp locative-args))
  ;; FIXME: This is slow as hell.
  (or (asdf:find-system symbol nil)
      (locate-error)))

(defmethod canonical-reference ((system asdf:system))
  (make-reference (asdf/find-system:primary-system-name system) 'asdf:system))

(defmethod document-object ((system asdf:system) stream)
  (with-heading (stream system
                        (format nil "~A ASDF System Details"
                                (asdf/find-system:primary-system-name system)))
    (flet ((foo (name fn &key type)
             (let ((value (funcall fn system)))
               (when value
                 (case type
                   ((:link)
                    (format stream "- ~A: [~A](~A)~%" name value value))
                   ((:mailto)
                    (format stream "- ~A: [~A](mailto:~A)~%"
                            name value value))
                   ((nil)
                    (format stream "- ~A: ~A~%" name value)))))))
      (foo "Version" 'asdf/component:component-version)
      (foo "Description" 'asdf/system:system-description)
      (foo "Licence" 'asdf/system:system-licence)
      (foo "Author" 'asdf/system:system-author)
      (foo "Maintainer" 'asdf/system:system-maintainer)
      (foo "Mailto" 'asdf/system:system-mailto :type :mailto)
      (foo "Homepage" 'asdf/system:system-homepage :type :link)
      (foo "Bug tracker" 'asdf/system:system-bug-tracker)
      (foo "Long description" 'asdf/system:system-long-description))))

(defmethod find-source ((system asdf:system))
  `(:location
    (:file ,(namestring (asdf/system:system-source-file system)))
    (:position 1)
    (:snippet "")))

λ

10.3 Reference Based Extensions

Let's see how to extend DOCUMENT and M-. navigation if there is no first class object to represent the thing of interest. Recall that LOCATE returns a REFERENCE object in this case. DOCUMENT-OBJECT and FIND-SOURCE defer to LOCATE-AND-DOCUMENT and LOCATE-AND-FIND-SOURCE which have LOCATIVE-TYPE in their argument list for EQL specializing pleasure. Here is a stripped down example of how the VARIABLE locative is defined:

(define-locative-type variable (&optional initform)
  "Refers to a global special variable. INITFORM, or if not specified,
  the global value of the variable is included in the documentation.")

(defmethod locate-object (symbol (locative-type (eql 'variable)) locative-args)
  (assert (<= (length locative-args) 1))
  (make-reference symbol (cons locative-type locative-args)))

(defmethod locate-and-document (symbol (locative-type (eql 'variable))
                                locative-args stream)
  (destructuring-bind (&optional (initform nil initformp)) locative-args
    (locate-and-print-bullet locative-type locative-args symbol stream)
    (write-char #\Space stream)
    (multiple-value-bind (value unboundp) (symbol-global-value symbol)
      (print-arglist (prin1-to-string (cond (initformp initform)
                                            (unboundp "-unbound-")
                                            (t value)))
                     stream))
    (terpri stream)
    (with-dislocated-symbols ((list symbol))
      (maybe-print-docstring symbol locative-type stream))))

(defmethod locate-and-find-source (symbol (locative-type (eql 'variable))
                                   locative-args)
  (declare (ignore locative-args))
  (find-one-location (swank-backend:find-definitions symbol)
                     '("variable" "defvar" "defparameter"
                       "special-declaration")))

We have covered the basic building blocks of reference based extensions. Now let's see how the obscure DEFINE-SYMBOL-LOCATIVE-TYPE and DEFINE-DEFINER-FOR-SYMBOL-LOCATIVE-TYPE macros work together to simplify the common task of associating definition and documentation with symbols in a certain context.

λ

10.4 Sections

Section objects rarely need to be dissected since DEFSECTION and DOCUMENT cover most needs. However, it is plausible that one wants to subclass them and maybe redefine how they are presented.

λ

11 Transcripts

What are transcripts for? When writing a tutorial, one often wants to include a REPL session with maybe a few defuns and a couple of forms whose output or return values are shown. Also, in a function's docstring an example call with concrete arguments and return values speaks volumes. A transcript is a text that looks like a repl session, but which has a light markup for printed output and return values, while no markup (i.e. prompt) for lisp forms. The PAX transcripts may include output and return values of all forms, or only selected ones. In either case the transcript itself can be easily generated from the source code.

The main worry associated with including examples in the documentation is that they tend to get out-of-sync with the code. This is solved by being able to parse back and update transcripts. In fact, this is exactly what happens during documentation generation with PAX. Code sections tagged cl-transcript are retranscribed and checked for inconsistency (that is, any difference in output or return values). If the consistency check fails, an error is signalled that includes a reference to the object being documented.

Going beyond documentation, transcript consistency checks can be used for writing simple tests in a very readable form. For example:

(+ 1 2)
=> 3

(values (princ :hello) (list 1 2))
.. HELLO
=> :HELLO
=> (1 2)

All in all, transcripts are a handy tool especially when combined with the Emacs support to regenerate them and with PYTHONIC-STRING-READER and its triple-quoted strings that allow one to work with nested strings with less noise. The triple-quote syntax can be enabled with:

(in-readtable pythonic-string-syntax)

λ

11.1 Transcribing with Emacs

Typical transcript usage from within Emacs is simple: add a lisp form to a docstring or comment at any indentation level. Move the cursor right after the end of the form as if you were to evaluate it with C-x C-e. The cursor is marked by #\^:

This is part of a docstring.

```cl-transcript
(values (princ :hello) (list 1 2))^
```

Note that the use of fenced code blocks with the language tag cl-transcript is only to tell PAX to perform consistency checks at documentation generation time.

Now invoke the elisp function mgl-pax-transcribe where the cursor is and the fenced code block from the docstring becomes:

(values (princ :hello) (list 1 2))
.. HELLO
=> :HELLO
=> (1 2)
^

Then you change the printed message and add a comment to the second return value:

(values (princ :hello-world) (list 1 2))
.. HELLO
=> :HELLO
=> (1
    ;; This value is arbitrary.
    2)

When generating the documentation you get a TRANSCRIPTION-CONSISTENCY-ERROR because the printed output and the first return value changed so you regenerate the documentation by marking the region of bounded by #\| and the cursor at #\^ in the example:

|(values (princ :hello-world) (list 1 2))
.. HELLO
=> :HELLO
=> (1
    ;; This value is arbitrary.
    2)
^

then invoke the elisp function mgl-pax-retranscribe-region to get:

(values (princ :hello-world) (list 1 2))
.. HELLO-WORLD
=> :HELLO-WORLD
=> (1
    ;; This value is arbitrary.
    2)
^

Note how the indentation and the comment of (1 2) was left alone but the output and the first return value got updated.

Alternatively, C-u 1 mgl-pax-transcribe will emit commented markup:

(values (princ :hello) (list 1 2))
;.. HELLO
;=> :HELLO
;=> (1 2)

C-u 0 mgl-pax-retranscribe-region will turn commented into non-commented markup. In general, the numeric prefix argument is the index of the syntax to be used in MGL-PAX:*SYNTAXES*. Without a prefix argument mgl-pax-retranscribe-region will not change the markup style.

Finally, not only do both functions work at any indentation level, but in comments too:

;;;; (values (princ :hello) (list 1 2))
;;;; .. HELLO
;;;; => :HELLO
;;;; => (1 2)

Transcription support in emacs can be enabled by adding this to your Emacs initialization file (or loading src/transcribe.el):

;;; MGL-PAX transcription

(defun mgl-pax-transcribe-last-expression ()
  "A bit like C-u C-x C-e (slime-eval-last-expression) that
inserts the output and values of the sexp before the point, this
does the same but with MGL-PAX:TRANSCRIBE. Use a numeric prefix
argument as in index to select one of the Common Lisp
MGL-PAX:*SYNTAXES* as the SYNTAX argument to MGL-PAX:TRANSCRIBE.
Without a prefix argument, the first syntax is used."
  (interactive)
  (insert
   (save-excursion
     (let* ((end (point))
            (start (progn (backward-sexp)
                          (move-beginning-of-line nil)
                          (point))))
       (mgl-pax-transcribe start end (mgl-pax-transcribe-syntax-arg)
                           nil nil nil)))))

(defun mgl-pax-retranscribe-region (start end)
  "Updates the transcription in the current region (as in calling
MGL-PAX:TRANSCRIBE with :UPDATE-ONLY T). Use a numeric prefix
argument as in index to select one of the Common Lisp
MGL-PAX:*SYNTAXES* as the SYNTAX argument to MGL-PAX:TRANSCRIBE.
Without a prefix argument, the syntax of the input will not be
changed."
  (interactive "r")
  (let* ((point-at-start-p (= (point) start))
         (point-at-end-p (= (point) end))
         (transcript (mgl-pax-transcribe start end
                                         (mgl-pax-transcribe-syntax-arg)
                                         t t nil)))
    (if point-at-start-p
        (save-excursion
          (goto-char start)
          (delete-region start end)
          (insert transcript))
      (save-excursion
          (goto-char start)
          (delete-region start end))
      (insert transcript))))

(defun mgl-pax-transcribe-syntax-arg ()
  (if current-prefix-arg
      (prefix-numeric-value current-prefix-arg)
    nil))

(defun mgl-pax-transcribe (start end syntax update-only echo
                                 first-line-special-p)
  (let ((transcription
         (slime-eval
          `(cl:if (cl:find-package :mgl-pax)
                  (cl:funcall
                   (cl:find-symbol
                    (cl:symbol-name :transcribe-for-emacs) :mgl-pax)
                   ,(buffer-substring-no-properties start end)
                   ',syntax ',update-only ',echo ',first-line-special-p)
                  t))))
    (if (eq transcription t)
        (error "MGL-PAX is not loaded.")
      transcription)))

λ

11.2 Transcript API

λ

12 Utilities