→ ↺ λ

Try Manual

Table of Contents

[in package TRY]

← ↑ → ↺ λ

1 The try ASDF System

← ↑ → ↺ λ

2 Links

Here is the official repository and the HTML documentation for the latest version.

← ↑ → ↺ λ

3 Tutorial

Try is a library for unit testing with equal support for interactive and non-interactive workflows. Tests are functions, and almost everything else is a condition, whose types feature prominently in parameterization.

Try is is what we get if we make tests functions and build a test framework on top of the condition system as Stefil did but also address the issue of rerunning and replaying, make the is check more capable, use the types of the condition hierarchy to parameterize what to debug, print, rerun, and finally document the whole thing.

Looking for Truth

The is Macro is a replacement for cl:assert, that can capture values of subforms to provide context to failures:

(is (= (1+ 5) 0))
.. debugger invoked on UNEXPECTED-RESULT-FAILURE:
..   UNEXPECTED-FAILURE in check:
..     (IS (= #1=(1+ 5) 0))
..   where
..     #1# = 6

This is a PAX transcript, output is prefixed with ... Readable and unreadable return values are prefixed with => and ==>, respectively.

Note the #n# syntax due to *print-circle*.

Checking Multiple Values

is automatically captures values of arguments to functions like 1+ in the above example. Values of other interesting subforms can be explicitly captured. is supports capturing multiple values and can be taught how to deal with macros. The combination of these features allows match-values to be implementable as tiny extension:

(is (match-values (values (1+ 5) "sdf")
      (= * 0)
      (string= * "sdf")))
.. debugger invoked on UNEXPECTED-RESULT-FAILURE:
..   UNEXPECTED-FAILURE in check:
..     (IS
..      (MATCH-VALUES #1=(VALUES (1+ 5) #2="sdf")
..        (= * 0)
..        (STRING= * "sdf")))
..   where
..     #1# == 6
..            #2#

In the body of match-values, * is bound to successive return values of some form, here (values (1+ 5) "sdf"). match-values comes with an automatic rewrite rule that captures the values of this form, which are printed above as #1# == 6 #2#. is is flexible enough that all other checks (signals, signals-not, invokes-debugger, invokes-debugger-not, fails, and in-time are built on top of it.

Writing Tests

Beyond is, a fancy assert, Try provides tests, which are Lisp functions that record their execution in trial objects. Let's define a test and run it:

(deftest should-work ()
  (is t))

(should-work)
.. SHOULD-WORK            ; TRIAL-START
..   ⋅ (IS T)             ; EXPECTED-RESULT-SUCCESS
.. ⋅ SHOULD-WORK ⋅1       ; EXPECTED-VERDICT-SUCCESS
..
==> #<TRIAL (SHOULD-WORK) EXPECTED-SUCCESS 0.000s ⋅1>

Try is driven by conditions, and the comments to the right give the type of the condition that is printed on that line. The ⋅ character marks successes.

We could have run our test with (try 'should-work) as well, which does pretty much the same thing except it defaults to never entering the debugger, whereas calling a test function directly enters the debugger on events whose type matches the type in the variable *debug*.

(try 'should-work)
.. SHOULD-WORK
..   ⋅ (IS T)
.. ⋅ SHOULD-WORK ⋅1
..
==> #<TRIAL (SHOULD-WORK) EXPECTED-SUCCESS 0.000s ⋅1>
Test Suites

Test suites are just tests that call other tests.

(deftest my-suite ()
  (should-work)
  (is (= (foo) 5)))

(defun foo ()
  4)

(try 'my-suite)
.. MY-SUITE                 ; TRIAL-START
..   SHOULD-WORK            ; TRIAL-START
..     ⋅ (IS T)             ; EXPECTED-RESULT-SUCCESS
..   ⋅ SHOULD-WORK ⋅1       ; EXPECTED-VERDICT-SUCCESS
..   ⊠ (IS (= #1=(FOO) 5))  ; UNEXPECTED-RESULT-FAILURE
..     where
..       #1# = 4
.. ⊠ MY-SUITE ⊠1 ⋅1         ; UNEXPECTED-VERDICT-FAILURE
..
==> #<TRIAL (MY-SUITE) UNEXPECTED-FAILURE 0.000s ⊠1 ⋅1>

⊠ marks unexpected-failures. Note how the failure of (is (= (foo) 5)) caused my-suite to fail as well. Finally, the ⊠1 and the ⋅1 in the trial's printed representation are the event counts.

Filtering Output

To focus on the important bits, we can print only the unexpected events:

(try 'my-suite :print 'unexpected)
.. MY-SUITE
..   ⊠ (IS (= #1=(FOO) 5))
..     where
..       #1# = 4
.. ⊠ MY-SUITE ⊠1 ⋅1
..
==> #<TRIAL (MY-SUITE) UNEXPECTED-FAILURE 0.000s ⊠1 ⋅1>

Note that should-work is still run, and its check's success is counted as evidenced by⋅1. The above effect can also be achieved without running the tests again with replay-events.

Debugging

Let's figure out what went wrong:

(my-suite)

;;; Here the debugger is invoked:
UNEXPECTED-FAILURE in check:
  (IS (= #1=(FOO) 5))
where
  #1# = 4
Restarts:
 0: [RECORD-EVENT] Record the event and continue.
 1: [FORCE-EXPECTED-SUCCESS] Change outcome to TRY:EXPECTED-RESULT-SUCCESS.
 2: [FORCE-UNEXPECTED-SUCCESS] Change outcome to TRY:UNEXPECTED-RESULT-SUCCESS.
 3: [FORCE-EXPECTED-FAILURE] Change outcome to TRY:EXPECTED-RESULT-FAILURE.
 4: [ABORT-CHECK] Change outcome to TRY:RESULT-ABORT*.
 5: [SKIP-CHECK] Change outcome to TRY:RESULT-SKIP.
 6: [RETRY-CHECK] Retry check.
 7: [ABORT-TRIAL] Record the event and abort trial TRY::MY-SUITE.
 8: [SKIP-TRIAL] Record the event and skip trial TRY::MY-SUITE.
 9: [RETRY-TRIAL] Record the event and retry trial TRY::MY-SUITE.
 10: [SET-TRY-DEBUG] Supply a new value for :DEBUG of TRY:TRY.
 11: [RETRY] Retry SLIME interactive evaluation request.

In the SLIME debugger, we press v on the frame of the call to my-suite to navigate to its definition, realize what the problem is and fix foo:

(defun foo ()
  5)

Now, we select the retry-trial restart, and on the retry my-suite passes. The full output is:

MY-SUITE
  SHOULD-WORK
    ⋅ (IS T)
  ⋅ SHOULD-WORK ⋅1
WARNING: redefining TRY::FOO in DEFUN
  ⊠ (IS (= #1=(FOO) 5))
    where
      #1# = 4
MY-SUITE retry #1
  SHOULD-WORK
    ⋅ (IS T)
  ⋅ SHOULD-WORK ⋅1
  ⋅ (IS (= (FOO) 5))
⋅ MY-SUITE ⋅2
Rerunning Stuff

Instead of working interactively, one can fix the failing test and rerun it. Now, let's fix my-suite and rerun it:

(deftest my-suite ()
  (should-work)
  (is nil))

(try 'my-suite)
.. MY-SUITE
..   SHOULD-WORK
..     ⋅ (IS T)
..   ⋅ SHOULD-WORK ⋅1
..   ⊠ (IS NIL)
.. ⊠ MY-SUITE ⊠1 ⋅1
..
==> #<TRIAL (MY-SUITE) UNEXPECTED-FAILURE 0.000s ⊠1 ⋅1>

(deftest my-suite ()
  (should-work)
  (is t))

(try !)
.. MY-SUITE
..   ⋅ (IS T)
.. ⋅ MY-SUITE ⋅1
..
==> #<TRIAL (MY-SUITE) EXPECTED-SUCCESS 0.004s ⋅1>

Here, ! refers to the most recent trial returned by try. When a trial is passed to try or is funcalled, trials in it that match the type in try's rerun argument are rerun (here, unexpected by default). should-work and its check are expected-successes, hence they don't match unexpected and are not rerun.

Conditional Execution

Conditional execution can be achieved simply testing the trial object returned by Tests.

(deftest my-suite ()
  (when (passedp (should-work))
    (is t :msg "a test that depends on SHOULD-WORK")
    (when (is nil)
      (is nil :msg "never run"))))
Skipping

Sometimes, we do not know up front that a test should not be executed. Calling skip-trial unwinds from the current-trial and sets it skipped.

(deftest my-suite ()
  (is t)
  (skip-trial)
  (is nil))

(my-suite)
==> #<TRIAL (MY-SUITE) SKIP 0.000s ⋅1>

In the above, (is t) was executed, but (is nil) was not.

Expecting Outcomes
(deftest known-broken ()
  (with-failure-expected (t)
    (is nil)))

(known-broken)
.. KNOWN-BROKEN
..   × (IS NIL)
.. ⋅ KNOWN-BROKEN ×1
..
==> #<TRIAL (KNOWN-BROKEN) EXPECTED-SUCCESS 0.000s ×1>

× marks expected-failures. (with-skip (t) ...) makes all checks successes and failures expected, which are counted in their own *categories* by default but don't make the enclosing tests to fail. Also see with-expected-outcome.

Running Tests on Definition

With *run-deftest-when*, tests on in various eval-when situations. To run tests on evaluation, as in SLIME C-M-x, slime-eval-defun:

(setq *run-deftest-when* :execute)

(deftest some-test ()
  (is t))
.. SOME-TEST
..   ⋅ (IS T)
.. ⋅ SOME-TEST ⋅1
..
=> SOME-TEST

(setq *run-deftest-when* nil)
Fixtures

There is no direct support for fixtures in Try. One can easily write macros like the following.

(defvar *server* nil)

(defmacro with-xxx (&body body)
  `(flet ((,with-xxx-body ()
            ,@body))
     (if *server*
         (with-xxx-body)
         (with-server (make-expensive-server)
           (with-xxx-body)))))

Plus, with support for selectively Rerunning Trials, the need for fixtures is lessened.

Packages

The suggested way of writing tests is to call test functions explicitly:

(defpackage :some-test-package
  (:use #:common-lisp #:try))
(in-package :some-test-package)

(deftest test-all ()
  (test-this)
  (test-that))

(deftest test-this ()
  (test-this/more))

(deftest test-this/more ()
  (is t))

(deftest test-that ()
  (is t))

(deftest not-called ()
  (is t))

(defun test ()
  (warn-on-tests-not-run ((find-package :some-test-package))
    (try 'test-all)))

(test)
.. TEST-ALL
..   TEST-THIS
..     TEST-THIS/MORE
..       ⋅ (IS T)
..     ⋅ TEST-THIS/MORE ⋅1
..   ⋅ TEST-THIS ⋅1
..   TEST-THAT
..     ⋅ (IS T)
..   ⋅ TEST-THAT ⋅1
.. ⋅ TEST-ALL ⋅2
.. WARNING: Test NOT-CALLED not run.
==> #<TRIAL (TEST-ALL) EXPECTED-SUCCESS 0.012s ⋅2>

Note how the test function uses warn-on-tests-not-run to catch any tests defined in some-test-package that were not run. Tests can be deleted by fmakunbound, unintern, or by redefining the function with defun. Tests defined in a given package can be listed with list-package-tests.

This style allows higher level tests to establish the dynamic environment necessary for lower level tests.

← ↑ → ↺ λ

4 Emacs Integration

The Elisp mgl-try interactive command runs a Try test and displays its output in a lisp-mode buffer with minor modes outline-mode and mgl-try-mode. It is assumed that the lisp is running under Slime. In the buffer,

← ↑ → ↺ λ

4.1 Emacs Setup

Load src/mgl-try.el in Emacs.

If you installed Try with Quicklisp, the location of mgl-try.el may change with updates, and you may want to copy the current version of mgl-try.el to a stable location:

(try:install-try-elisp "~/quicklisp/")

Then, assuming the Elisp file is in the quicklisp directory, add something like this to your .emacs:

(load "~/quicklisp/mgl-try.el")

← ↑ → ↺ λ

5 Events

Try is built around events implemented as conditions. Matching the types of events to *debug*, *count*, *collect*, *rerun*, *print*, and *describe* is what gives Try its flexibility.

← ↑ → ↺ λ

5.1 Middle Layer of Events

The event hierarchy is fairly involved, so let's start in the middle. The condition event has 4 disjoint subclasses:

(let (;; We don't want to debug nor print a backtrace for the error below.
      (*debug* nil)
      (*describe* nil))
  ;; signals TRIAL-START / VERDICT-ABORT* on entry / exit
  (with-test (demo)
    ;; signals EXPECTED-RESULT-SUCCESS
    (is t)
    ;; signals UNHANDLED-ERROR with a nested CL:ERROR
    (error "xxx")))
.. DEMO                       ; TRIAL-START
..   ⋅ (IS T)                 ; EXPECTED-RESULT-SUCCESS (⋅)
..   ⊟ "xxx" (SIMPLE-ERROR)   ; UNHANDLED-ERROR (⊟)
.. ⊟ DEMO ⊟1 ⋅1               ; VERDICT-ABORT* (⊟)
..
==> #<TRIAL (WITH-TEST (DEMO)) ABORT* 0.004s ⊟1 ⋅1>

← ↑ → ↺ λ

5.2 Concrete Events

The non-abstract condition classes of events that are actually signalled are called concrete.

trial-start is a concrete event class. results and verdicts have six concrete subclasses:

error* is an abstract class with two concrete subclasses:

These are the 15 concrete event classes.

← ↑ → ↺ λ

5.3 Event Glue

These condition classes group various bits of the Concrete Events and the Middle Layer of Events for ease of reference.

Concrete event classes except trial-start are subclasses of hyphen-separated words in their name. For example, unexpected-result-failure inherits from unexpected, result, and failure, so it matches types such as unexpected or (and unexpected result).

← ↑ → ↺ λ

5.4 Printing Events

← ↑ → ↺ λ

5.5 Event Restarts

Only record-event is applicable to all events. See Check Restarts, Trial Restarts for more.

← ↑ → ↺ λ

5.6 Outcomes

← ↑ → ↺ λ

5.6.1 Outcome Restarts

← ↑ → ↺ λ

5.6.2 Checks

Checks are like cl:asserts, they check whether some condition holds and signal an outcome. The outcome signalled for checks is a subclass of result.

Take, for example, (is (= x 5)). Depending on whether x is indeed 5, some kind of result success or failure will be signalled. with-expected-outcome determines whether it's expected or unexpected, and we have one of expected-result-success, unexpected-result-success, expected-result-failure, unexpected-result-failure to signal. Furthermore, if with-skip is in effect, then result-skip is signalled.

The result is signalled with #'signal if it is a pass, else it's signalled with #'error. This distinction matters only if the event is not handled, which is never the case in a trial. Standalone checks though - those that are not enclosed by a trial - invoke the debugger on results which are not of type pass.

The signalled result is not final until record-event is invoked on it, and it can be changed with the Outcome Restarts and the Check Restarts.

← ↑ → ↺ λ

Check Restarts

← ↑ → ↺ λ

5.6.3 Trials

← ↑ → ↺ λ

Trial Events

← ↑ → ↺ λ

Trial Verdicts

When a trial finished, a verdict is signalled. The verdict's type is determined as follows.

The verdict of this type is signalled, but its type can be changed by the Outcome Restarts or the Trial Restarts before record-event is invoked on it.

← ↑ → ↺ λ

Trial Restarts

There are three restarts available for manipulating running trials: abort-trial, skip-trial, and retry-trial. They may be invoked programatically or from the debugger. abort-trial is also invoked by try when encountering unhandled-error.

The functions below invoke one of these restarts associated with a trial. It is an error to call them on trials that are not runningp, but they may be called on trials other than the current-trial. In that case, any intervening trials are skipped.

;; Skipped trials are marked with '-' in the output.
(with-test (outer)
  (with-test (inner)
    (is t)
    (skip-trial nil outer)))
.. OUTER
..   INNER
..     ⋅ (IS T)
..   - INNER ⋅1
.. - OUTER ⋅1
..
==> #<TRIAL (WITH-TEST (OUTER)) SKIP 0.000s ⋅1>

Furthermore, all three restarts initiate a non-local exit to return from the trial. If during the unwinding of the stack, the non-local-exit is cancelled (see cancelled non-local exit), the appropriate restart will be invoked upon returning from the trial. In the following example, the non-local exit from a skip is cancelled by a throw.

(with-test (some-test)
  (catch 'foo
    (unwind-protect
         (skip-trial)
      (throw 'foo nil)))
  (is t :msg "check after skip"))
.. SOME-TEST
..   ⋅ check after skip
.. - SOME-TEST ⋅1
..
==> #<TRIAL (WITH-TEST (SOME-TEST)) SKIP 0.000s ⋅1>

In the next example, the non-local exit from a skip is cancelled by an error(0 1), which triggers an abort-trial.

(let ((*debug* nil)
      (*describe* nil))
  (with-test (foo)
    (unwind-protect
         (skip-trial)
      (error "xxx"))))
.. FOO
..   ⊟ "xxx" (SIMPLE-ERROR)
.. ⊟ FOO ⊟1
..
==> #<TRIAL (WITH-TEST (FOO)) ABORT* 0.000s ⊟1>

All three restarts may be invoked on any event, including the trial's own trial-start and verdict. If their condition argument is an event (retry-trial has a special case here), they also record it (as in record-event) to ensure that when they handle an event in the debugger or programatically that event is not dropped.

← ↑ → ↺ λ

5.7 Errors

← ↑ → ↺ λ

5.8 Categories

Categories determine how event types are printed and events of what types are counted together.

The default value of *categories* is

((abort*             :marker "⊟")
 (unexpected-failure :marker "⊠")
 (unexpected-success :marker "⊡")
 (skip               :marker "-")
 (expected-failure   :marker "×")
 (expected-success   :marker "⋅"))

which says that all concrete events that are of type abort* (i.e. result-abort*, verdict-abort*, unhandled-error, and nlx) are to be marked with "⊟" when printed (see Printing Events). Also, the six types define six counters for Counting Events. Note that unexpected events have the same marker but squared as their expected counterpart.

← ↑ → ↺ λ

6 The is Macro

is is the fundamental one among Checks, on which all the others are built, and it is a replacement for cl:assert that can capture values of subforms to provide context to failures:

(is (= (1+ 5) 0))
.. debugger invoked on UNEXPECTED-RESULT-FAILURE:
..   UNEXPECTED-FAILURE in check:
..     (IS (= #1=(1+ 5) 0))
..   where
..     #1# = 6

is automatically captures values of arguments to functions like 1+ in the above example. Values of other interesting subforms can be explicitly requested to be captured. is supports capturing multiple values and can be taught how to deal with macros. The combination of these features allows match-values to be implementable as tiny extension:

(is (match-values (values (1+ 5) "sdf")
      (= * 0)
      (string= * "sdf")))
.. debugger invoked on UNEXPECTED-RESULT-FAILURE:
..   UNEXPECTED-FAILURE in check:
..     (IS
..      (MATCH-VALUES #1=(VALUES (1+ 5) #2="sdf")
..        (= * 0)
..        (STRING= * "sdf")))
..   where
..     #1# == 6
..            #2#

is is flexible enough that all other checks (signals, signals-not, invokes-debugger, invokes-debugger-not, fails, and in-time are built on top of it.

← ↑ → ↺ λ

6.1 Format Specifier Forms

A format specifier form is a Lisp form, typically an argument to macro, standing for the format-control and format-args arguments to the format function.

It may be a constant string:

(is nil :msg "FORMAT-CONTROL~%with no args.")
.. debugger invoked on UNEXPECTED-RESULT-FAILURE:
..   UNEXPECTED-FAILURE in check:
..     FORMAT-CONTROL
..     with no args.

It may be a list whose first element is a constant string, and the rest are the format arguments to be evaluated:

(is nil :msg ("Implicit LIST ~A." "form"))
.. debugger invoked on UNEXPECTED-RESULT-FAILURE:
..   UNEXPECTED-FAILURE in check:
..     Implicit LIST form.

Or it may be a form that evaluates to a list like (format-control &rest format-args):

(is nil :msg (list "Full ~A." "form"))
.. debugger invoked on UNEXPECTED-RESULT-FAILURE:
..   UNEXPECTED-FAILURE in check:
..     Full form.

Finally, it may evaluate to nil, in which case some context specific default is implied.

← ↑ → ↺ λ

6.2 Captures

During the evaluation of the form argument of is, evaluation of any form (e.g. a subform of form) may be recorded, which are called captures.

← ↑ → ↺ λ

6.2.1 Automatic Captures

is automatically captures some subforms of form that are likely to be informative. In particular, if form is a function call, then non-constant arguments are automatically captured:

(is (= 3 (1+ 2) (- 4 3)))
.. debugger invoked on UNEXPECTED-RESULT-FAILURE:
..   UNEXPECTED-FAILURE in check:
..     (IS (= 3 #1=(1+ 2) #2=(- 4 3)))
..   where
..     #1# = 3
..     #2# = 1

By default, automatic captures are not made for subforms deeper in form, except for when form is a call to null, endp and not:

(is (null (find (1+ 1) '(1 2 3))))
.. debugger invoked on UNEXPECTED-RESULT-FAILURE:
..   UNEXPECTED-FAILURE in check:
..     (IS (NULL #1=(FIND #2=(1+ 1) '(1 2 3))))
..   where
..     #2# = 2
..     #1# = 2
(is (endp (member (1+ 1) '(1 2 3))))
.. debugger invoked on UNEXPECTED-RESULT-FAILURE:
..   UNEXPECTED-FAILURE in check:
..     (IS (ENDP #1=(MEMBER #2=(1+ 1) '(1 2 3))))
..   where
..     #2# = 2
..     #1# = (2 3)

Note that the argument of not is not captured as it is assumed to be nil or t. If that's not true, use null.

(is (not (equal (1+ 5) 6)))
.. debugger invoked on UNEXPECTED-RESULT-FAILURE:
..   UNEXPECTED-FAILURE in check:
..     (IS (NOT (EQUAL #1=(1+ 5) 6)))
..   where
..     #1# = 6

Other automatic captures are discussed with the relevant functionality such as match-values.

← ↑ → ↺ λ

Writing Automatic Capture Rules

← ↑ → ↺ λ

6.2.2 Explicit Captures

In addition to automatic captures, which are prescribed by rewriting rules (see Writing Automatic Capture Rules), explicit, ad-hoc captures can also be made.

(is (let ((x 1))
      (= (capture x) 2)))
.. debugger invoked on UNEXPECTED-RESULT-FAILURE:
..   UNEXPECTED-FAILURE in check:
..     (IS
..      (LET ((X 1))
..        (= (CAPTURE X) 2)))
..   where
..     X = 1

If capture showing up in the form that is prints is undesirable, then % may be used instead:

(is (let ((x 1))
      (= (% x) 2)))
.. debugger invoked on UNEXPECTED-RESULT-FAILURE:
..   UNEXPECTED-FAILURE in check:
..     (IS
..      (LET ((X 1))
..        (= X 2)))
..   where
..     X = 1

Multiple values may be captured with capture-values and its secretive counterpart %%:

(is (= (%% (values 1 2)) 2))
.. debugger invoked on UNEXPECTED-RESULT-FAILURE:
..   UNEXPECTED-FAILURE in check:
..     (IS (= #1=(VALUES 1 2) 2))
..   where
..     #1# == 1
..            2

where printing == instead of = indicates that this is a multiple value capture.

← ↑ → ↺ λ

7 Check Library

In the following, various checks built on top of is are described. Many of them share a number of arguments, which are described here.

← ↑ → ↺ λ

7.1 Checking Conditions

The macros signals, signals-not, invokes-debugger, and invokes-debugger-not all check whether a condition of a given type, possibly also matching a predicate, was signalled. In addition to those already described in Check Library, these macros share a number of arguments.

Matching conditions are those that are of type condition-type (not evaluated) and satisfy the predicate pred.

When pred is nil, it always matches. When it is a string, then it matches if it is a substring of the printed representation of the condition being handled (by princ under with-standard-io-syntax). When it is a function, it matches if it returns true when called with the condition as its argument.

The check is performed in the cleanup form of an unwind-protect around body.

handler is called when a matching condition is found. It can be a function, t, or nil. When it is a function, it is called from the condition handler (signals and signals-not) or the debugger hook (invokes-debugger and invokes-debugger-not) with the matching condition. handler may perform a non-local exit. When handler is t, the matching condition is handled by performing a non-local exit to just outside body. If the exit completes, body is treated as if it had returned normally, and on-return is consulted. When handler is nil, no addition action is performed when a matching condition is found.

The default ctx describes the result of the matching process in terms of *condition-matched-p* and *best-matching-condition*.

← ↑ → ↺ λ

7.2 Miscellaneous Checks

← ↑ → ↺ λ

7.3 Check Utilities

These utilities are not checks (which signal outcomes) but simple functions and macros that may be useful for writing is checks.

← ↑ → ↺ λ

7.3.1 Comparing Floats

Float comparisons following https://randomascii.wordpress.com/2012/02/25/comparing-floating-point-numbers-2012-edition/.

← ↑ → ↺ λ

8 Tests

In Try, tests are Lisp functions that record their execution in trial objects. trials are to tests what function call traces are to functions. In more detail, tests

See deftest and with-test for more precise descriptions.

← ↑ → ↺ λ

8.1 Calling Test Functions

Tests can be run explicitly by invoking the try function or implicitly by calling a test function:

(deftest my-test ()
  (is t))

(my-test)
.. MY-TEST
..   ⋅ (IS T)
.. ⋅ MY-TEST ⋅1
..
==> #<TRIAL (MY-TEST) EXPECTED-SUCCESS 0.004s ⋅1>

The situation is similar with a with-test:

(with-test (my-test)
  (is t))
.. MY-TEST
..   ⋅ (IS T)
.. ⋅ MY-TEST ⋅1
..
==> #<TRIAL (WITH-TEST (MY-TEST)) EXPECTED-SUCCESS 0.000s ⋅1>

Behind the scenes, the outermost test function calls try with

(try trial :debug *debug* :collect *collect* :rerun *rerun*
     :print *print* :describe *describe*
     :stream *stream* :printer *printer*)

try then calls the test function belonging to trial. The rest of the behaviour is described in Explicit try.

← ↑ → ↺ λ

8.2 Explicit try

Instead of invoking the test function directly, tests can also be run by invoking the try function.

(deftest my-test ()
  (is t))

(try 'my-test)
.. MY-TEST
..   ⋅ (IS T)
.. ⋅ MY-TEST ⋅1
..
==> #<TRIAL (MY-TEST) EXPECTED-SUCCESS 0.000s ⋅1>

The situation is similar with a with-test, only that try wraps an extra trial around the execution of the lambda(0 1) to ensure that all events are signalled within a trial.

(try (lambda ()
       (with-test (my-test)
         (is t))))
.. (TRY #<FUNCTION (LAMBDA ()) {531FE50B}>)
..   MY-TEST
..     ⋅ (IS T)
..   ⋅ MY-TEST ⋅1
.. ⋅ (TRY #<FUNCTION (LAMBDA ()) {531FE50B}>) ⋅1
..
==> #<TRIAL (TRY #<FUNCTION (LAMBDA ()) {531FE50B}>) EXPECTED-SUCCESS 0.000s ⋅1>

Invoking tests with an explicit try is very similar to just calling the test functions directly (see Calling Test Functions). The differences are that try

Those arguments default to *try-debug*, *try-collect*, etc, which parallel and default to *debug*, *collect*, etc if set to :unspecified. *try-debug* is nil, the rest of them are :unspecified.

These defaults encourage the use of an explicit try call in the non-interactive case and calling the test functions directly in the interactive one, but this is not enforced in any way.

← ↑ → ↺ λ

8.2.1 Testables

Valid first arguments to try are called testables. A testable may be:

In the function designator cases, try calls the designated function. trials, being funcallable instances, designate themselves. If the trial is not runningp, then it will be rerun (see Rerunning Trials). Don't invoke try with runningp trials (but see Implementation of Implicit try for discussion).

When given a list of testables, try calls each testable one by one.

Finally, a package stands for the result of calling list-package-tests on that package.

← ↑ → ↺ λ

8.2.2 Implementation of Implicit try

What's happening in the implementation is that a test function, when it is called, checks whether it is running under the try function. If it isn't, then it invokes try with its trial. try realizes the trial cannot be rerun yet (see Rerunning Trials) because it is runningp, sets up its event handlers for debugging, collecting, printing, and invokes the trial as if it were rerun but without skipping anything based on the rerun argument. Thus the following are infinite recursions:

(with-test (recurse)
  (try recurse))

(with-test (recurse)
  (funcall recurse))

← ↑ → ↺ λ

8.3 Printing Events

try instantiates a printer of the type given by its printer argument. All events recorded by try are sent to this printer. The printer then prints events that match the type given by the print argument of try. Events that also match the describe argument of try are printed with context information (see is) and backtraces (see unhandled-error).

Although the printing is primarily customized with global special variables, changing the value of those variables after the printer object is instantiated by try has no effect. This is to ensure consistent output with nested try calls of differing printer setups.

← ↑ → ↺ λ

8.4 Counting Events

trials have a counter for each category in *categories*. When an event is recorded by try and its type matches *count*, the counters of all categories matching the event type are incremented in the current-trial. When a trial finishes and a verdict is recorded, the trial's event counters are added to that of its parent's (if any). The counts are printed with verdicts (see Printing Events).

If both *count* and *categories* are unchanged from the their default values, then only leaf events are counted, and we get separate counters for abort*, unexpected-failure, unexpected-success, skip, expected-failure, and expected-success.

(let ((*debug* nil))
  (with-test (outer)
    (with-test (inner)
      (is t))
    (is t)
    (is nil)))
.. OUTER
..   INNER
..     ⋅ (IS T)
..   ⋅ INNER ⋅1
..   ⋅ (IS T)
..   ⊠ (IS NIL)
.. ⊠ OUTER ⊠1 ⋅2
..
==> #<TRIAL (WITH-TEST (OUTER)) UNEXPECTED-FAILURE 0.000s ⊠1 ⋅2>

As the above example shows, expected-verdict-success and expected-result-success are both marked with "⋅", but only expected-result-success is counted due to *count* being leaf.

← ↑ → ↺ λ

8.5 Collecting Events

When an event is recorded and the type of the event matches the collect type argument of try, then a corresponding object is pushed onto children of the current-trial for subsequent Rerunning Trials or Reprocessing Trials.

In particular, if the matching event is a leaf, then the event itself is collected. If the matching event is a trial-event, then its trial is collected. Furthermore, trials which collected anything are always collected by their parent.

By default, both implicit and explicit calls to try collect the unexpected (see *collect* and *try-collect*), and consequently all the enclosing trials.

← ↑ → ↺ λ

8.6 Rerunning Trials

When a trial is funcalled or passed to try, the test that created the trial is invoked, and it may be run again in its entirety or in part. As the test runs, it may invoke other tests. Any test (including the top-level one) is skipped if it does not correspond to a collected trial or its trial-start event and verdict do not match the rerun argument of try. When that happens, the corresponding function call immediately returns the trial object.

← ↑ → ↺ λ

8.7 Reprocessing Trials

← ↑ → ↺ λ

9 Implementation Notes

Try is supported on ABCL, AllegroCL, CLISP, CCL, CMUCL, ECL and SBCL.

← ↑ ↺ λ

10 Glossary