→ ↺ λ

Cube Manual

Table of Contents

• 1 Links
• 2 Introduction
• 3 Basics
• 4 Synchronization
• 5 Facets
• 6 Facet Extension API
• 7 The Default Implementation of CALL-WITH-FACET*
• 8 Lifetime
  • 8.1 Facet Barriers

[in package MGL-CUBE]

← ↑ → ↺ λ

1 Links

Here is the official repository and the HTML documentation for the latest version.

← ↑ → ↺ λ

2 Introduction

This is the library on which MGL-MAT (see MAT Manual) is built. The idea of automatically translating between various representations may be useful for other applications, so this got its own package and all ties to MGL-MAT has been severed.

This package defines CUBE, an abstract base class that provides a framework for automatic conversion between various representations of the same data. To define a cube, CUBE needs to be subclassed and the Facet Extension API be implemented.

If you are only interested in how to use cubes in general, read Basics, Lifetime and Facet Barriers.

If you want to implement a new cube datatype, then see Facets, Facet Extension API, and The Default Implementation of CALL-WITH-FACET*.

← ↑ → ↺ λ

3 Basics

Here we learn what a CUBE is and how to access the data in it with WITH-FACET.

• [class] CUBE

  A datacube that has various representations of the same stuff. These representations go by the name `facet'. Clients must use WITH-FACET to acquire a dynamic extent reference to a facet. With the information provided in the DIRECTION argument of WITH-FACET, the cube keeps track of which facets are up-to-date and copies data between them as necessary.

  The cube is an abstract class, it does not provide useful behavior in itself. One must subclass it and implement the Facet Extension API.

  Also see Lifetime and Facet Barriers.

• [macro] WITH-FACET (VAR (CUBE FACET-NAME &KEY (DIRECTION :IO) TYPE)) &BODY BODY

  Find or create the facet with FACET-NAME in CUBE and bind VAR to the representation of CUBE's data provided by that facet. This representation is called the facet's value. The value is to be treated as dynamic extent: it is not allowed to keep a reference to it. For the description of the DIRECTION parameter, see the type DIRECTION.

  If TYPE is specified, then VAR is declared to be of that type.

• [type] DIRECTION

  Used by WITH-FACET, DIRECTION can be :INPUT, :OUTPUT or :IO.

  • :INPUT promises that the facet will only be read and never written. Other up-to-date facets of the same cube remain up-to-date. If the facet in question is not up-to-date then data is copied to it from one of the up-to-date facets (see SELECT-COPY-SOURCE-FOR-FACET*).

  • :OUTPUT promises that all data will be overwritten without reading any data. All up-to-date facets become non-up-to-date, while this facet is marked as up-to-date. No copying of data takes place.

  • :IO promises nothing about the type of access. All up-to-date facets become non-up-to-date, while this facet is marked as up-to-date. If the facet in question is not up-to-date then data is copied to it from one of the up-to-date facets (see SELECT-COPY-SOURCE-FOR-FACET*).

  Any number of WITH-FACETs with direction :INPUT may be active at the same time, but :IO and :OUTPUT cannot coexists with another WITH-FACET regardless of the direction. The exception for this rule is that an inner WITH-FACET does not conflict with an enclosing WITH-FACET if they are for the same facet (but inner WITH-FACETs for another facet or for the same facet from another thread do).

  See CHECK-NO-WRITERS and CHECK-NO-WATCHERS called by The Default Implementation of CALL-WITH-FACET*.

• [macro] WITH-FACETS (&REST FACET-BINDING-SPECS) &BODY BODY

  A shorthand for writing nested WITH-FACET calls.

  (with-facet (f1 (c1 'name1 :direction :input))
    (with-facet (f2 (c2 'name2 :direction :output))
      ...))

  is equivalent to:

  (with-facets ((f1 (c1 'name1 :direction :input))
                (f2 (c2 'name2 :direction :output)))
    ...)

← ↑ → ↺ λ

4 Synchronization

Cubes keep track of which facets are used, which are up-to-date to be able to perform automatic translation between facets. WITH-FACET and other operations access and make changes to this metadata so thread safety is a concern. In this section, we detail how to relax the default thread safety guarantees.

A related concern is async signal safety which arises most often when C-c'ing or killing a thread or when the extremely nasty WITH-TIMEOUT macro is used. In a nutshell, changes to cube metadata are always made with interrupts disabled so things should be async signal safe.

• [accessor] SYNCHRONIZATION CUBE (:SYNCHRONIZATION = *DEFAULT-SYNCHRONIZATION*)

  By default, setup and teardown of facets by WITH-FACET is performed in a thread safe way. Corrupting internal data structures of cubes is not fun, but in the name of performance, synchronization can be turned off either dynamically or on a per instance basis.

  If T, then access to cube metadata is always synchronized. If NIL, then never. If :MAYBE, then whether access is synchronized is determined by *MAYBE-SYNCHRONIZE-CUBE* that's true by default.

  The default is the value of *DEFAULT-SYNCHRONIZATION* that's :MAYBE by default.

  Note that the body of a WITH-FACET is never synchronized with anyone, apart from the implicit reader/writer conflict (see DIRECTION).

• [variable] *DEFAULT-SYNCHRONIZATION* :MAYBE

  The default value for SYNCHRONIZATION of new cubes.

• [variable] *MAYBE-SYNCHRONIZE-CUBE* T

  Determines whether access the cube metadata is synchronized for cubes with SYNCHRONIZATION :MAYBE.

← ↑ → ↺ λ

5 Facets

The basic currency for implementing new cube types is the FACET. Simply using a cube only involves facet names and values, never facets themselves.

• [function] FACETS CUBE

  Return the facets of CUBE.

• [function] FIND-FACET CUBE FACET-NAME

  Return the facet of CUBE for the facet with FACET-NAME or NIL if no such facet exists.

• [class] FACET STRUCTURE-OBJECT

  A cube has facets, as we discussed in Basics. Facets holds the data in a particular representation, this is called the value of the facet. A facet holds one such value and some metadata pertaining to it: its FACET-NAME(0 1), whether it's up-to-date (FACET-UP-TO-DATE-P), etc. FACET objects are never seen when simply using a cube, they are for implementing the Facet Extension API.

• [structure-accessor] FACET-NAME

  A symbol that uniquely identifies the facet within a cube.

• [structure-accessor] FACET-VALUE

  This is what's normally exposed by WITH-FACET.

• [structure-accessor] FACET-DESCRIPTION

  Returned by MAKE-FACET* as its second value, this is an arbitrary object in which additional information can be stored.

• [structure-accessor] FACET-UP-TO-DATE-P

  Whether the cube has changed since this facet has been last updated. See FACET-UP-TO-DATE-P*.

• [structure-accessor] FACET-N-WATCHERS

  The number of active WITH-FACETs. Updated by WATCH-FACET and UNWATCH-FACET.

• [structure-accessor] FACET-WATCHER-THREADS

  The threads (one for each watcher) that have active WITH-FACETs.

• [structure-accessor] FACET-DIRECTION

  The direction of the last WITH-FACET on this facet.

← ↑ → ↺ λ

6 Facet Extension API

Many of the generic functions in this section take FACET arguments. FACET is a structure and is not intended to be subclassed. To be able to add specialized methods, the name of the facet (FACET-NAME) is also passed as the argument right in front of the corresponding facet argument.

In summary, define EQL specializers on facet name arguments, and use FACET-DESCRIPTION to associate arbitrary information with facets.

• [generic-function] MAKE-FACET* CUBE FACET-NAME

  Called by WITH-FACET (or more directly WATCH-FACET) when there is no facet with FACET-NAME. As the first value, return a new object capable of storing CUBE's data in the facet with FACET-NAME. As the second value, return a facet description which will be available as FACET-DESCRIPTION. As the third value, return a generalized boolean indicating whether this facet must be explicitly destroyed (in which case a finalizer will be added to CUBE).

• [generic-function] DESTROY-FACET* FACET-NAME FACET

  Free the resources associated with FACET with FACET-NAME. The cube this facet belongs to is not among the parameters because this method can be called from a finalizer on the cube (so we can't have a reference to the cube portably) which also means that it may run in an unpredictable thread.

• [generic-function] COPY-FACET* CUBE FROM-FACET-NAME FROM-FACET TO-FACET-NAME TO-FACET

  Copy the CUBE's data from FROM-FACET with FROM-FACET-NAME to TO-FACET with TO-FACET-NAME. Called by WITH-FACET (or more directly WATCH-FACET) when necessary. FROM-FACET is what SELECT-COPY-SOURCE-FOR-FACET* returned.

• [generic-function] CALL-WITH-FACET* CUBE FACET-NAME DIRECTION FN

  Call FN with an up-to-date FACET-VALUE that belongs to FACET-NAME of CUBE. WITH-FACET is directly implemented in terms of this function. See The Default Implementation of CALL-WITH-FACET* for the gory details.

  Specializations will most likely want to call the default implementation (with CALL-NEXT-METHOD) but with a lambda that transforms FACET-VALUE before passing it on to FN.

• [generic-function] FACET-UP-TO-DATE-P* CUBE FACET-NAME FACET

  Check if FACET with FACET-NAME has been updated since the latest change to CUBE (that is, since the access to other facets with DIRECTION of :IO or :OUTPUT). The default method simply calls FACET-UP-TO-DATE-P on FACET.

  One reason to specialize this is when some facets actually share common storage, so updating one make the other up-to-date as well.

• [generic-function] SELECT-COPY-SOURCE-FOR-FACET* CUBE TO-NAME TO-FACET

  Called when TO-FACET with TO-NAME is about to be updated by copying data from an up-to-date facet. Return the facet (or its name) from which data shall be copied. Note that if the returned facet is not FACET-UP-TO-DATE-P, then it will be updated first and another SELECT-COPY-SOURCE-FOR-FACET will take place, so be careful not to get into endless recursion. The default method simply returns the first up-to-date facet.

PAX integration follows, don't worry about it if you don't use PAX, but you really should (see MGL-PAX:@MGL-PAX-MANUAL).

• [locative] FACET-NAME

  The FACET-NAME [locative] is the to refer to stuff defined with DEFINE-FACET-NAME.

• [macro] DEFINE-FACET-NAME SYMBOL LAMBDA-LIST &BODY DOCSTRING

  Just a macro to document that SYMBOL refers to a facet name (as in the FACET-NAME). This is totally confusing, so here is an example of how MGL-MAT (see MAT Manual) documents the MGL-MAT:BACKING-ARRAY facet:

  (define-facet-name backing-array ()
    "The corresponding facet is a one dimensional lisp array.")

  Which makes it possible to refer to this definition (refer as in link and M-. to) MGL-MAT:BACKING-ARRAY facet-name. See MGL-PAX:@MGL-PAX-MANUAL for more.

Also see The Default Implementation of CALL-WITH-FACET*.

← ↑ → ↺ λ

7 The Default Implementation of CALL-WITH-FACET*

• [method] CALL-WITH-FACET* (CUBE CUBE) FACET-NAME DIRECTION FN

  The default implementation of CALL-WITH-FACET* is defined in terms of the WATCH-FACET and the UNWATCH-FACET generic functions. These can be considered part of the Facet Extension API.

• [generic-function] WATCH-FACET CUBE FACET-NAME DIRECTION

  This is what the default CALL-WITH-FACET* method, in terms of which WITH-FACET is implemented, calls first. The default method takes care of creating facets, copying and tracking up-to-dateness.

  Calls CHECK-NO-WRITERS (unless *LET-INPUT-THROUGH-P*) and CHECK-NO-WATCHERS (unless *LET-OUTPUT-THROUGH-P*) depending on DIRECTION to detect situations with a writer being concurrent to readers/writers because that would screw up the tracking of up-to-dateness.

  The default implementation should suffice most of the time. MGL-MAT specializes it to override the DIRECTION arg, if it's :OUTPUT but not all elements are visible due to reshaping, so that invisible elements are still copied over.

• [generic-function] UNWATCH-FACET CUBE FACET-NAME

  This is what the default CALL-WITH-FACET* method, in terms of which WITH-FACET is implemented, calls last. The default method takes care of taking down facets. External resource managers may want to hook into this to handle unused facets.

• [variable] *LET-INPUT-THROUGH-P* NIL

  If true, WITH-FACETS (more precisely, the default implementation of CALL-WITH-FACET*) with :DIRECTION :INPUT does not call CHECK-NO-WRITERS. This knob is intended to be bound locally for debugging purposes.

• [variable] *LET-OUTPUT-THROUGH-P* NIL

  If true, WITH-FACETS (more precisely, the default implementation of CALL-WITH-FACET*) with :DIRECTION :IO or :OUTPUT does not call CHECK-NO-WATCHERS. This knob is intended to be bound locally for debugging purposes.

• [function] CHECK-NO-WRITERS CUBE FACET-NAME MESSAGE-FORMAT &REST MESSAGE-ARGS

  Signal an error if CUBE has facets (with names other than FACET-NAME) being written (i.e. direction is :IO or :OUTPUT).

• [function] CHECK-NO-WATCHERS CUBE FACET-NAME MESSAGE-FORMAT &REST MESSAGE-ARGS

  Signal an error if CUBE has facets (with names other than FACET-NAME) being regardless of the direction.

← ↑ → ↺ λ

8 Lifetime

Lifetime management of facets is manual (but facets of garbage cubes are freed automatically by a finalizer, see MAKE-FACET*). One may destroy a single facet or all facets of a cube with DESTROY-FACET and DESTROY-CUBE, respectively. Also see Facet Barriers.

• [function] DESTROY-FACET CUBE FACET-NAME

  Free resources associated with the facet with FACET-NAME and remove it from FACETS of CUBE.

• [function] DESTROY-CUBE CUBE

  Destroy all facets of CUBE with DESTROY-FACET.

In some cases it is useful to declare the intent to use a facet in the future to prevent its destruction. Hence, every facet has reference count which starts from 0. The reference count is incremented and decremented by ADD-FACET-REFERENCE-BY-NAME and REMOVE-FACET-REFERENCE-BY-NAME, respectively. If it is positive, then the facet will not be destroyed by explicit DESTROY-FACET and DESTROY-CUBE calls, but it will still be destroyed by the finalizer to prevent resource leaks caused by stray references.

• [function] ADD-FACET-REFERENCE-BY-NAME CUBE FACET-NAME

  Make sure FACET-NAME exists on CUBE and increment its reference count. Return the FACET behind FACET-NAME.

• [function] REMOVE-FACET-REFERENCE-BY-NAME CUBE FACET-NAME

  Decrement the reference count of the facet with FACET-NAME of CUBE. It is an error if the facet does not exists or if the reference count becomes negative.

• [function] REMOVE-FACET-REFERENCE FACET

  Decrement the reference count of FACET. It is an error if the facet is already destroyed or if the reference count becomes negative. This function has the same purpose as REMOVE-FACET-REFERENCE-BY-NAME, but by having a single FACET argument, it's more suited for use in finalizers because it does not keep the whole CUBE alive.

← ↑ ↺ λ

8.1 Facet Barriers

A facility to control lifetime of facets tied to a dynamic extent. Also see Lifetime.

• [macro] WITH-FACET-BARRIER (CUBE-TYPE ENSURES DESTROYS) &BODY BODY

  When BODY exits, destroy facets which:

  • are of cubes with CUBE-TYPE

  • have a facet name among DESTROYS

  • were created in the dynamic extent of BODY

  Before destroying the facets, it is ensured that facets with names among ENSURES are up-to-date. WITH-FACET-BARRIERs can be nested, in case of multiple barriers matching the cube's type and the created facet's name, the innermost one takes precedence.

  The purpose of this macro is twofold. First, it makes it easy to temporarily work with a certain facet of many cubes without leaving newly created facets around. Second, it can be used to make sure that facets whose extent is tied to some dynamic boundary (such as the thread in which they were created) are destroyed.

• [function] COUNT-BARRED-FACETS FACET-NAME &KEY (TYPE 'CUBE)

  Count facets with FACET-NAME of cubes of TYPE which will be destroyed by a facet barrier.