1. Metaprogramming, Metaclasses
&
The Metaobject Protocol
Raymond Pierre de Lacaze
rpl@lispnyc.org
LispNYC, June 11th 2013

2. Overview
• This talk presents material from The Art of the Metaobject Protocol
by Gregor Kiczles, Jim de Rivieres, Daniel G. Bobrow, MIT Press, 1991
• Metaprogramming
– Programming with macros
– Metaprogramming in other languages
• Metaclasses
– objects, metaobjects and metaclasses
– CLOS in a nutshell
– Use case of a metaclass: AllegroCache
• Metaobject Protocol
– motivations underlying a MOP
– Part of the implementation of Closette
– Designing and Implementing a metaobject protocol

3. Metaprogramming (Wikipedia)
• Metaprogramming is the writing of computer
programs that write or manipulate other
programs, or themselves, as their data.
• The language in which the metaprogram is
written is called the metalanguage
• The ability of a programming language to be its
own metalanguage is called reflextion or
reflexivity.
• Having the programming language itself as a first-
class data type is known as homoicomicity.
– Lisp, Forth, Rebol
• Homoiconicity: Code = Data

4. Homoiconicity Example
code.lisp: (defun add (x y)(+ x y))
USER(12): (setf code (with-open-file (file "C:code.lisp") (read file)))
(defun add (x y)(+ x y))
CL-USER(13): (type-of code)
CONS
CL-USER(14): (first code)
DEFUN
CL-USER(15): (fourth code)
(+ X Y)
CL-USER(16): (setf (second code) 'subtract)
SUBTRACT
CL-USER(19): (setf (fourth code) '(- x y))
(- X Y)
CL-USER(20): code
(DEFUN SUBTRACT (X Y) (- X Y))
CL-USER(21): (eval code)
SUBTRACT
CL-USER(22): (subtract 5 6)
-1

5. Approaches to Metaprogramming
(Wikipedia)
• Statically typed functional languages
– Usage of dependent types allows proving that generated code is
never invalid.
• Template metaprogramming
– C "X Macros"
– C++ Templates
• Staged meta-programming
– MetaML
– MetaOCaml
• Macro systems
– Lisp hygienic macros
– MacroML
– Template Haskell

6. Part I: LISP MACROS

7. Why Macros?
Say we want to add a new language feature:
(unless* <test> <expr>)
;; Using a function we could do:
> (defun unless* (test expr)
(if test nil expr))
UNLESS*
> (unless* nil (print "This should print."))
This should print.
nil
> (unless* t (print "This should not print"))
This should not print
nil
Uh oh! The function unless* evaluates <expr> regardless of <test>!

8. Macros in a Nutshell
• Macros do not evaluate their arguments
• Expanded at compile-time into another
programmatic form (AST transform) as specified
by the definition of the macro
• It is this resulting programmatic form that is
actually compiled by the compiler.
• In CL can use back-quote, comma & comma-at as
convenient list manipulators to write macros
• In Clojure can use syntax-quote, tilde & tilde-at
convenient list manipulators to write macros

9. My First Macro
> (defmacro unless* (test expr)
(list 'if test nil expr))
UNLESS*
;; This works as before
> (unless* nil (println "This should print."))
This should print.
nil
;; This now behaves correctly
> (unless* t (println "This should not print"))
nil
;; You can use backquote and comma instead of explicit list constructors
> (defmacro unless* (test expr)
`(if ,test nil ,expr))
UNLESS*

10. Debugging Macros
;; This is not correct
> (defmacro INVOKE-OP (op arg1 arg2 &rest args)
`(funcall ',op ,arg1 ,arg2 ,args))
INVOKE-OP
> (macroexpand '(invoke-op * 3 4 5 6))
(funcall '* 3 4 (5 6))
;; This is correct
> (defmacro INVOKE-OP (op arg1 arg2 &rest args)
`(funcall ',op ,arg1 ,arg2 ,@args))
> (macroexpand '(invoke-op * 3 4 5 6))
(funcall '* 3 4 5 6))

11. Side Effects & Variable Capture in Macros
• When you define a macro, it is unsafe to evaluate the arguments
more than once in the event that they have side effects.
• Lisp provides gensym to allow you to define local variables in safe
way and also avoid unintentionally repeating side-effects
;; This is bad: <expr> is evaluated twice!
> (defmacro unless* (test expr)
`(let ()
(format t “~%Expr: ~a” ,expr)
(if ,test nil ,expr)))
UNLESS*
> (unless* nil (print "foo"))
"foo"
Expr: foo
"foo"
"foo"
;; This is good: Use gensym to avoid variable
;; capture and avoid repeating side-effecting forms
>(defmacro unless* (test expr)
(let ((result (gensym)))
`(let ((,result ,expr))
(format t “~%Expr: ~a” ,result)
(if ,test nil ,result))
UNLESS*
> (unless* nil (print "foo"))
"foo"
Expr: foo
"foo"

12. Recursive Macros
> (defmacro LISP ()`(,(lisp) In Summer Projects))
LISP
> (lisp)
Error: Stack overflow (signal 1000)
[condition type: SYNCHRONOUS-OPERATING-SYSTEM-SIGNAL]
>(defmacro LISP (n)
`(if (> ,n 0) `(,(lisp (1- ,n)) In Summer Projects) 'LISP)
> (lisp 0)
LISP
> (lisp 1)
(LISP IN SUMMER PROJECTS)
> (lisp 2)
((LISP IN SUMMER PROJECTS) IN SUMMER PROJECTS)
> (lisp 3)
(((LISP IN SUMMER PROJECTS) IN SUMMER PROJECTS) IN SUMMER PROJECTS)

13. Macros Summary
• Don’t use macros if you can achieve the same
thing with a simple function
• Macros are useful for implementing DSLs or
adding new language features
• Use macroexpand to debug macros
• Leverage backquote, comma, comma-at for
macro readability.
• Use gensym to write hygienic macros
• Paul Graham’s “On Lisp” is the macro book

14. Part II: Metaclasses

15. What is a Metaclass?
• In an object orient system the term object refers
the domain entities that are being modeled, e.g.
a bank-account object.
• Programmers have access to these objects
• A metaobject refers to the objects the language
uses to model/implement your domain objects ,
e.g. a bank-account-class object
• Programmers do not have access to these
metaobjects (typically)
• Only Language Implementers have access to
these metaobjects (typically)
• A metaclass is the class of a metaobject

16. Metaobject & Metaclass Example
;; In CLOS define a bank account class
> (defclass bank-account () ((account-number)(owner)(balance)))
#<STANDARD-CLASS BANK-ACCOUNT>
;; Create a bank account object
> (make-instance 'bank-account)
#<BANK-ACCOUNT {10029ED253}>
;; The class of the bank account object is a metaobject
> (class-of *)
#<STANDARD-CLASS BANK-ACCOUNT>
;; The class of this class metaobject is a metaclass
> (class-of *)
#<STANDARD-CLASS STANDARD-CLASS>

17. Basic CLOS: defclass
> (defclass bank-account ()
((account-number :initarg :account-number
:accessor bank-account-number)
(account-balance :initarg :account-balance
:accessor bank-account-balance
:initform 0)))
#<STANDARD-CLASS BANK-ACCOUNT>
> (make-instance 'bank-account :account-number 123456789 :account-balance 500)
#<BANK-ACCOUNT {1002E20513}>
> (describe *)
#<BANK-ACCOUNT {1002E20513}>
[standard-object]
Slots with :INSTANCE allocation:
ACCOUNT-NUMBER = 123456789
ACCOUNT-BALANCE = 500
> (bank-account-balance **)
500

18. Basic CLOS: defgeneric & defmethod
> (defgeneric PLUS (arg1 arg2))
#<STANDARD-GENERIC-FUNCTION PLUS (0)>
> (defmethod PLUS ((arg1 NUMBER)(arg2 NUMBER))
(+ arg1 arg2))
#<STANDARD-METHOD PLUS (NUMBER NUMBER) {100313BFF3}>
> (defmethod PLUS ((arg1 STRING)(arg2 STRING))
(concatenate 'string arg1 arg2))
#<STANDARD-METHOD PLUS (STRING STRING) {100319C953}>
> (plus 1 2)
3
> (plus "a" ”b”)
"ab”

19. AllegroCache: Object Persistence in Lisp
• AllegroCache – A high-performance, dynamic
object caching database system.
• Implements full transaction model with long and
short transactions, and meets the classic ACID
compliancy and maintains referential integrity.
• 64-bit real-time data caching
• http://www.franz.com/products/allegrocache/

20. The Persistent-Class Metaclass
> (defclass POINT ()
((name :reader point-name :index :any-unique)
(x :initarg :x :accessor point-x :index :any)
(y :initarg :y :accessor point-x :index :any)
(rho :accessor point-rho :allocation :instance)
(theta :accessor point-theta :allocation :instance))
(:metaclass db.ac::persistent-class))
#<DB.ALLEGROCACHE:PERSISTENT-CLASS POINT>
> (setf p1 (make-instance 'point :x 20 :y 30))
Error: Attempt to do a database operation without an open database
> (db.ac::open-file-database "C:ProjectstrunkLanguagesMOPDB“
:if-does-not-exist :create)
#<AllegroCache db "C:ProjectstrunkLanguagesMOPDB" @ #x218b8942>
> (setf p1 (make-instance 'point :x 20 :y 30))
#<POINT oid: 12, ver 5, trans: NIL, modified @ #x2197943a>

21. Instances of a Specialized Metaclass
> (describe p1)
#<POINT oid: 14, ver 7, trans: NIL, modified @ #x21ad2aba> is an
instance of #<DB.ALLEGROCACHE:PERSISTENT-CLASS POINT>:
The following slots have :INSTANCE allocation:
OID 14
SLOTS #((:UNBOUND) (:UNBOUND) (:UNBOUND))
NEW-SLOTS #(POINT-1 20 30)
DCLASS #<DB.ALLEGROCACHE::FILE-DASH-CLASS @ #x219796a2>
VERSION 7
PREV-VERSION NIL
TRANS NIL
MODIFIED :END
RHO <unbound>
THETA <unbound>
The following slots have nonstandard allocation as shown:
NAME :PERSISTENT POINT-1
X :PERSISTENT 20
Y :PERSISTENT 30

22. Metaclass
• Metaclasses allow you to extend existing
language constructs. E.g. defclass
• This can be more natural than a DSL or a
separate API outside the language
• Allow you to safely change or extend the
behavior of the language
• Metaclasses are a very powerful underutilized
programming paradigm. (IMHO)

23. Part III: The Metaobject Protocol

24. Protocol Definitions
• An object protocol is a collection of methods
that operate on some collection of objects.
• A metaobject protocol is a collection of
methods that operate on some collection of
metaobjects.
• A metaclass refers to the class of a metaobject

25. Implementing defclass
;;; All the code on the next few slides is taken from the AMOP book.
(defmacro DEFCLASS (name direct-superclasses direct-slots &rest options)
`(ensure-class ',name
:direct-superclasses ,(canonicalize-direct-superclasses direct-superclasses)
:direct-slots ,(canonicalize-direct-slots direct-slots)
,@(canonicalize-defclass-options options)
(defun ENSURE-CLASS (name &rest all-keys)
(if (find-class name nil)
(error "Can't redefine the class named ~S in Closette." name)
(let ((class (apply #'make-instance 'standard-class :name name all-keys)))
(setf (find-class name) class)
class)))
(let ((class-table (make-hash-table :test #'eq :size 20)))
(defun (SETF FIND-CLASS) (new-value symbol)
(setf (gethash symbol class-table) new-value))
)

26. Implementing defclass (cont.)
(defclass STANDARD-CLASS ()
((name :initarg :name
:accessor class-name)
(direct-superclasses :initarg :direct-superclasses
:accessor class-direct-superclasses)
(direct-slots :initarg :direct-slots
:accessor class-direct-slots)
(class-precedence-list :initarg :class-precedence-list
:accessor class-precedence-list)
(effective-slots :accessor class-slots)
(direct-subclasses :initform ()
:accessor class-direct-subclasses)
(direct-methods :initform ()
:accessor class-direct-methods)))

27. Implementing defclass (cont.)
(defmethod INITIALIZE-INSTANCE :after ((class STANDARD-CLASS)
&key direct-superclasses direct-slots)
(let ((supers (or direct-superclasses `(,(find-class 'standard-object)))))
(setf (class-direct-superclasses class) supers)
(dolist (superclass supers)
(push class (class-direct-subclasses superclass))))
(let ((slots (mapcar #'(lambda (slot-properties)
(apply #'make-direct-slot-definition slot-properties))
direct-slots)))
(setf (class-direct-slots class) slots)
(dolist (direct-slot slots)
(dolist (reader (slot-definition-readers direct-slot))
(add-reader-method class reader (slot-definition-name direct-slot))))
(dolist (direct-slot slots)
(dolist (writer (slot-definition-writers direct-slot))
(add-writer-method class writer (slot-definition-name direct-slot)))))
(finalize-inheritance))

28. Implementing defclass (cont.)
(defun FINALIZE-INHERITANCE (class)
(setf (class-precedence-list class)(compute-class-precedence-list))
(setf (class-slots class) (compute-slots class)))
(defun COMPUTE-CLASS-PRECEDENCE-LIST (class)
(let ((classes-to-order (collect-superclasses* class)))
(topological-sort classes-to-order
(remove-duplicates
(mapappend #'local-precedence-ordering classes-to-order))
#'std-tie-breaker-rule)))
(defun COLLECT-SUPERCLASSES* (class)
(remove-duplicates
(cons class
(mapappend #'collect-superclasses* (class-direct-superclasses class)))))

29. Metaobject Protocol Definition
• A metaobject protocol is an API that allows you to
access and manipulate metaobjects.
• “A metaobject protocol is an interface to elements of a
language that are normally hidden from users of the
language. Providing such an interface allows users of
the language to tailor the language to their own needs
and in such a way provides a more powerful and
flexible language. The CLOS metaobject protocol
provides an interface to program elements such as
classes and methods, thereby allowing users to control
aspects of the language such as how instances of a
class are created or how method dispatching works”.
Raymond de Lacaze, JLUGM, 2000

30. Introspective vs. Intercessory
• AMOP has both introspective and intercessory
aspects.
• Introspective
– Allows programmers to see the on-backstage.
Metaobjects can be created, retrieved and examined.
• Intercessory
– Allows programmers to manipulate the on-backstage
thereby affecting the on-stage behavior

32. Example MOP Usage
;; Usage Example 1: An instance-counting metaclass
;; Provide a slot in the metaclass to track number of instances
(defclass COUNTED-CLASS (STANDARD-CLASS)
((counter :initform 0)))
;; Then add an :after method on make-instance that bumps count
(defmethod MAKE-INSTANCE :after ((class COUNTED-CLASS) &key)
(incf (slot-value class) ‘counter))

33. Example MOP Usage (cont.)
• Need to define a class with a metaclass other than
standard-class.
• We can’t use defclass, because as previously written it
always uses standard-class as the metaclass
(setf (find-class ‘counted-rectangle)
(make-instance ‘counted-class
:name ‘counted-rectangle
:direct-superclasses (list (find-class ‘rectangle))
:direct-slots nil))

34. Extending defclass Implementation
;;; We would like to simply write the following
(defclass COUNTED-RECTANGLE (rectangle)
()
(:meta-class counted-class))
;; Simply need to extend the definition of ensure-class
(defun ENSURE-CLASS (name &rest all-keys
&key (metaclass (find-class ‘standard-class)))
(if (find-class name nil)
(error "Can't redefine the class named ~S in Closette." name)
(let ((class (apply #'make-instance metaclass :name name all-keys)))
(setf (find-class name) class)
class)))

35. Understanding Method Combination
> (defclass a ()())
#<STANDARD-CLASS A>
> (defclass b (a)())
#<STANDARD-CLASS B>
> (defclass c (a b)())
#<STANDARD-CLASS C>
a b
c
• Add primary methods on a, b, c
• Add :before method on a, b, c
• Add :after method on a, b, c
• Add :around methods on a, b, c

36. Adding All Possible Methods
(defmethod FOO ((obj A))
(format t "~%Primary method on class A")
(call-next-method))
(defmethod FOO ((obj B))
(format t "~%Primary method on class B"))
(defmethod FOO ((obj C))
(format t "~%Primary method on class C")
(call-next-method))
(defmethod FOO :before ((obj A))
(format t "~%Before method on class A"))
(defmethod FOO :before ((obj B))
(format t "~%Before method on class B"))
(defmethod FOO :before ((obj C))
(format t "~%Before method on class C"))
(defmethod FOO :after ((obj A))
(format t "~%After method on class A"))
(defmethod FOO :after ((obj B))
(format t "~%After method on class B"))
(defmethod FOO :after ((obj C))
(format t "~%After method on class C"))
(defmethod FOO :around ((obj A))
(format t "~%Around method (before) on class A")
(call-next-method)
(format t "~%Around method (after) on class A"))
(defmethod FOO :around ((obj B))
(format t "~%Around method (before) on class B")
(call-next-method)
(format t "~%Around method (after) on class B"))
(defmethod FOO :around ((obj C))
(format t "~%Around method (before) on class C")
(call-next-method)
(format t "~%Around method (after) on class C"))
> (setf x (make-instance 'c))
#<C @ #x209681b2>

37. CLOS Method Combination
CL-USER(45): (foo x)
Around method (before) on class C
Around method (before) on class A
Around method (before) on class B
Before method on class C
Before method on class A
Before method on class B
Primary method on class C
Primary method on class A
Primary method on class B
After method on class B
After method on class A
After method on class C
Around method (after) on class B
Around method (after) on class A
Around method (after) on class C
NIL

38. Changing Method Combination
(defclass a ()())
(defclass b ()())
(defclass c ()())
(defclass s (a b)())
(defclass r (a c)())
(defclass q (r s)())
• Different Lisp object systems used different method combination
• How can we reuse code written in Flavors, in CLOS?
• We would like to be able to change method combination order
Flavors: (q s a b r c standard-object t)
Loops: (q s b r a c standard-object t)
CLOS: (q s r a b c standard-object t)
a b c
s r
q

39. Compute-Class-Precedence-List
• Do not allow programmers to directly set class-precedence-list slot of the class metaobject but.
• Allow them to control the computation that is used when that slot I set
• Make compute-class-precedence-list a GF and specialize the original function on standard-class
• Require that the class be the first in the list and that standard-object and t be the last two.
(defclass flavors-class (standard-class) ())
(defmethod compute-class-precedence ((class flavors-class))
(append (remove-duplicates (depth-first-preorder-superclasses* class)
:from-end t)
(list (find-class ‘standard-object)
(find-class ‘t))))

40. Additional AMOP Examples
• Adding slot attributes
– compute-slots (GF)
– compute-effective-slots (GF)
• Adding default-initargs
– ensure-class
– make-instance
– finalize-inheritance (GF)
• Instance Allocation
– allocate-instance (GF)
Note: In the examples we’ve seen so far we’ve had to rewrite ensure-class twice

41. Designing a MOP
• Design revolves around CLOS metaobjects
– Class metaobjects
– Method metaobject
– Generic function metaobjects
– Slot definition metaobjects
– Specializer metaobjects
– Method combination objects
• AMOP is designed to provide an introspective and
intercessory API for manipulating and
customizing these metaobjects

42. Designing a MOP
• Design: MOP should provide mechanisms to extend the syntax and
manipulate the behavior of CLOS without:
– Needing to rewrite the programmer user interface macros: defclass,
defgeneric and defmethod
– Needing to rewrite the functions that implement these macros, namely the
ensure-x functions and the compute-x functions
• Implementation
– This is mostly accomplished by making the compute-X functions (on the
various metaobjects) generic functions, thus using CLOS to implement the
implementation of CLOS.
– This approach also provides a level of safety because users of the MOP cannot
directly set the values of the slots of the metaobjects, they can only control
their computation which is only performed once and cannot subsequently
changed.

43. The Final defclass
(defmacro DEFCLASS (name direct-superclasses direct-slots &rest options)
(let* ((metaclass-option (find ‘:metaclass options :key #’car))
(metaclass-name (or (second metaclass-option) ‘standard-class))
(sample-class-metaobject
(allocate-instance (find-class metaclass-name)))
(canonical-supers
(canonicalize-direct-superclasses direct-superclasses))
(canonical-slots
(canonicalize-direct-slots direct-slots))
(canonical-options
(canonicalize-defclass-options sample-class-metaobject
(remove metaclass-option options))))
`(ensure-class ‘,name
:direct-superclasses ,canonical-supers
: direct-slots ,canonical-slots
:metaclass (find-class ‘,metaclass)
,@canonical-options)))

44. CLOS Metaclass Hierarchy

45. When to use a MOP
• Any time you need to alter how a metaobject behaves.
For instance, instead of doing a method lookup on a
local table, perform an RPC call.
• Memoizing method calls
• Complex inheritance (this may be of dubious value)
• Loosening or strengthening a type system
• Implementing Dynamic Dispatch (if necessary)
• Persistence
• Replication
This list is from: http://community.schemewiki.org/?meta-object-protocol

46. Additional Links
• http://en.wikipedia.org/wiki/Metaprogramming
• https://en.wikipedia.org/wiki/Metaclass
• http://en.wikipedia.org/wiki/Metaobject
• http://www.dreamsongs.com/CLOS.html
• http://www.franz.com/services/conferences_seminars/jlugm00/conference/Talk02_deLacaze1.pdf
• http://www.cs.cmu.edu/afs/cs/project/ai-repository/ai/lang/lisp/oop/clos/closette/0.html
• http://www.cs.cmu.edu/afs/cs/project/ai-repository/ai/lang/lisp/doc/standard/ansi/mop/
• http://community.schemewiki.org/?meta-object-protocol
• http://www.gnu.org/software/guile/manual/html_node/GOOPS.html
• http://clojure.org/multimethods

47. Additional Definitions
• Dependent Types (Wikipedia)
In computer science and logic, a dependent type is a type that depends on a value. Dependent types
play a central role in intuitionistic type theory and in the design of functional programming languages
like ATS, Agda and Epigram and Idris
An example is the type of n-tuples of real numbers. This is a dependent type because the type depends
on the value n.
• Staged meta-programming (Wikipedia)
Incremental compiling of new machine code during runtime. Under certain circumstances, significant
speedups are possible using multi-stage programming, because more detailed information about the
data to process is available at runtime than at the regular compile time, so the incremental compiler can
optimize away many cases of condition checking etc.