Foreign Types

The purpose of the foreign types package is to permit the creation, reading and modification of objects that are described in non-lisp terms. By non-lisp we generally mean C or C++.

History

The need for this package came about from the porting of Common Graphics to Acl version 5.0. Common Graphics was built using a very different foreign types package.    It wasn't possible to convert Common Graphics foreign type calls to Acl 4.3 foreign type calls without losing efficiency.  Thus for Acl 5.0 we designed the a new foreign types package, one which can support the needs of Common Graphics (through simple macro transformation of the old calls), one which can open code to generate efficient code, and one with the features of the acl4.3 foreign types package, and one which supports the features of Lisp introduced in acl5.0.

The cstruct object (known officially as (array excl::foreign) has been enhanced so that the first slot contains a lisp object. Futhermore, make-array and the garbage collector have been enhanced to allow lisp-valued arrays to be stored in static space. Combining these two enhancements, you get the ability to use cstructs to be stored in static space and for those cstructs to contain a pointer to their lisp class (in the first slot).

The foreign types package

The foreign types package tries to blend the best of the aclunix and aclwin foreign types package.

Examples

To get an idea of how this package works, here are some examples. First we show how we can define, allocate, set and access values in a foreign structure

;; define the structure
user(3): (ff::def-foreign-type my-point (:struct (x :int) (y :int)))
#<foreign-functions::foreign-structure my-point>
;; allocate an object, using the default allocation type of :foreign
user(4): (setq obj (ff:allocate-fobject 'my-point))
#<foreign object of class my-point>
;; set a slot in the object
user(5): (setf (ff:fslot-value obj 'x) 3)
3
;; verify that the slot is set with the correct value
user(6): (ff:fslot-value obj 'x)
3

The def-foreign-type macro defines the my-point structure and returns the clos class that was defined. Note that the metaclass of a foreign structure is ff:foreign-structure.

Next an object is allocated with allocate-fobject. We didn't specify an allocation type, thus the type :foreign was used. A :foreign object is stored in the lisp heap in an (array excl:foreign) object (which is commonly called cstruct object).  A nice feature of a :foreign object is that it is typed. You can use that type to specialize on objects of this foreign type in clos generic functions.

Another advantage of a foreign object being typed is that in the (setf fslot-value) and fslot-value calls we didn't have to specify the type of obj. The type was automatically determined at runtime.

Runtime determination of the type is handy and enhances the safety of the program since checks will be made at runtime to ensure that the desired access is appropriate for the object given. There is a cost to this check though, and if the foreign structure access is to be done many times, you'll want to use an accessor that allows you to specify the type at compile time:

user(9): (setf (ff:fslot-value-typed 'my-point :foreign obj 'x) 3)
3
user(10): (ff:fslot-value-typed 'my-point :foreign obj 'x)
3
user(11): 

The fslot-value-typed function takes two extra arguments: a type and an allocation method. With certain settings of the optimization values (safety, size, space, speed), the compiler will generate code to do the access in a few machine instructions.

Allocations and accesses can be done of types that have no name. These are called anonymous types.

user(15): (setq obj (ff:allocate-fobject '(:struct (x :int) (y :int))))
#(#(18942245 t 901 nil nil 8 nil) 0 0)
user(16): (setf (ff:fslot-value-typed '(:struct (x :int) (y :int)) :foreign obj 'x) 234)
234
user(17): (ff:fslot-value-typed '(:struct (x :int) (y :int)) :foreign obj 'x)
234

When you use anonymous types, you must use fslot-value-typed. This may be relaxed in the future to permit fslot-value to be used.

The Syntax for Foreign Types

The type syntax of C is mostly postfix with occasional prefix bits. Also, C tries to get by describing structures using the fewest numbers of characters, and that doesn't always make things readable. Previous foreign types packages have tried to mimic the C syntax, leading sometimes to even more confusion since they couldn't mimic it exactly. This package uses prefix syntax exclusively and is a bit more verbose where that is warranted. The syntax for a foreign type (ftype below) is described next.

ftype := scalar-type
	 composite-type
	 function-type
	 user-type

primitive-type :=   :int
	            :long
	            :short
	            :char
		    :void
	            :unsigned-int
	            :unsigned-long
	            :unsigned-short
	            :unsigned-char
                    :float
                    :double
	       
scalar-type :=  primitive-type
		(* ftype)
		(& ftype)
                (:aligned ftype)

composite-type := (:struct sfield ...)
		  (:class  field ...)  
		  (:union  field ...)
		  (:array  ftype  [dim ...])
		  

function-type  := (:function (ftype ...) ftype [attributes])

user-type  := <symbol>      [where <symbol> has an associated foreign type]
    
dim := <positive integer>

field := ftype
	 (field-name ftype)
[fields in structures can contain bit specifiers]
sfield := field
          bit-specifier
          multibit-specifier
bit-specifier :=
          (:bit number-of-bits)
          (field-name (:bit number-of-bits))
number-of-bits := <integer>
multibit-specifier :=
	  (:bits number-of-bytes bit-specifier ...)
          (field-name (:bits number-of-bytes bit-specifier ...))
number-of-bytes := <integer>

Some notes on the syntax above

Primitive Types

The sizes of the primitives types vary by machine architecture, as this table shows

Type Size Alignment Notes
:void 0 0 Used only in (* :void) type specifications
:char 1 1 A signed one byte access
:unsigned-char 1 1  
:short 2 2 A signed two byte access
:unsigned-short 2 2  
:int 4 4 A signed four byte access
:unsigned-int 4 4  
:long 4 on all machines except the Dec Alpha, where it is 8 4 on all machines except the Dec Alpha, where it is 8 A signed access of an architecture specific size.
:unsigned-long 4 on all machines except the Dec Alpha, where it is 8 4 on all machines except the Dec Alpha, where it is 8  
:float 4 4  
:double 8 8 on all machines except the rs/6000 and Linux on an x86 where it is 4  

Allocation types

Objects can be allocated in a variety of places. The default allocation location is :foreign.

Aligned Pointers

Lisp can reference data stored in the Lisp heap or outside the heap in what we call C-space.   Objects in C-space are normally referenced by their addresses, and on many plaforms addresses in C- space are so large they must be represented as bignums.   In most programs the overhead of allocating a bignum to represent a C-space object isn't a big factor in the overall speed of the program.   In those cases where the allocation is a problem we offer the aligned pointer to a C-space object. 

If the address of the C-space object is a multiple of four, then we can divide the address by four and its value will be small enough to be represented by  a lisp fixnum object.   We call such a fixnum an aligned pointer.    There is no space overhead in allocating a fixnum.

It's impossible for Lisp to distinguish an aligned pointer from a normal C-space pointer.   Thus when an aligned pointer is used, the programmer must specify that the value is an aligned pointer.

There are two ways to create an aligned pointer: allocating an object with an :aligned allocation type or  referencing a slot of a foreign object that is declared of type (:aligned some-type).    Next, we'll show these cases in detail.

Given any foreign type foo we can allocate an aligned pointer to it with

(allocate-fobject 'foo :aligned)

The return value will always be a fixnum.

Suppose we have types point and rect:

(def-foreign-type point
    (:struct (x :int)
             (y :int)))
(def-foreign-type rect
    (:struct  (topleft (* point))
              (bottomright (:aligned point))))

Suppose the variable rr contains a pointer to a rect object.    We'll further assume that the pointer to the rect object was passed back to us from a C program and that the pointer is a normal :c (not aligned) pointer.  We can access the x slot of the topleft and bottomright fields using the same kind of expression:

(fslot-value-typed 'rect :c rr :topleft '* :x)
(fslot-value-typed 'rect :c rr :bottomright '* :x)

This shows that you can treat the (* ftype) and (:aligned ftype) specifier the same when you're referencing objects through them.

If you just access the pointer values, you'll see big differences.

(fslot-value-typed 'rect :c rr :topleft)

is a normal :c pointer whereas

(fslot-value-typed 'rect :c rr :bottomright)

is an :aligned pointer.

Using aligned pointers requires careful programming.  Here are the rules for using aligned pointers:

  1. If an aligned pointer is used in fslot-value-typed or (setf fslot-value-typed) then the allocation type of :aligned must be specified.
  2. Setting a slot of a object declared to be (:aligned some-type) must be done with an aligned pointer.  The function address-to-aligined is useful in creating an aligned pointer from a normal :c pointer.
  3. When accessing a slot of an object declared as (:aligned some-type), the pointer contained therein must have its low two bits zero. Failure to abide by this will likely result in illegal lisp object pointers being stored in the heap, which will usually cause lisp to exit during the next garbage collection when the illegal pointers are discovered.
  4. fslot-address and fslot-address-typed will always return a normal :c pointer.

 

 

The Programming Interface

(address-to-aligned address)

Convert the integer pointer to an object in memory to an aligned pointer.  See the section on aligned pointers to more information.


(aligned-to-address aligned)

Convert the aligned pointer (which is a fixnum) to the address of the object into memory to which it points.  See the section on aligned pointers for more information.


(allocate-fobject type &optional allocation size)

Allocate an object of the given type in heap described by the allocation argument. If the size argument is given, then it is the minimum size (in bytes) of the data portion of the object that will be allocated. The valid allocation arguments are shown above.


(canonical-ftype type)

If type is or names a foreign type, return the symbol or list that describes that type, otherwise return nil.

If type is a symbol defined using def-foreign-type, then the definition form is returned.  If type is one of the primitive foreign type symbols or is a list in the form valid for def-foreign-type, then type itself is returned.  If type is a symbol that has been given a foreign type definition through def-foreign-type, then the foreign definition is returned.  Using canonical-ftype allows a quick determination of whether a symbol names a simple type or a structured type.


(def-foreign-type  name definition)     [macro]

defines name to be a user-defined foreign type with the given definition. Name must either be a symbol or a list beginning with a symbol and followed by attributes (see below). Definition is not evaluated and must be a foreign type description (see above).

The def-foreign-type macro immediately defines the given type in the current lisp.   It also expands into a form that causes that type definition to be made when the resulting form is evaluated

The attribute that can be specified is:

Attribute Type What
:pack integer worst case alignment needed by data objects.

For example

(def-foreign-type (foostruct (:pack 1)) (:struct (x :char) (y :int)))

Would pack the integer right next to the character, without the normal 3 bytes of padding.


(ensure-foreign-type &key name definition)   [function]

This is the functional equivalent of def-foreign-type.


(free-fobject obj)

Free an object that was allocated by allocate-fobject with the :allocation of :c. An object should only be freed once.


(free-fobject-aligned obj)

Free an object that was allocated by allocate-fobject with the :allocation of :aligned. An object should only be freed once.


(fslot-value-typed type allocation object &rest slot-names)

Access a slot from an object. The type must be

The allocation must be one of :foreign, :foreign-static-gc, :lisp,:c or nil. If the allocation is nil then the allocation type will be computed from the object argument. Note that an allocation type of :foreign or :foreign-static-gc will yield identical results, so you can specify either.

The slot-names are symbols or integers. Symbols name the slots to access. Integers are used to specify array indicies.

The symbol naming a slot can either be the exact symbol used when the type was defined, or it can be a keyword package symbol with the same symbol-name as the one used to define the slot.

The value accessed must be a primitive value. It cannot be a structure or a whole array.

The special slot-name asterisk '*' is used to denote dereferencing a pointer. If a slot has the type (* foo) then you use the * slot name to indicate that you want to follow the pointer to the foo-typed object.   If the object is an array, then the asterisk will access the first element of the array (the element at index zero).

When the allocation argument is given (i.e. not nil), and the slot names are all constants, then this function will be open coded by the compiler if the comp:optimize-fslot-value-switch is true.


(fslot-value object &rest slot-names)

This is like fslot-value-type except it can only be used to access slots from objects with :foreign or :foreign-static-gc allocations, since these are the only objects that are runtime typed.

This function is a lot more convenient to use than fslot-value-type since the type and allocation needn't be specified, however it can't at present be open coded. Thus for speed critical parts of the program, fslot-value-type should be used.


(fslot-address-typed type allocation object &rest slot-names)

This is just like fslot-value-typed except that it returns the address of the object rather than the value. Asking for the address of a :lisp allocated object isn't useful since that object can move during a garbage collection and a program can't predict when a garbage collection can occur.


(fslot-address object &rest slot-names)

This is just like fslot-address-typed except that it works only for :foreign and :foreign-static-gc objects and can't be open coded by the compiler.


(foreign-type-p name)

name is a symbol. If name is the name of a foreign type defined using this package, then t is returned.


(with-stack-fobject (var type) &rest body)

Allocate an object of type type on the stack and bind it to var while evaluating body The object will be of allocation type :foreign for the purposes of accessing it with fslot-value and associated functions. The object will disappear after control leaves body thus the program must not mantain any pointers to the object past this point. Currently, if this form is evaluated by the interpreter, the object will be allocated as a :foreign object, which means it will be in the lisp heap.  This may not be what you want since you may need to allocate an object that doesn't move during garbage collection.  In the future, we will arrange that if stack allocation can't be done, then the object will be allocated as :foreign-static-gc.


Passing Foreign Objects to Foreign Functions

We will take a bottom up approach to describing just what foreign type descriptions mean, and how that relates to what a C program would see receiving a foreign object. We'll use the :int type as an example

  1. Suppose you want to pass an integer to to a foreign function. You do that by just passing the integer value in the foreign function call. You don't need to use the foreign type structures at all.
  2. Suppose you execute this: (allocate-fobject :int) What does this do? Does it return an integer? No, it doesn't. As per case 1, we can use lisp integers to represent integers to be passed to foreign code. In order to understand (allocate-fobject :int) you should remember that allocate-fobject always allocates foreign structures (or arrays of foreign structures) and thus you can rewrite this as (allocate-fobject (:struct (nil :int))) [The nil means that this slot has no name.] Thus you can see that allocate-fobject is going to create a foreign structure that has one field, an :int valued field. If you pass the result of this allocate-fobject to C, what happens is that the lisp foreign-function interface passes a pointer to start of the :int field. Thus the C program should declare a parameter of type 'int *' to receive this value.
  3. Suppose you execute this: (allocate-fobject '(* :int)) From the discussion above we can conclude that this creates a structure with one unnamed field of type (* :int). When this object is passed to C, it should declare the argument as 'int **'
  4. Suppose you execute this: (allocate-fobject '(:array :int 5)) You end up with a structure with 5 :int objects. If passed to C, you can either declare are the arguments to be 'int *' or int[], depending on the syntax you want to use to reference the objects.

Now lets look at what fslot-value operations are possible on each of the kinds of objects mentioned above

  1. a raw integer
    lisp integers are constants. Their values can't be changed.
  2. (setq x (allocate-fobject :int))
    This is a structure with single unnamed slot of type :int. We might refer to this an a :int box. We can get the value with (fslot-value x) and set it with (setf (fslot-value x) 4) or, more verbosely: get the value with (fslot-value-typed :int nil x) set it with (setf (fslot-value-typed :int nil x) 4)
  3. (setq x (allocate-fobject '(* :int)))
    with this we can either get/set the value in the box, or what it points to. To set the value in the box: (setf (fslot-value x) 1321231) or (setf (fslot-value-typed '(* :int) nil x) 1321231).  To set the value at the location pointed to by the value in the box (setf (fslot-value x '*) 1231) or (setf (fslot-value-typed '(* :int) nil x '*) 1231)
  4. (setq x (allocate-fobject '(:array :int 5)))
    we can get/set each individual object, for example: (setf (fslot-value x 3) 4444) or (setf (fslot-value-typed '(:array :int 5) nil x 3) 4444)

Notes:

  1. A foreign type X is equivalent to the foreign type (:struct (nil X))
  2. We never specified an allocation type in the examples above, and this is because what is written above applies to all allocation types (:foreign, :foreign-static-gc,   :lisp, :c)

Quiz:

Now a little quiz to see how well you understand what was done above. A lisp function is passed an integer value str which is the address of a sequence of characters in memory. How do you access the third character in the string?

a. (fslot-value-typed '(* :char) :c str 2)
b. (fslot-value-typed '(array :char 10) :c str 2)
c. all of the above
d. none of the above

The answer to the quiz:
The correct answer is b. Answer a can't be right since the type (* :char) is the same as (:struct (nil (* :char))) and thus says that str points to a structure in memory with one slot, that slot being a pointer to a character string. But we know that str itself points to the string. Answer b is correct. The size of the array we specified isn't important (as long as it is greater than the index we are using the access the array).

Calling C code:

As we've seen above, we take a C type (be it a struct or primitive type) and create a Lisp equivalent type. Let X be the C type and Y the Lisp type. When we pass Y to C, the C code gets not an X object but an X* object since we always pass a pointer to a foreign struture. Likewise when C returns a X structure, it doesn't usually return the X structure, it returns an X* value. Lisp then sees that value as an instance of the Y foreign type. Thus going to C we add a '*' to the type since we pass by reference. Coming back from C we remove a '*' from the type since Lisp always refers to types by pointers, thus using the '*' is superfulous.


This page was last modified on 04/01/98 11:19 AM