This document serves as a specification of the Foreign Functions Interface (FFI) in Allegro CL 5.0.
The description of Foreign Types is no longer in this document and can now be found here.
Implemented in package FOREIGN-FUNCTIONS (nick FF) to avoid proliferation of package names.
All existing 4.3 exported symbols become deprecated, and preserved only for compatibility. Compatibility modules can be required or are autoloaded to provide this compatibility.
The new 5.0 Foreign Function Interface (FFI) defines a new set of symbols for almost all functions and macros. There may be some overlap in usage of keywords symbols.
DEF-FOREIGN-CALL name-and-options arglist &key option ... MACRO DEF-FOREIGN-VARIABLE name-and-options &key type convention MACRO DEF-FOREIGN-TYPE name-and-options def ... MACRO DEFUN-FOREIGN-CALLABLE name-and-opts arglist return &body body MACRO REGISTER-FOREIGN-CALLABLE Function UNREGISTER-FOREIGN-CALLABLE Function REGISTER-LISP-VALUE Function UNREGISTER-LISP-VALUE Function LISP-VALUE Function CONVERT-FOREIGN-NAME Function GET-ENTRY-POINT Function LIST-ALL-FOREIGN-LIBRARIES Function UNLOAD-FOREIGN-LIBRARY Function LOAD extensions
DEF-FOREIGN-CALL name-and-options arglist &key kwopt ... MACRO
This macro is used to describe a function written in a foreign language (usually C). The action of this macro is to define a lisp function which when called will pass control to the foreign function, making appropriate data transformations on arguments to and the return value from the foreign function.
name-and-options -> name-symbol ;; default conversion to external name -> (lisp-name-symbol external-name) external-name -> [convert-function] [external-name-string] arglist -> () ;; Implies default argument processing ;; (like old defforeign :arguments t spec) -> (:void) ;; Explicitly looking for no arguments -> (arg ...) arg -> name -> (name complex-type-spec) ;; If name is nil, a dummy name will be supplied. type-spec -> foreign-type -> (complex-type-spec) complex-type-spec-> foreign-type [lisp-type [translate-type]] translate-type -> ??? ;; Not yet implemented lisp-type -> any Lisp type ;; This constrains the runtime arg to this ;; Lisp type. When trusting declarations we ;; assume the Lisp type and optimize accordingly. ;; If declarations are appropriate the compiler ;; may generate checks to validate the lisp type. kwopt -> :RETURNING type-spec -> :CONVENTION { :C | :STDCALL | :FORTRAN | :FASTCALL} ;; On NT, C (cdecl) and stdcall are now equivalent. -> :ARG-CHECKING { NIL | T } -> :CALL-DIRECT { NIL | T } -> :METHOD-INDEX index ;; Ordinal index of C++ member ;; method; vtbl is first argument. -> :CALLBACK { NIL | T };; Callback currently forced to T. -> :RELEASE-HEAP { :NEVER | :ALWAYS | :WHEN-OK }
DEF-FOREIGN-CALL [MACRO] Arguments: name-and-options arglist &key :returning :convention :arg-checking :call-direct :callback :method-index :release-heap :optimize-for-space
This macro creates the specification which allows lisp to correctly call non-lisp code. Like other defining forms, its macroexpansion clearly shows what will occur and at what eval-when times. The execution of the expanded form always returns the lisp name being defined. The definition that is installed is a lisp function that serves as a wrapper and initiates the foreign call.
name-and-options can either be a symbol or a list of a symbol and an external-name specification. The symbol represents the lisp-name for which the foreign-call definition will be installed. The external name specification can be either a string specifying a literal external name, or it can be a symbol, which represents the name of a conversion function. That conversion function must take one required argument and at least the :language keyword argument, and must be defined at the time the macro expansion is executed. At that time this conversion function will be called and will receive the lisp-name specified, as well as the arguments :language lang where lang is the value of the :convention keyword.
arglist is either the empty list (), which implies much the same as it does in C (arguments are not checked for type or number), or it can be a list of the one entry :void, in which case 0 arguments are explicitly required (also as in C), or it can be a list of argument specifications.
Argument specifications are available with a rich set of syntax and defaults which allow for a C "feel" while still retaining the lisp semantics and power. The basic argument list is simply a list of names, e.g. (a b c d), whose types default to C int types converted in the usual manner from lisp integer types. Any one argument can be specified more fully, as a list of the name followed optionally followed by the foreign type optionally followed by the lisp type (optionally followed by a translation specification, which has not been implemented). The foreign type can either be a built-in foreign type or a type defined by def-foreign-type, or it can be one of the keywords :lisp, :foreign-address, or :single-float-no-proto. The lisp type can be any valid lisp type. If the foreign type is not specified, it defaults to :int (and the lisp type thus defaults to integer). If the foreign type is specified but the lisp type is not, a reasonable lisp type is chosen to correspond to the foreign type. If both foreign and lisp types are chosen, they will be checked for consistency and a warning might be given. A foreign type specification that includes a reference spec such as (& :int) will be interpreted as a pass-by-reference argument.
The special lisp type specification :no-proto is provided for use with a :float foreign type; it is equivalent to the :single-float-no-proto.
The :returning keyword specifies how the value returned from the foreign function will be interpreted. If not specified, the foreign type :int is assumed, with a conversion to lisp type integer (with a possible overflow to a bignum). If the :returning keyword is specified, it can either be a foreign type, or a list of a foreign type and a lisp type (optionally followed by a translation specification, which has not been implemented), or it can be one of the keywords :lisp, :foreign-address, or :single-float-from-double. If the lisp type is not specified, a reasonable lisp type is chosen based on the foreign type. If both foreign and lisp types are chosen, they will be checked for consistency and a warning might be given. A common idiom is to use :returning (:int fixnum) to specify that the foreign value returned is to be simply shifted into a fixnum value, with no consing and simple truncation of the top two bits on overflow.
The :convention keyword allows the specialization of calling conventions due to language or operating-system distinctions. The default convention is :c, and is adequate for most situations. (Note that on NT and Windows95 the c/stdcall convention distinction is required for callbacks via DEFUN-FOREIGN-CALLABLE, but is not required in DEF-FOREIGN-CALL).
Other than :c, the current conventions defined are:
The :arg-checking keyword causes the lisp wrapper function to first check the lisp types against the lisp argument type specifications. When nil, no argument checking is done (although the number of arguments might still be checked). If non-nil, the lisp-types specified or implied in the argument specification is used to check the actual arguments to the wrapper; if a mismatch occurs, error is called. The default value is T, unless the null-list form of argument specification is used.
The :call-direct keyword causes no changes to the lisp wrapper itself, but allows for other lisp functions to call the foreign function directly when compiled after the DEF-FOREIGN-CALL form is in effect. In order for the compilation of a direct-call to be successful, the argument and return types must imply simple type conversions which the compiler can handle. That list of direct-callable conversions is constantly changing, but can be examined by calling the function COMP:LIST-CALL-DIRECT-POSSIBILITIES.
If for any reason a call to the foreign function cannot be compiled into a direct-call, a warning is issued and a call to the wrapper is generated.
The :method-index keyword allows for calling of C++ style member-methods. The value is an index into the virtual table of the C++ class. Symbolic index specifications are not directly supported.
The :callback keyword is non-operative, but is retained in the hopes that its functionality can be revived in future versions. A null value indicates a promise by the programmer that the foreign function will never call-back into lisp. Unfortunately, due to the nature of OS threads implementations, this promise is currently impossible to keep. The value of this keyword is always taken as T, and a warning is issued if specified to nil.
The :release-heap keyword allows the foreign function to operate in native-OS threads, without causing conflicting demands on the lisp heap. The values for this keyword are discussed in the notes section.
The :optimize-for-space keyword provides for minimal space requirement for foreign-call wrappers. This option is best used in conjunction with the :call-direct option. If non-nil, :optimize-for-space will ensure that the wrapper definition takes up very little room, usually as a closure. This usually comes at a cost of speed, and so only makes sense when call-direct is used to compile all actual calls to the foreign function directly, so that the lisp wrapper is not called normally at all.
The native-threads implementation of Allegro CL changes some basic assumptions of the user interface. There is always exactly one native thread per Lisp Process, but there is not necessarily a Lisp process for every thread. Threads are free to run whenever they want; however, only one thread at a time can access the Lisp heap (for read or write); a thread cannot access the Lisp heap unless it has "acquired" the heap, which is only possible after another thread has "released" the heap.
Acquisition and release of the lisp heap are discussed in other documents.. DEF-FOREIGN-CALL allows for the specification of whether to release the heap or not during a call. The possibilities for the :release-heap keyword are:
Here are some examples of def-foreign-call usage (from our foreign test suite):
(def-foreign-call add2 (x y))
Call a function, probably named "add2" in C, whose first arg is named "x" and is an integer in lisp and which is converted to an int for passing to C. If the integer is larger than can be held in a C int, it is truncated. As with the first arg, the second arg named "y" is an integer converted to a C int. The return value is interpreted as a C int type, and is converted to a lisp integer (which may either be a fixnum or consed as a bignum).
(def-foreign-call t_double ((x :double) (y :double single-float) (z :int fixnum)) :returning :double))
Call a function, probably named "t_double" in C, whose first arg is a double-float both in lisp and in C, and whose second arg is a single-float in lisp but is converted to double float for passing into C (this is the the calling convention used by some non ANSI C compilers and by others when the arguments are not prototyped), and the third argument is a fixnum lisp passed as an integer to C. The function returns and boxes a double-float value to lisp.
(def-foreign-call (t-float dash-to-underscore) ((x :double) (y (:float :no-proto)) (z :int fixnum) (w (* :char) string)) :returning #-(or (and sun4 (not svr4)) sun3q) :float #+(or (and sun4 (not svr4)) sun3q) (:double single-float)
Call a function, named "t_float" in C (assuming proper conversion by dash-to-underscore), whose first arg is a double-float both in lisp and in C. Like the previous example, the second arg is a float in lisp, and is converted to double float for passing into C. The third arg named "z" is a fixnum passed as an int, and "w" is a (null-terminated) lisp string, whose first-character-address is passed to C (beware, the string may move if a gc is allowed). Depending on the architecture, the C function will return either a double (from older C compilers) or a float, each interpreted and boxed as a lisp single-float value.
(def-foreign-call c_array ((str (* * :char) (simple-array simple-string (*))) (n :int fixnum)) :returning :char)
Call a function whose C name is probably c_array, whose "str" argument is an array of strings, properly converted (by copying from the lisp parts). The second arg "n" is a lisp fixnum shifted to make a C int. The C function returns a char which is made into a lisp character.
(def-foreign-call (add-font-resource ics-mode-convert "AddFontResource") ((filename (* :char))))
This is an actual Windows NT call. Depending on whether the lisp is an International Character Set (ICS) lisp or not, either AddFontResourceW or AddFontResourceA is chosen and called with a lisp string converted to the string appropriate to the character set. The function returns a properly converted integer.
DEF-FOREIGN-VARIABLE name-and-options &key kwopt ... MACRO
This macro is used to describe a variable exported by a foreign language (usually C). The action of this macro is to define a lisp symbol-macro which when evaluated will obtain the address of the foreign variable and either retrieve or set the value of the variable based on the access-type.
name-and-options -> name-symbol ;; default conversion to external name -> (lisp-name-symbol external-name) external-name -> [convert-function] [external-name-string] kwopt -> :type access-type -> :CONVENTION { :C | :FORTRAN}
DEF-FOREIGN-VARIABLE [MACRO] Arguments: name-and-options &key :type :convention
This macro creates the specification which allows lisp to correctly reference non-lisp variable data. Like other defining forms, its macroexpansion clearly shows what will occur and at what eval-when times. The execution of the expanded form always returns the lisp name being defined. The definition that is installed is a symbol-macro that accesses the non-lisp value correctly.
name-and-options can either be a symbol or a list of a symbol and an external-name specification. The symbol represents the lisp-name for which the foreign-variable definition will be installed. The external name specification can be either a string specifying a literal external name, or it can be a symbol, which represents the name of a conversion function. That conversion function must take one required argument and at least the :language keyword argument, and must be defined at the time the macro expansion is executed. At that time this conversion function will be called and will receive the lisp-name specified, as well as the arguments :language lang where lang is the value of the :convention keyword.
type is the access type of the variable, and can be any of the valid access types for sys:memref and sys:memref-int. The default access type is :unsigned-long.
convention is used by the macro to properly translate the lisp name into a foreign name, and is given to the conversion function as its :language argument.
Here are some examples of def-foreign-variable usage:
user(1): (ff:def-foreign-variable sigblockdebug) sigblockdebug user(2): sigblockdebug 0 user(3): (setq sigblockdebug 1) 1 user(4): (ff:def-foreign-variable (print-lso-relocation "print_lso_relocation")) print-lso-relocation user(5): print-lso-relocation 0 user(6): (setq print-lso-relocation 1) 1 user(7):
In these examples, variables from the Allegro CL internal runtime are set up to allow lisp to access the variables in a lispy way. Since these variables are not documented, it would take a C debugger to examine and verify that the values of the C variables had indeed been changed.
WARNING: Do not attempt to use this macro on the C "errno" variable, nor on any other variable that might be bound on a per-thread basis; doing so might cause (incorrect) information to be returned for the wrong thread.
DEFUN-FOREIGN-CALLABLE [MACRO] Arguments: name arglist
This macro creates a function that can be called from non-lisp code. It is intended that such a function be callable from lisp as well, but such functionality has not yet been provided.
name is a symbol representing the name of the new function to install.
arglist is an argument list specification. each argument is either a symbol representing the name of an argument of type :signed-long, or it is a list of two elements containing the argument name and foreign-type, respectively.
The first form in the body can be a declaration form, whose valid options are :convention and :unwind.
In order to use this a function defined by DEFUN-FOREIGN-CALLABLE, the function must first be registered via REGISTER-FOREIGN-CALLABLE.
The name is coerced to a string and the case mode is massaged, and prefixes/suffixes are added based on the language (which defaults to :c) to create an appropriate name to match the equivalent foreign entry point name. For example, 'sbrk might become "sbrk" on some systems, or "_sbrk" on others. Or a call to a fortran 'subx might be translated to "subx_".
Note: This is the default conversion function for DEF-FOREIGN-CALL, and is applied at foreign call definition time, either at the top level or as a source or fasl file is being loaded. Other functions may be used instead; in particular we supply a function called EXCL::ICS-MODE-CONVERT, which is used in Windows 95 and NT API interface definitions to select between the 8-bit ascii and wide versions of the library function (signified by appending "A" or "W", respectively).
REGISTER-FOREIGN-CALLABLE name &optional table-index convert-to-c Function
Name arg is a symbol defined with DEFUN-FOREIGN-CALLABLE.
Table-index arg can be a fixnum to select a specific table entry to replace, or the symbol :reuse to specify that if the a function with the same name is already registered, the function will be replaced at the same index.
If convert-to-c is specified, then an attempt will be made to convert the return value to a foreign value, otherwise the return value is returned unchanged. It is important to note that the default is nil, and that most often the desired effect is obtained by setting this option to T. The default of nil is retained for compatibility reasons.
Returns 5 values
UNREGISTER-FOREIGN-CALLABLE name &optional table-index Function
Removes the function registered in the index'th entry in the table.
REGISTER-LISP-VALUE Function UNREGISTER-LISP-VALUE Function GET-ENTRY-POINT Function LISP-VALUE Function UNLOAD-FOREIGN-LIBRARY Function
Pretty much the 4.3 definitions, some with new names.
LIST-ALL-FOREIGN-LIBRARIES &key Function
Returns a list of all foreign-libraries that are loaded into the current lisp image. Keywords remain undocumented, but a useful keyword in many architectures is :return-structs, which when true cause a list of structures to be returned with more information than just the library names.
This section contains notes on foreign functions for the UNIX ACL programmer porting to Windows. We recommend using Microsoft Visual C++ (MSVC++) 4.2 or 5.0. We have not tested with any other C compiler. The rest of the notes in this section apply only to MSVC++ 4.2 and 5.0.
The steps to using foreign functions on Windows are:
To access C++ functions from Lisp you must insure ensure C++ name mangling does not occur. The easiest way to do this is to use a file extension of .c instead of .cpp. If you must use the .cpp file extension, then use the extern "C" linkage specification, like this:
extern "C" int foo (int a) { return (a + 1); }
Import the symbols you need from other libraries by specifying the .lib file to the linker. There are two important entry points in acl500.dll which users of the foreign function interface might need: lisp_call_address and lisp_value. To use these entry points from your .dll, you must import the symbols using the linker using acl500.lib provided in the Allegro directory. For example, to compile the example on page 10-42 of the Allegro Common Lisp User Guide, you would need to specify acl500.lib on your cl command line, like this (assuming the function c_calls_lisp is in the file foo.c):
cl -D_MT -MD -LD -Fefoo.dll foo.c
c:\acl432\acl500.lib user32.lib gdi32.lib
kernel32.lib comctl32.lib comdlg32.lib
winmm.lib advapi32.lib msvcrt.lib
After evaluating (load "foo.dll") in Lisp, the rest of the session in the User Guide is the same.
Export the symbols you want to be visible from Lisp by using a linker .def file, or by using the _declspec(dllexport) declaration:
extern "C" _declspec(dllexport) int foo (int a) { return (a + 1); }
Lastly, compile and link your C code into a .dll:
We have tested ACL against Watcom's Fortran 77 Version 11, which will work if proper steps are taken. Watcom allows for several calling sequences, but only one style is compatible with the argument and return value passing that ACL uses to be compatible with Windows C++. To achieve this, the /sc option must be used to specify stack-based argument passing, and the following pragma must be used (starting in column 1):
*$pragma aux
floatfunc value [8087]
for each floatfunc in the source that returns a float value. Our experience is that these pragmas may cause warning messages on other architectures, but will otherwise work without causing errors.
Also, a file with a .lnk extension should be created, which specifies the link options; advanced Fortran users may want to specify link options on the command line, however. For our example, we will describe a file called bar.lnk:
system nt_dll initinstance terminstance import calltolisp 'call_lisp.dll' export floatfunc export funca export funcb file bar
Note that exports are necessary for each function that will be accessed from the lisp side. Also, it is possible to call-back into lisp from Fortran, though it must be done indirectly through a C function in a specified .dll file.
Once the source files are set up, the following commands can be run:
wfc386 /bd /ms /sc /fpi87 bar.f wlink @bar wlib bar +bar.dll
You can use ff:defforeign like you would on UNIX, by requiring the compatibility module :ffcompat, although we recommend that you try out the new ff:def-foreign-call interface as well. The direct-call interface works also, but some of the new options for native-threading are not available yet; instead, direct calls always act as if they are declared with the :release-heap :never option. Use the :call-direct argument to ff:defforeign, as well as other necessary arguments, to allow direct C calling to be compiled inline by the Lisp compiler.
Windows usually expects callbacks to be declared stdcall. You should check your Windows documentation carefully to verify the required calling convention. From the Lisp side, you will need to use a new declaration to ff:defun-c-callable:
(declare (:convention {:c,:stdcall,:method, or :fastcall,}) (:unwind value))
The (:unwind value) declaration says that throwing past this boundary is to be performed by returning value (not evaluated) to the C caller after arranging when control eventually returns to the Lisp-that-called-C-before-this-callback, the throw will be continued from that point. This effect does not require any special action on the part of the Lisp-to-C calling code, except that it had to have been built with a C link (no leaf calls).
Absence of an :unwind declaration is equivalent to having (declare (:unwind 0)).
Either :c or :stdcall can now be used interchangeably on def-foreign-call definitions. The stack will be properly updated, regardless of the style the foreign function expects to be called with. Callbacks however, must be declared with the proper convention.
C:\TMP>type foo.c #ifdef _WIN32 #define DllExport __declspec(dllexport) #else #define DllExport #endif DllExport int test_fun(int foo) { return foo + 101; }
Compile foo.c into foo.dll:
C:\TMP>cl -D_MT -MD -nologo -LD -Zi -W3 -Fefoo.dll foo.c user32.lib gdi32.lib kernel32.lib comctl32.lib comdlg32.lib winmm.lib advapi32.lib msvcrt.lib foo.c Creating library foo.lib and object foo.exp
Then, in Lisp:
user(1): (load "foo.dll") ; Foreign loading foo.dll. t user(2): (ff:def-foreign-call (test "test_fun") (a)) t user(3): (test 10) 111 user(4):
or, using the old defforeign interface:
user(1): (load "foo.dll") ; Foreign loading foo.dll. t user(2): (ff:defforeign 'test :arguments '(integer) :entry-point "test_fun" :return-type :integer) t user(3): (test 10) 111 user(4):
On Solaris, you must produce .so files which are loadable into ACL. The -K pic flag is optional (only on Solaris), and creates slightly different modes of sharing when used. As an example:
% cc -c -K pic foo.c % ld -G -o foo.so foo.o
Fortran is similar:
% f77 -c bar.f % ld -G -o bar.so bar.o
It is often useful to use the -z defs option to find which references are not yet resolved. If an undefined symbol is detected that should be present in the lisp (such as lisp_call_address) then that shared-library must be included in the command line. For example, for a call to lisp_call_address() in libacl50.so:
% ld -G -o foo.so foo.o libacl50.so
The shared objects loadable by ACL are .sl files. You make these by using cc and ld in this way:
% cc +z -Ae +DA1.1 -c foo.c % ld -b -o foo.sl foo.o
and then load the shared library into ACL:
(load "foo.sl")
For Fortran, the commands are:
% f77 +z +DA1.1 -c bar.f % ld -b -o bar.sl bar.o
You must use the -n32 switch to the C compiler and linker to be able to produce .so files which are loadable into ACL. For example:
% cc -n32 -c -K PIC foo.c % ld -n32 -shared -all -o foo.so foo.o or for Fortran: % f77 -n32 -c bar.f % ld -n32 -shared -all -o bar.so bar.o
You must use the proper taso options to the C compiler and you should also use the make_shared script supplied in the bin directory of the distribution, to produce .so files which are loadable into ACL. For example:
% cc -c -G 0 -taso -xtaso -xtaso_short foo.c % bin/make_shared -o foo.so foo.o and for Fortran: % f77 -c -taso bar.f % bin/make_shared -o bar.so bar.o -lm
You must use the make_shared script supplied in the bin directory of the distribution to produce .so files which are loadable into ACL. For example:
% cc -c -D_BSD -D_NO_PROTO -D_NONSTD_TYPES -D_MBI=void foo.c % bin/make_shared -o foo.so foo.o and for Fortran: % xlf -c bar.f % bin/make_shared -o bar.so bar.o -lm
Be sure to also follow the procedures for stubbing files as is described for AIX in the Allegro 4.3 User Guide.