:Description: Protobufs for Common Lisp
-:Author: Scott McKay <swm@google.com>
-:Date: $Date: 2012-05-07 14:58:00 -0500 (Mon, 7 May 2012) $
+:Author: Scott McKay <swmckay@gmail.com>
+:Date: $Date: 2012-08-31 11:13 -0500 (Fri, 31 Aug 2012) $
.. contents::
..
2.1 .proto file to Lisp conversion
2.2 CLOS classes to .proto conversion
2.3 Using .proto files directly
+ 2.3.1 A note on Lisp packages
2.4 Using the Protobufs macros
2.4.1 Protobufs types
2.4.2 Protobufs service stubs
4.1 Extensions functions
4.2 Initialization functions
4.3 Python compatibility functions
+ 5 Lisp-only extensions
+ 5.1 Type aliases
Introduction
for Protobufs.
If *alias-existing-classes* is true (the default), the generated
-code will include ``:alias-for`` so that there will be no clash
+Lisp code will include ``:alias-for`` so that there will be no clash
with the existing Lisp class.
::
system.)
+A note on Lisp packages
+~~~~~~~~~~~~~~~~~~~~~~~
+
+When using an existing .proto file directly, it will likely contain a
+``package`` line, but not a ``lisp_package`` line. CL-Protobufs needs
+to choose some package to use. Here is what it does:
+
+ - The package name from the ``package`` line is converted to a more
+ Lisp-like name, e.g., ``fortune_teller`` becomes ``fortune-teller``.
+ - If the Lisp package exists (i.e., you have previously used
+ ``defpackage`` to define the packaged), then CL-Protobufs just
+ uses it.
+ - If the Lisp package does not exist, CL-Protobufs creates a new
+ package of the given name that uses no other packages, not even
+ the ``common-lisp`` package. In addition, the symbols naming all
+ of the enum types, message types, field name and service method
+ names are exported from the new package.
+
+
Using the Protobufs macros
--------------------------
(color :type color))
(proto:define-service color-wheel ()
(get-color (get-color-request color)
- :options ("deadline" "1.0"))
+ :options (:deadline 1.0))
(add-color (add-color-request color)
- :options ("deadline" "1.0"))))
+ :options (:deadline 1.0))))
This will create the Protobufs model objects, Lisp classes and enum
types that correspond to the model. The .proto file of the same schema
service ColorWheel {
rpc GetColor (GetColorRequest) returns (Color) {
- option deadline = "1.0";
+ option deadline = 1.0;
}
rpc AddColor (AddColorRequest) returns (Color) {
- option deadline = "1.0";
+ option deadline = 1.0;
}
}
turn into repeated fields representing by lists or vectors,
respectively.
+Note also that the macros have assigned indexes to the fields for each
+method; similarly, they will assign values to enumerations as well.
+*This is not stable*, that is, if you add new fields or enum values,
+the indexes could change, which would result in an incompatible
+Protobufs schema.
+
::
proto:define-schema (type (&key name syntax import [Macro]
corresponding to a .proto file of that name. By a "schema", we mean an
object that corresponds to the contents of one .proto file. If *name*
is not supplied, the Protobufs name of the schema is the camel-cased
-rendition of *type* (e.g., ``color-wheel`` becomes ``ColorWheel``);
-otherwise the Protobufs name is the string *name*.
+rendition of *type* (e.g., the schema named ``color-wheel``, by
+default, becomes ``ColorWheel``); otherwise the Protobufs name is the
+string *name*.
*imports* is a list of pathname strings to be imported. This corresponds
to ``import`` in a .proto file. Note that ``proto:define-schema`` can
enum type in Lisp.
If *alias-for* is given, no Lisp deftype is defined. Instead, the enum
-will be used as an alias for an enum type that already exists in Lisp.
+will be used as an alias for a ``member`` type that already exists in Lisp.
You can use ``option (lisp_alias)`` in a .proto file to give the Lisp
alias for an enum type.
in the .proto file.
*body* consists of the enum values, each of which is either a symbol
-or a list of the form ``(name index)``. By default, the indexes start at
-0 and are incremented by 1 for each new enum value.
+or a list either of the form ``(name index)`` or ``(name &key index)``.
+By default, and if you have not explicitly given an index, the indexes
+start at 0 and are incremented by 1 for each new enum value. For
+schema forward and backward compatibility, you should always use the
+explicit form, either ``(name index)`` or ``(name &key index)``.
``proto:define-enum`` can be used only within ``proto:define-schema``
or ``proto:define-message``.
options documentation)
&body fields)
-Defines a Protobuf message and a corresponding Lisp defclass whose name
-is given by the symbol *type*. If *name* is not supplied, the Protobufs
-name of the class is the camel-cased rendition of *type*; otherwise the
-Protobufs name is the string *name*. If *conc-name* is given, it will
-be used as the prefix for all of the slot accessor names. In a .proto
-file, you can use ``option (lisp_name)`` to override the default name
-for the class in Lisp.
+Defines a Protobuf message and a corresponding Lisp defclass whose
+name is given by the symbol *type*. If *name* is not supplied, the
+Protobufs name of the class is the camel-cased rendition of *type*
+(e.g., the class named ``color-wheel``, by default, becomes
+``ColorWheel``); otherwise the Protobufs name is the string *name*. If
+*conc-name* is given, it will be used as the prefix for all of the
+slot accessor names. In a .proto file, you can use ``option (lisp_name)``
+to override the default name for the class in Lisp.
If *alias-for* is given, no Lisp defclass is defined. Instead, the
message will be used as an alias for a class that already exists in
The body *fields* consists of fields, ``proto:define-enum``,
``proto:define-message`` or ``proto:define-extension`` forms.
-Fields take the form ``(slot &key type name default reader writer)``.
+Fields take the form ``(slot &key index type name default reader writer)``.
*slot* can be either a symbol giving the slot name or a list of the
form ``(slot index)``. By default, the field indexes start at 1 and
are incremented by 1 for each new field value. *type* is the type of
-the slot. *name* can be used to override the defaultly generated
-Protobufs field name (for example, ``color-name`` becomes
-``colorName``). *default* is the default value for the slot. *reader*
-is the name of a Lisp slot reader function to use to get the value during
-serialization, as opposed to using ``slot-value``; this is meant to be
-used when aliasing an existing class. *writer* can be similarly used
-to name a Lisp slot writer function.
+the slot. For schema forward and backward compatibility, you should
+always use either the ``(slot index)`` form or supply ``:index``.
+
+*name* can be used to override the defaultly generated Protobufs field
+name (for example, a Lisp field called ``color-name``, by default,
+becomes ``color_name``). *default* is the default value for the
+slot. *reader* is the name of a Lisp slot reader function to use to
+get the value during serialization, as opposed to using
+``slot-value``; this is meant to be used when aliasing an existing
+class. *writer* can be similarly used to name a Lisp slot writer
+function.
Note that the Protobufs does not support full Lisp type expressions in
the types of fields. The following type expressions are supported:
in the .proto file.
The body is a set of method specs of the form
-``(name (input-type output-type) &key options documentation)``.
-*name* is a symbol naming the RPC method. *input-type* and
-*output-type* may either be symbols or a list of the form ``(type &key name)``.
+``(name (input-type [=>] output-type &key streams) &key options documentation)``.
+
+For each method spec, *name* is a symbol naming the RPC method.
+*input-type* and *output-type* give the input and output types of the method;
+they may either be symbols or a list of the form ``(type &key name)``.
+You can optionally include the symbol ``=>`` between the input and
+output types; this seems to improve readability.
+
+*streams* is also the name of a type, and provides a hook to RPC
+implementations that implement "streaming".
``proto:define-service`` can only be used within ``proto:define-schema``.
Protobufs types
----------------
+~~~~~~~~~~~~~~~
The following types are defined in the ``protobufs`` package:
Protobufs service stubs
------------------------
+~~~~~~~~~~~~~~~~~~~~~~~
When you use the ``proto:define-service`` macro to define a service
with some methods, the macro defines "stubs" (CLOS generic functions)
-for each of the methods in the service. Each method gets a client stub
-and a server stub whose signatures are, respectively::
+for each of the methods in the service. Each method named ``foo`` gets
+a client stub and a server stub whose signatures are, respectively::
- (rpc-channel input output &key callback) => output
- (rpc-channel input output) => output
+ call-foo (rpc-channel request &key callback) => response
+ foo-impl (rpc-channel request) => response
+
+These methods are interned in a different lisp package, ``XXX-RPC``,
+where ``XXX`` is the name of the lisp package into which the rest of
+the schema's symbols are interned. This is done so that message field
+accessors methods can't collide with the stubs.
The type of *rpc-channel* is unspecified, but is meant to be a
-"channel" over which some sort of RPC call will be done. The types of
-*input* and *output* are classes that were defined via
+"channel" over which the RPC call will be done. The types of *request*
+and *response* are message classes that were defined via
Protobufs. *callback* is a function of two arguments, the RPC channel
-and the output; it is intended for use by asynchronous RPC calls.
+and the response; it is intended for use by asynchronous RPC calls.
For example, this fragment defines four stubs::
(get-color (get-color-request color))
(add-color (add-color-request color)))
-The client stubs are ``get-color`` and ``add-color``, the server stubs
-are ``do-get-color`` and ``do-add-color``. An RPC library will implement
-a method for the client stub. You must fill in the server stub yourself;
-it will implement the desired functionality.
+The client stubs are ``call-get-color`` and ``call-add-color``, the
+server stubs are ``get-color-impl`` and ``add-color-impl``. An RPC
+library will implement a method for the client stub. You must fill in
+the server stub yourself; it will implement the desired functionality.
+
+The client stub also gets a single method defined for it that looks like
+something like this::
+
+ (defmethod call-foo (rpc-channel (request input-type) &key callback)
+ (let ((call (and *rpc-package* *rpc-call-function*)))
+ (funcall call rpc-channel method request :callback callback)))
+
+where *rpc-channel*, *request* and *callback* are as above.
+The special variables ``*rpc-package*`` and ``*rpc-call-function*``
+are filled in when the RPC package is loaded. *method* is the
+``proto:protobuf-method`` that describes the method; this is
+included so that the RPC implementation can determine what type
+of response object to create, what timeout to use, etc.
It is beyond the scope of this Protobufs library to provide the RPC
service; that is the domain of another library.
Returns the number of bytes required to serialize *object* using the
wire format. *object* is an object whose Lisp class corresponds to a
Protobufs message.
+
+
+Lisp-only extensions
+====================
+
+CL-Protobufs includes some Lisp-only extensions that have no
+counterpart in Protobufs, but which "ground out" to compatible
+Protobufs code.
+
+
+Type aliases
+------------
+
+::
+
+ proto:define-type-alias (type (&key name alias-for [Macro]
+ documentation)
+ &key lisp-type proto-type
+ serializer deserializer)
+
+Defines a Lisp type alias named *type* whose Lisp type is *lisp-type*
+and whose Protobufs type is *proto-type*. *lisp-type* must be a valid
+Lisp type expression; *proto-type* myst be a Protobufs primitive type
+(e.g., ``int32``, ``string``).
+
+*serializer* is a function of one argument that takes an object of
+type *lisp-type* and returns an object having the Protobufs primitive
+type *proto-type*. *deserializer* is a function of one argument that
+takes an object of type *proto-type* and returns an object having the
+type *lisp-type*.
+
+If *name* is not supplied, the Protobufs name of the type alias is the
+camel-cased rendition of *type*; otherwise the Protobufs name is the
+string *name*.
+
+If *alias-for* is given, no Lisp deftype for ``type`` is
+defined. Instead, the type alias is assumed to refer to a
+previously-defined Lisp type.
+
+For example, this Lisp schema::
+
+ (proto:define-schema revision-history
+ (:package revision-history)
+ (proto:define-type-alias date ()
+ :lisp-type integer
+ :proto-type string
+ :serializer integer-to-date
+ :deserializer date-to-integer)
+ (proto:define-message revision ()
+ (proto:define-message metadata ()
+ (author :type (or null string))
+ (revision :type (or null string))
+ (date :type (or null date)))
+ (name :type string)
+ (description :type string)))
+
+will generate this Protobufs schema::
+
+ message Revision {
+ message Metadata {
+ optional string author = 1;
+ optional string revision = 2;
+ // alias maps Lisp integer to Protobufs string
+ optional string date = 3;
+ }
+ required string name = 1;
+ required string description = 2;
+ }