:Description: Protobufs for Common Lisp
-:Author: Scott McKay <swm@google.com>
-:Date: $Date: 2012-05-07 14:58:00 -0500 (Mon, 7 May 2012) $
+:Author: Scott McKay <swmckay@gmail.com>
+:Date: $Date: 2012-08-31 11:13 -0500 (Fri, 31 Aug 2012) $
.. contents::
..
2.1 .proto file to Lisp conversion
2.2 CLOS classes to .proto conversion
2.3 Using .proto files directly
+ 2.3.1 A note on Lisp packages
2.4 Using the Protobufs macros
+ 2.4.1 Protobufs types
+ 2.4.2 Protobufs service stubs
3 Serializing and deserializing
3.1 Wire format
3.2 Text format
4.1 Extensions functions
4.2 Initialization functions
4.3 Python compatibility functions
+ 5 Lisp-only extensions
+ 5.1 Type aliases
Introduction
============
-The Common Lisp Protobufs library provides a way for Common Lisp
-programs to use existing (or define new) Protobufs "schemas", and
-serialize and deserialize objects to and from the Protobufs wire and
-text formats.
+The Common Lisp Protobufs library provides a fully in-Lisp way for
+Common Lisp programs to use existing, or define new Protobufs
+"schemas", and serialize and deserialize objects to and from the
+Protobufs wire and text formats.
-To use it, first load the ASDF declaration file ``protobufs/protobufs.asd``
-and then use ASDF to load the library named ``:protobufs``.
+To use it, first load the ASDF declaration file ``protobufs/cl-protobufs.asd``
+and then use ASDF to load the library named ``:cl-protobufs``.
Implementation notes
--------------------
The Protobufs library defines a set of model classes that describes a
-protobufs "schema" (i.e., one .proto file). There is a class that
-describes a schema, its options, enums and enum values, messages and
-fields, and services and methods.
+protobufs "schema" (i.e., one .proto file). These classes describe a
+schema, its options, enums and enum values, messages and fields, and
+services and methods.
+
+Unlike the 'protobuf' library described at ``http://common-lisp.net/project/protobuf``,
+this implementation is entirely written in Common Lisp. It provides
+some things that the 'protobuf' library does not, in particular,
+the ability to define Protobufs schemas entirely within Lisp and
+the ability to "export" a set of CLOS classes as a Protobufs schema.
The library provides the means to convert several kinds of inputs into
the Protobufs models, including:
convert to Lisp classes (more precisely, to the macros defined by the
Protobufs library), you can use ``proto:parse-schema-from-file`` to
read the .proto file and then use ``proto:write-schema`` to write a
-new .lisp file. (This is what that ASDF module type ``:proto`` does.)
+new .lisp file. (This is what that ASDF module type ``:protobuf-file``
+does.)
::
- proto:parse-schema-from-file (filename) [Function]
+ proto:parse-schema-from-file (pathname [Function]
+ &key name class conc-name)
-Parses the contents of the file given by *filename*, and returns the
+Parses the contents of the file given by *pathname*, and returns the
Protobufs model (a set object objects rooted at ``proto:protobuf-schema``)
corresponding to the parsed file. The name of the Protobufs schema is
generated automatically from the file name.
+*name*, *class* and *conc-name* are as for ``proto:parse-schema-from-stream``.
+The defaults for *name* and *class* are produced by taking the name of the
+file and generating a name string and a class name symbol.
+
::
- proto:parse-schema-from-stream (stream &key name class) [Function]
+ proto:parse-schema-from-stream (stream [Function]
+ &key name class conc-name)
Parses the contents of the stream *stream*, and returns the Protobufs
-schema corresponding to the parsed file. If *name* is supplied, it gives
-the Protobufs name for the schema. If *class* is supplied, it gives the
-Lisp name.
+schema corresponding to the parsed file.
+
+If *name* is supplied, it gives the Protobufs name (a string) for the
+schema. If *class* is supplied, it gives the Lisp name (a symbol). These
+are only used for display purposes.
+
+*conc-name* is the default "conc name" to use for all of the messages
+in the file. The default is "", which has the effect of giving eponymous
+slot accessors to all of the classes generating during the import process.
::
for Protobufs.
If *alias-existing-classes* is true (the default), the generated
-code will include ``:alias-for`` so that there will be no clash
+Lisp code will include ``:alias-for`` so that there will be no clash
with the existing Lisp class.
::
Using .proto files directly
---------------------------
-In addition to using the tools described above to convert between .proto
-files and .lisp files, you can also use .proto files directly in ASDF
-systems. Just use the ASDF module type ``:proto`` in your system, and
-compile and load the system in the usual way. This will create both the
-Protobufs model and the Lisp classes that correspond to the Protobufs
-messages. (Note that it will also leave a .lisp file having the same
-name as the .proto file in the file system.)
+In addition to using the tools described above to convert between
+.proto files and .lisp files, you can also use .proto files directly
+in ASDF systems. Just use the ASDF module type ``:protobuf-file`` in
+your system, and compile and load the system in the usual way. This
+will create both the Protobufs model and the Lisp classes that
+correspond to the Protobufs messages. (Note that it will also leave a
+.lisp file having the same name as the .proto file in the file
+system.)
+
+
+A note on Lisp packages
+~~~~~~~~~~~~~~~~~~~~~~~
+
+When using an existing .proto file directly, it will likely contain a
+``package`` line, but not a ``lisp_package`` line. CL-Protobufs needs
+to choose some package to use. Here is what it does:
+
+ - The package name from the ``package`` line is converted to a more
+ Lisp-like name, e.g., ``fortune_teller`` becomes ``fortune-teller``.
+ - If the Lisp package exists (i.e., you have previously used
+ ``defpackage`` to define the packaged), then CL-Protobufs just
+ uses it.
+ - If the Lisp package does not exist, CL-Protobufs creates a new
+ package of the given name that uses no other packages, not even
+ the ``common-lisp`` package. In addition, the symbols naming all
+ of the enum types, message types, field name and service method
+ names are exported from the new package.
Using the Protobufs macros
(color :type color))
(proto:define-service color-wheel ()
(get-color (get-color-request color)
- :options ("deadline" "1.0"))
+ :options (:deadline 1.0))
(add-color (add-color-request color)
- :options ("deadline" "1.0"))))
+ :options (:deadline 1.0))))
This will create the Protobufs model objects, Lisp classes and enum
types that correspond to the model. The .proto file of the same schema
-looks like this::
+looks something like this::
syntax = "proto2";
package com.google.colorwheel;
- import "net/proto2/proto/descriptor.proto";
-
- extend proto2.MessageOptions {
- optional string lisp_package = 195801;
- optional string lisp_name = 195802;
- optional string lisp_alias = 195803;
- }
-
option (lisp_package) = "color-wheel";
message ColorWheel {
service ColorWheel {
rpc GetColor (GetColorRequest) returns (Color) {
- option deadline = "1.0";
+ option deadline = 1.0;
}
rpc AddColor (AddColorRequest) returns (Color) {
- option deadline = "1.0";
+ option deadline = 1.0;
}
}
Note that Lisp types ``(or null <T>)`` turn into optional fields,
-and Lisp types ``(proto:list-of <T>)`` turn into repeated fields.
+and Lisp types ``(proto:list-of <T>)`` and ``(proto:vector-of <T>)``
+turn into repeated fields representing by lists or vectors,
+respectively.
+
+Note also that the macros have assigned indexes to the fields for each
+method; similarly, they will assign values to enumerations as well.
+*This is not stable*, that is, if you add new fields or enum values,
+the indexes could change, which would result in an incompatible
+Protobufs schema.
::
corresponding to a .proto file of that name. By a "schema", we mean an
object that corresponds to the contents of one .proto file. If *name*
is not supplied, the Protobufs name of the schema is the camel-cased
-rendition of *type* (e.g., ``color-wheel`` becomes ``ColorWheel``);
-otherwise the Protobufs name is the string *name*.
+rendition of *type* (e.g., the schema named ``color-wheel``, by
+default, becomes ``ColorWheel``); otherwise the Protobufs name is the
+string *name*.
*imports* is a list of pathname strings to be imported. This corresponds
to ``import`` in a .proto file. Note that ``proto:define-schema`` can
enum type in Lisp.
If *alias-for* is given, no Lisp deftype is defined. Instead, the enum
-will be used as an alias for an enum type that already exists in Lisp.
+will be used as an alias for a ``member`` type that already exists in Lisp.
You can use ``option (lisp_alias)`` in a .proto file to give the Lisp
alias for an enum type.
in the .proto file.
*body* consists of the enum values, each of which is either a symbol
-or a list of the form ``(name index)``. By default, the indexes start at
-0 and are incremented by 1 for each new enum value.
+or a list either of the form ``(name index)`` or ``(name &key index)``.
+By default, and if you have not explicitly given an index, the indexes
+start at 0 and are incremented by 1 for each new enum value. For
+schema forward and backward compatibility, you should always use the
+explicit form, either ``(name index)`` or ``(name &key index)``.
``proto:define-enum`` can be used only within ``proto:define-schema``
or ``proto:define-message``.
options documentation)
&body fields)
-Defines a Protobuf message and a corresponding Lisp defclass whose name
-is given by the symbol *type*. If *name* is not supplied, the Protobufs
-name of the class is the camel-cased rendition of *type*; otherwise the
-Protobufs name is the string *name*. If *conc-name* is given, it will
-be used as the prefix for all of the slot accessor names. In a .proto
-file, you can use ``option (lisp_name)`` to override the default name
-for the class in Lisp.
+Defines a Protobuf message and a corresponding Lisp defclass whose
+name is given by the symbol *type*. If *name* is not supplied, the
+Protobufs name of the class is the camel-cased rendition of *type*
+(e.g., the class named ``color-wheel``, by default, becomes
+``ColorWheel``); otherwise the Protobufs name is the string *name*. If
+*conc-name* is given, it will be used as the prefix for all of the
+slot accessor names. In a .proto file, you can use ``option (lisp_name)``
+to override the default name for the class in Lisp.
If *alias-for* is given, no Lisp defclass is defined. Instead, the
message will be used as an alias for a class that already exists in
The body *fields* consists of fields, ``proto:define-enum``,
``proto:define-message`` or ``proto:define-extension`` forms.
-Fields take the form ``(slot &key type name default reader writer)``.
+Fields take the form ``(slot &key index type name default reader writer)``.
*slot* can be either a symbol giving the slot name or a list of the
form ``(slot index)``. By default, the field indexes start at 1 and
are incremented by 1 for each new field value. *type* is the type of
-the slot. *name* can be used to override the defaultly generated
-Protobufs field name (for example, ``color-name`` becomes
-``colorName``). *default* is the default value for the slot. *reader*
-is a Lisp slot reader function to use to get the value during
-serialization, as opposed to using ``slot-value``; this is meant to be
-used when aliasing an existing class. *writer* can be similarly used
-to give a Lisp slot writer function.
+the slot. For schema forward and backward compatibility, you should
+always use either the ``(slot index)`` form or supply ``:index``.
+
+*name* can be used to override the defaultly generated Protobufs field
+name (for example, a Lisp field called ``color-name``, by default,
+becomes ``color_name``). *default* is the default value for the
+slot. *reader* is the name of a Lisp slot reader function to use to
+get the value during serialization, as opposed to using
+``slot-value``; this is meant to be used when aliasing an existing
+class. *writer* can be similarly used to name a Lisp slot writer
+function.
Note that the Protobufs does not support full Lisp type expressions in
the types of fields. The following type expressions are supported:
- ``(member ...)``, where all the members are symbols or keywords or ``nil``
- the name of a class that corresponds to another Protobufs message
- ``(proto:list-of <T>)``, where ``<T>`` is any of the above types
+ - ``(proto:vector-of <T>)``, where ``<T>`` is any of the above types
- ``(or <T> null)``, where ``<T>`` is any of the above types
-``member`` corresponds to a Protobufs ``enum``, ``proto:list-of`` to
-a repeated field, and ``(or <T> null)`` to an optional field. The other
-types correspond to the various Protobufs scalar field types.
+``member`` corresponds to a Protobufs ``enum``. ``(or <T> null)``
+corresponds to an optional field. ``proto:list-of`` corresponds to a
+repeated field, and the Lisp slot will be typed as a list. ``proto:vector-of``
+corresponds to a repeated field, and the Lisp slot will be typed as an
+adjustable array with a fill pointer. The other types correspond to
+the various Protobufs scalar field types.
``proto:define-message`` can be used only within ``proto:define-schema``
or ``proto:define-message``.
::
- proto:define-extend (type (&key name [Macro]
+ proto:define-extend (type (&key name conc-name [Macro]
options documentation)
&body fields)
in the .proto file.
The body is a set of method specs of the form
-``(name (input-type output-type) &key options documentation)``.
-*name* is a symbol naming the RPC method. *input-type* and
-*output-type* may either be symbols or a list of the form ``(type &key name)``.
+``(name (input-type [=>] output-type &key streams) &key options documentation)``.
+
+For each method spec, *name* is a symbol naming the RPC method.
+*input-type* and *output-type* give the input and output types of the method;
+they may either be symbols or a list of the form ``(type &key name)``.
+You can optionally include the symbol ``=>`` between the input and
+output types; this seems to improve readability.
+
+*streams* is also the name of a type, and provides a hook to RPC
+implementations that implement "streaming".
``proto:define-service`` can only be used within ``proto:define-schema``.
+Protobufs types
+~~~~~~~~~~~~~~~
+
+The following types are defined in the ``protobufs`` package:
+
+ - ``proto:int32``, which corresponds to the Protobufs ``int32`` type
+ - ``proto:int64``, which corresponds to the Protobufs ``int64`` type
+ - ``proto:uint32``, which corresponds to the Protobufs ``uint32`` type
+ - ``proto:uint64``, which corresponds to the Protobufs ``uint64`` type
+ - ``proto:sint32``, which corresponds to the Protobufs ``sint32`` type
+ - ``proto:sint64``, which corresponds to the Protobufs ``sint64`` type
+ - ``proto:fixed32``, which corresponds to the Protobufs ``fixed32`` type
+ - ``proto:fixed64``, which corresponds to the Protobufs ``fixed64`` type
+ - ``proto:sfixed32``, which corresponds to the Protobufs ``sfixed32`` type
+ - ``proto:sfixed64``, which corresponds to the Protobufs ``sfixed32`` type
+ - ``proto:byte-vector``, which corresponds to the Protobufs ``bytes`` type
+ - ``proto:list-of``, which corresponds to a repeated field
+ - ``proto:vector-of``, which corresponds to a repeated field
+
+The following existing Lisp type correspond to other Protobufs types:
+
+ - ``string`` is the Protobufs UTF-8 encoded ``string`` type
+ - ``boolean`` is the Protobufs ``bool`` type
+ - ``float`` is the Protobufs ``float`` type
+ - ``double-float`` is the Protobufs ``double`` type
+ - ``member`` of a set of keywords generates a Protobufs ``enum`` type
+
+Note that ``(or <T> null)`` corresponds to an optional field.
+
+
+Protobufs service stubs
+~~~~~~~~~~~~~~~~~~~~~~~
+
+When you use the ``proto:define-service`` macro to define a service
+with some methods, the macro defines "stubs" (CLOS generic functions)
+for each of the methods in the service. Each method named ``foo`` gets
+a client stub and a server stub whose signatures are, respectively::
+
+ foo (rpc-channel request &key callback) => response
+ do-foo (rpc-channel request) => response
+
+The type of *rpc-channel* is unspecified, but is meant to be a
+"channel" over which the RPC call will be done. The types of *request*
+and *response* are message classes that were defined via
+Protobufs. *callback* is a function of two arguments, the RPC channel
+and the response; it is intended for use by asynchronous RPC calls.
+
+For example, this fragment defines four stubs::
+
+ (proto:define-service color-wheel ()
+ (get-color (get-color-request color))
+ (add-color (add-color-request color)))
+
+The client stubs are ``get-color`` and ``add-color``, the server stubs
+are ``do-get-color`` and ``do-add-color``. An RPC library will implement
+a method for the client stub. You must fill in the server stub yourself;
+it will implement the desired functionality.
+
+The client stub also gets a single method defined for it that looks like
+something like this::
+
+ (defmethod foo (rpc-channel (request input-type) &key callback)
+ (let ((call (and *rpc-package* *rpc-call-function*)))
+ (funcall call rpc-channel method request :callback callback)))
+
+where *rpc-channel*, *request* and *callback* are as above.
+The special variables ``*rpc-package*`` and ``*rpc-call-function*``
+are filled in when the RPC package is loaded. *method* is the
+``proto:protobuf-method`` that describes the method; this is
+included so that the RPC implementation can determine what type
+of response object to create, what timeout to use, etc.
+
+It is beyond the scope of this Protobufs library to provide the RPC
+service; that is the domain of another library.
+
+
Serializing and deserializing
=============================
Returns the number of bytes required to serialize *object* using the
wire format. *object* is an object whose Lisp class corresponds to a
Protobufs message.
+
+
+Lisp-only extensions
+====================
+
+CL-Protobufs includes some Lisp-only extensions that have no
+counterpart in Protobufs, but which "ground out" to compatible
+Protobufs code.
+
+
+Type aliases
+------------
+
+::
+
+ proto:define-type-alias (type (&key name alias-for [Macro]
+ documentation)
+ &key lisp-type proto-type
+ serializer deserializer)
+
+Defines a Lisp type alias named *type* whose Lisp type is *lisp-type*
+and whose Protobufs type is *proto-type*. *lisp-type* must be a valid
+Lisp type expression; *proto-type* myst be a Protobufs primitive type
+(e.g., ``int32``, ``string``).
+
+*serializer* is a function of one argument that takes an object of
+type *lisp-type* and returns an object having the Protobufs primitive
+type *proto-type*. *deserializer* is a function of one argument that
+takes an object of type *proto-type* and returns an object having the
+type *lisp-type*.
+
+If *name* is not supplied, the Protobufs name of the type alias is the
+camel-cased rendition of *type*; otherwise the Protobufs name is the
+string *name*.
+
+If *alias-for* is given, no Lisp deftype for ``type`` is
+defined. Instead, the type alias is assumed to refer to a
+previously-defined Lisp type.
+
+For example, this Lisp schema::
+
+ (proto:define-schema revision-history
+ (:package revision-history)
+ (proto:define-type-alias date ()
+ :lisp-type integer
+ :proto-type string
+ :serializer integer-to-date
+ :deserializer date-to-integer)
+ (proto:define-message revision ()
+ (proto:define-message metadata ()
+ (author :type (or null string))
+ (revision :type (or null string))
+ (date :type (or null date)))
+ (name :type string)
+ (description :type string)))
+
+will generate this Protobufs schema::
+
+ message Revision {
+ message Metadata {
+ optional string author = 1;
+ optional string revision = 2;
+ // alias maps Lisp integer to Protobufs string
+ optional string date = 3;
+ }
+ required string name = 1;
+ required string description = 2;
+ }