:Description: Protobufs for Common Lisp
-:Author: Scott McKay <swm@google.com>
-:Date: $Date: 2012-04-10 14:18:00 -0500 (Tue, 10 Apr 2012) $
+:Author: Scott McKay <swmckay@gmail.com>
+:Date: $Date: 2012-08-31 11:13 -0500 (Fri, 31 Aug 2012) $
.. contents::
..
2.1 .proto file to Lisp conversion
2.2 CLOS classes to .proto conversion
2.3 Using .proto files directly
+ 2.3.1 A note on Lisp packages
2.4 Using the Protobufs macros
+ 2.4.1 Protobufs types
+ 2.4.2 Protobufs service stubs
3 Serializing and deserializing
3.1 Wire format
- 3.1 Text format
- 4 Python compatibility functions
+ 3.2 Text format
+ 4 Other API functions
+ 4.1 Extensions functions
+ 4.2 Initialization functions
+ 4.3 Python compatibility functions
+ 5 Lisp-only extensions
+ 5.1 Type aliases
Introduction
============
-The Protobufs library provides a way for Common Lisp programs to use
-existing (or define new) Protobufs "schemas", and serialize and
-deserialize objects to and from the Protobufs wire and text formats.
+The Common Lisp Protobufs library provides a fully in-Lisp way for
+Common Lisp programs to use existing, or define new Protobufs
+"schemas", and serialize and deserialize objects to and from the
+Protobufs wire and text formats.
-To use it, first load the ASDF declaration file ``protobufs/protobufs.asd``
-and then use ASDF to load the library.
+To use it, first load the ASDF declaration file ``protobufs/cl-protobufs.asd``
+and then use ASDF to load the library named ``:cl-protobufs``.
Implementation notes
--------------------
The Protobufs library defines a set of model classes that describes a
-protobufs "schema". There is a class that describes one .proto file
-(i.e., one "schema"), options, enums and enum values, messages and
-fields, and services and RPCs.
+protobufs "schema" (i.e., one .proto file). These classes describe a
+schema, its options, enums and enum values, messages and fields, and
+services and methods.
+
+Unlike the 'protobuf' library described at ``http://common-lisp.net/project/protobuf``,
+this implementation is entirely written in Common Lisp. It provides
+some things that the 'protobuf' library does not, in particular,
+the ability to define Protobufs schemas entirely within Lisp and
+the ability to "export" a set of CLOS classes as a Protobufs schema.
The library provides the means to convert several kinds of inputs into
the Protobufs models, including:
- Parse an existing .proto file into a set of model objects.
- Convert a set of related CLOS classes into a set of model objects.
- - Compile a ``proto:define-proto`` macro call into a set of model objects.
+ - Compile a ``proto:define-schema`` macro call into a set of model objects.
It also provides two ways to convert the model objects into outputs:
::
- proto:protobuf [Class]
+ proto:protobuf-schema [Class]
The class the represents a Protobufs schema, i.e., one .proto file.
-It has slots for the name, options, enums, messages and services.
-
+It has slots for the name, options, enums, messages and services. The
+name is equal to the name of the .proto file, without the file type.
::
The class that represents a Protobufs option.
It has slots for a key and a value.
-
::
proto:protobuf-enum [Class]
The class that represents a Protobufs enum type.
It has slots for the enum name and its values.
-
::
proto:protobuf-enum-value [Class]
proto:protobuf-service [Class]
The class that represents a Protobufs service.
-It has slots for the name, options and RPCs.
+It has slots for the name, options and methods.
::
- proto:protobuf-rpc [Class]
+ proto:protobuf-method [Class]
-The class that represents one RPC descriptions in a Protobufs service.
+The class that represents one method description in a Protobufs service.
It has slots for the name, input type, output type and options.
===========================
There are several ways to define a Protobufs schema: convert a .proto
-file to a .lisp source file and then using the Lisp file; convert a set
+file to a .lisp source file and then use the Lisp file; convert a set
of Lisp classes to a Protobufs model, and then use either the .lisp or
-the .proto representation of the model; use a .proto file directly in as
-ASDF system; or use the Protobufs macros in a Lisp source file.
+the .proto representation of the model; use a .proto file directly in
+an ASDF system; or use the Protobufs macros in a Lisp source file.
.proto file to Lisp conversion
If you have an existing .proto source file that you would like to
convert to Lisp classes (more precisely, to the macros defined by the
-Protobufs library), you can use ``proto:parse-protobuf-from-file`` to read the
-.proto file and then use ``proto:write-protobuf`` to write a new .lisp file.
+Protobufs library), you can use ``proto:parse-schema-from-file`` to
+read the .proto file and then use ``proto:write-schema`` to write a
+new .lisp file. (This is what that ASDF module type ``:protobuf-file``
+does.)
::
- proto:parse-protobuf-from-file (filename) [Function]
+ proto:parse-schema-from-file (pathname [Function]
+ &key name class conc-name)
-Parses the contents of the file given by *filename*, and returns the
-Protobufs model (a set object objects rooted at ``proto:protobuf``)
+Parses the contents of the file given by *pathname*, and returns the
+Protobufs model (a set object objects rooted at ``proto:protobuf-schema``)
corresponding to the parsed file. The name of the Protobufs schema is
generated automatically from the file name.
+*name*, *class* and *conc-name* are as for ``proto:parse-schema-from-stream``.
+The defaults for *name* and *class* are produced by taking the name of the
+file and generating a name string and a class name symbol.
::
- proto:parse-protobuf-from-stream (stream &key name class) [Function]
+ proto:parse-schema-from-stream (stream [Function]
+ &key name class conc-name)
Parses the contents of the stream *stream*, and returns the Protobufs
-schema corresponding to the parsed file. If *name* is supplied, it gives
-the Protobufs name for the schema. If *class* is supplied, it gives the
-Lisp name.
+schema corresponding to the parsed file.
+If *name* is supplied, it gives the Protobufs name (a string) for the
+schema. If *class* is supplied, it gives the Lisp name (a symbol). These
+are only used for display purposes.
+
+*conc-name* is the default "conc name" to use for all of the messages
+in the file. The default is "", which has the effect of giving eponymous
+slot accessors to all of the classes generating during the import process.
::
- proto:write-protobuf (protobuf &key stream type) [Function]
+ proto:write-schema (schema &key stream type) [Function]
-Pretty-prints the Protobufs schema *protobuf* onto the stream,
+Pretty-prints the Protobufs schema *schema* onto the stream *stream*,
which defaults to ``*standard-output*``.
-``type`` can be either ``:proto`` or ``:lisp``.
+*type* can be either ``:proto`` or ``:lisp``.
CLOS classes to .proto conversion
---------------------------------
If you have an existing set of CLOS classes that you would like to
-convert to a Protobufs schema, you can use ``proto:generate-protobuf-schema-from-classes``.
+convert to a Protobufs schema, you can use ``proto:generate-schema-from-classes``.
Note that the Protobufs schema is an *approximation* of a good schema.
You should review it and, if necessary, change it (and probably the Lisp
::
- proto:generate-protobuf-schema-for-classes (classes [Function]
- &key name package)
+ proto:generate-schema-for-classes (classes [Function]
+ &key name package lisp-package
+ slot-filter type-filter enum-filter value-filter
+ alias-existing-classes)
Given a list of class names *classes*, this generates a Protobufs schema
for the classes, generating any necessary enum types that correspond to
-Lisp ``member`` types.
-
-*name* and *package* can be supplied to give the Protobufs name and package.
-
+Lisp ``member`` types. The return value is the model, rooted at an instance
+of ``proto:protobuf-schema``.
+
+*name* and *package* can be supplied to give the Protobufs name and
+package. *lisp-package* can be supplied to give the name of the Lisp
+package, if it is different from *package*. (Note that you should
+still use ``in-package`` at the top of .lisp files, and it should
+match the value of *lisp-package*.)
+
+*slot-filter*, *type-filter*, *enum-filter* and *value-filter* are
+filtering functions that can be used to weed out things from the Lisp
+classes that should not be included in the Protobufs schema.
+
+*slot-filter* is a function of two arguments, a list of all the slots
+in the class and the slot currently being processed, and should return
+true if the slot is to be kept or ``nil`` if it to be discarded. For
+example, if there are internal implementation slots in a class that
+need not appear in the Protobufs description, it can be used to filter
+them out.
+
+*type-filter* is a function of one argument, the type (of a slot); it
+should return a "transformed" type if any is required. For example,
+complex ``and`` and ``or`` types can't be directly represented in
+Protobufs; this can be used to substitute something workable.
+
+*enum-filter* is a function of one argument, a list of all the values
+of a ``member`` type; it should return the transformed values. For
+example, there maybe be some enumeration values that don't make sense;
+they can be discarded by the filter.
+
+*value-filter* is a function of one argument, the value of a slot
+initform. It should transform the value into a scalar value suitable
+for Protobufs.
+
+If *alias-existing-classes* is true (the default), the generated
+Lisp code will include ``:alias-for`` so that there will be no clash
+with the existing Lisp class.
::
- proto:write-protobuf-schema-for-classes (classes [Function]
- &key stream type name package)
+ proto:write-schema-for-classes (classes [Function]
+ &key stream type name package lisp-package
+ slot-filter type-filter enum-filter value-filter
+ alias-existing-classes)
Given a list of class names *classes*, this generates a Protobufs schema
for the classes, generating enum types as necessary, and then
pretty-prints the result onto *stream*. *type* can be either ``:proto``
-(the default) or ``:lisp``.
+(the default) or ``:lisp``; it controls which format the generated
+code will be printed in. The return value is the model, rooted at an
+instance of ``proto:protobuf-schema``.
+
+*name* and *package* can be supplied to give the Protobufs name and
+package. *lisp-package* can be supplied to give the name of the Lisp
+package, if it is different from *package*.
-*name* and *package* can be supplied to give the Protobufs name and package.
+*slot-filter*, *type-filter*, *enum-filter* and *value-filter* are
+as for ``proto:generate-schema-for-classes``.
+
+*alias-existing-classes* is as for ``proto:generate-schema-for-classes``.
Using .proto files directly
---------------------------
-In addition to using the tools described above to convert between .proto
-files and .lisp files, you can also use .proto files directly in ASDF
-systems. Just use the ASDF module type ``:proto`` in your system, and
-compile and load the system in the usual way. This will create both the
-Protobufs model and the Lisp classes that correspond to the Protobufs
-messages.
+In addition to using the tools described above to convert between
+.proto files and .lisp files, you can also use .proto files directly
+in ASDF systems. Just use the ASDF module type ``:protobuf-file`` in
+your system, and compile and load the system in the usual way. This
+will create both the Protobufs model and the Lisp classes that
+correspond to the Protobufs messages. (Note that it will also leave a
+.lisp file having the same name as the .proto file in the file
+system.)
+
+
+A note on Lisp packages
+~~~~~~~~~~~~~~~~~~~~~~~
+
+When using an existing .proto file directly, it will likely contain a
+``package`` line, but not a ``lisp_package`` line. CL-Protobufs needs
+to choose some package to use. Here is what it does:
+
+ - The package name from the ``package`` line is converted to a more
+ Lisp-like name, e.g., ``fortune_teller`` becomes ``fortune-teller``.
+ - If the Lisp package exists (i.e., you have previously used
+ ``defpackage`` to define the packaged), then CL-Protobufs just
+ uses it.
+ - If the Lisp package does not exist, CL-Protobufs creates a new
+ package of the given name that uses no other packages, not even
+ the ``common-lisp`` package. In addition, the symbols naming all
+ of the enum types, message types, field name and service method
+ names are exported from the new package.
Using the Protobufs macros
You can define a Protobufs schema entirely within Lisp by using the
following macros. For example::
- (proto:define-proto color-wheel
- (:package color-wheel
- :documentation "Color wheel example")
+ (proto:define-schema color-wheel
+ (:package com.google.colorwheel
+ :lisp-package color-wheel)
(proto:define-message color-wheel
(:conc-name color-wheel-)
(name :type string)
(colors :type (proto:list-of color) :default ()))
(proto:define-message color
- (:conc-name color-
- :documentation "A (named) color")
+ (:conc-name color-)
(name :type (or string null))
(r-value :type integer)
(g-value :type integer)
- (b-value :type integer))
+ (b-value :type integer)
+ (proto:define-extension 1000 max))
+ (proto:define-extend color ()
+ ((opacity 1000) :type (or null integer)))
(proto:define-message get-color-request ()
(wheel :type color-wheel)
(name :type string))
(color :type color))
(proto:define-service color-wheel ()
(get-color (get-color-request color)
- :options ("deadline" "1.0")
- :documentation "Look up a color by name")
+ :options (:deadline 1.0))
(add-color (add-color-request color)
- :options ("deadline" "1.0")
- :documentation "Add a new color to the wheel")))
+ :options (:deadline 1.0))))
This will create the Protobufs model objects, Lisp classes and enum
types that correspond to the model. The .proto file of the same schema
-looks like this::
+looks something like this::
syntax = "proto2";
- package color_wheel;
+ package com.google.colorwheel;
+
+ option (lisp_package) = "color-wheel";
message ColorWheel {
required string name = 1;
required int64 rValue = 2;
required int64 gValue = 3;
required int64 bValue = 4;
+ extensions 1000 to max;
+ }
+
+ extend Color {
+ optional int64 opacity = 1000;
}
message GetColorRequest {
service ColorWheel {
rpc GetColor (GetColorRequest) returns (Color) {
- option deadline = "1.0";
+ option deadline = 1.0;
}
rpc AddColor (AddColorRequest) returns (Color) {
- option deadline = "1.0";
+ option deadline = 1.0;
}
}
+Note that Lisp types ``(or null <T>)`` turn into optional fields,
+and Lisp types ``(proto:list-of <T>)`` and ``(proto:vector-of <T>)``
+turn into repeated fields representing by lists or vectors,
+respectively.
-::
-
- proto:define-proto (type (&key name syntax package import [Macro]
- optimize options documentation)
- &body messages)
+Note also that the macros have assigned indexes to the fields for each
+method; similarly, they will assign values to enumerations as well.
+*This is not stable*, that is, if you add new fields or enum values,
+the indexes could change, which would result in an incompatible
+Protobufs schema.
-Defines a Protobufs schema whose name is given by the symbol *type*,
-corresponding to a .proto file of that name. If *name* is not supplied,
-the Protobufs name of the schema is the camel-cased rendition of *type*
-(e.g., ``color-wheel`` becomes ``ColorWheel``); otherwise the Protobufs
-name is the string *name*.
+::
-*syntax* and package* are strings that give the Protobufs syntax and
-*package name. *imports* is a list of pathname strings to be imported.
+ proto:define-schema (type (&key name syntax import [Macro]
+ package lisp-package
+ optimize options documentation)
+ &body messages)
+
+Defines a Protobufs "schema" whose name is given by the symbol *type*,
+corresponding to a .proto file of that name. By a "schema", we mean an
+object that corresponds to the contents of one .proto file. If *name*
+is not supplied, the Protobufs name of the schema is the camel-cased
+rendition of *type* (e.g., the schema named ``color-wheel``, by
+default, becomes ``ColorWheel``); otherwise the Protobufs name is the
+string *name*.
+
+*imports* is a list of pathname strings to be imported. This corresponds
+to ``import`` in a .proto file. Note that ``proto:define-schema`` can
+import both .proto files and .lisp files containing Protobufs macros,
+but the generated .proto code will convert all of these to imports of
+.proto files.
+
+*syntax* and *package* are strings that give the Protobufs syntax and
+package name. *lisp-package* can be supplied to give the name of the
+Lisp package, if it is different from *package*. *package* corresponds
+to ``package`` in a .proto file. If you want to specify a Lisp package
+in a .proto file, you can use ``option (lisp_package)``.
*optimize* can be either ``:space`` (the default) or ``:speed``. When it
- is ``:space`` the serialization methods generated for each message are
- compact, but slower; when it is ``:speed``, the serialization methods
- will be much faster, but will take more space.
+is ``:space`` the serialization methods generated for each message are
+compact, but slower; when it is ``:speed``, the serialization methods
+will be much faster, but will take more space. This corresponds to
+``option optimize_for = CODE_SIZE|SPEED`` in a .proto file.
*options* is a property list whose keys and values are both strings,
for example, ``:option ("java_package" "com.yoyodyne.overthruster")``.
-The are used unchanged in the .proto file.
+They are passed along unchanged to the generated .proto file.
*documentation* is a documentation string that is preserved as a comment
- in the .proto file.
+in the .proto file.
*body* consists of any number of calls to ``proto:define-enum``,
-``proto:define-message`` or ``proto:define-service``.
-
+``proto:define-message``, ``proto:define-extend`` or ``proto:define-service``.
::
is given by the symbol *type*. If *name* is not supplied, the Protobufs
name of the enum is the camel-cased rendition of *type*; otherwise the
Protobufs name is the string *name*. If *conc-name* is given, it will
-be used as the prefix for all of the enum value names.
+be used as the prefix for all of the enum value names. In a .proto file,
+you can use ``option (lisp_name)`` to override the default name for the
+enum type in Lisp.
If *alias-for* is given, no Lisp deftype is defined. Instead, the enum
-will be used as an alias for an enum type that already exists in Lisp.
+will be used as an alias for a ``member`` type that already exists in Lisp.
+You can use ``option (lisp_alias)`` in a .proto file to give the Lisp
+alias for an enum type.
*options* is a property list whose keys and values are both strings.
*documentation* is a documentation string that is preserved as a comment
- in the .proto file.
+in the .proto file.
-``body`` consists of the enum values, each of which is either a symbol
-or a list of the form ``(name index)``. By default, the indexes start at
-0 and are incremented by 1 for each new enum value.
+*body* consists of the enum values, each of which is either a symbol
+or a list either of the form ``(name index)`` or ``(name &key index)``.
+By default, and if you have not explicitly given an index, the indexes
+start at 0 and are incremented by 1 for each new enum value. For
+schema forward and backward compatibility, you should always use the
+explicit form, either ``(name index)`` or ``(name &key index)``.
+``proto:define-enum`` can be used only within ``proto:define-schema``
+or ``proto:define-message``.
::
options documentation)
&body fields)
-Defines a Protobuf message and a corresponding Lisp defclass whose name
-is given by the symbol *type*. If *name* is not supplied, the Protobufs
-name of the enum is the camel-cased rendition of *type*; otherwise the
-Protobufs name is the string *name*. If *conc-name* is given, it will
-be used as the prefix for all of the slot accessor names.
+Defines a Protobuf message and a corresponding Lisp defclass whose
+name is given by the symbol *type*. If *name* is not supplied, the
+Protobufs name of the class is the camel-cased rendition of *type*
+(e.g., the class named ``color-wheel``, by default, becomes
+``ColorWheel``); otherwise the Protobufs name is the string *name*. If
+*conc-name* is given, it will be used as the prefix for all of the
+slot accessor names. In a .proto file, you can use ``option (lisp_name)``
+to override the default name for the class in Lisp.
If *alias-for* is given, no Lisp defclass is defined. Instead, the
message will be used as an alias for a class that already exists in
Lisp. This feature is intended to be used to define messages that will
-be serialized from existing Lisp classes; unless you get the slot names
-or readers exactly right for each field, it will be the case that trying
-to (de)serialize into a Lisp object won't work.
+be serialized from existing Lisp classes; unless you get the slot names,
+readers and writers exactly right for each field, it will be the case
+that trying to (de)serialize into a(n aliased) Lisp object won't work.
+You can use ``option (lisp_alias)`` in a .proto file to give the Lisp
+alias for the class corresponding to a message.
*options* is a property list whose keys and values are both strings.
*documentation* is a documentation string that is preserved as a comment
- in the .proto file.
+in the .proto file.
-The body consists of fields, or ``proto:define-enum``,
+The body *fields* consists of fields, ``proto:define-enum``,
``proto:define-message`` or ``proto:define-extension`` forms.
-Fields take the form ``(slot &key type name default reader)``. *slot*
-can be either a symbol giving the slot name or a list of the form
-``(slot index)``. By default, the field indexes start at 1 and are
-incremented by 1 for each new field value. *type* is the type of the
-slot. *name* can be used to override the defaultly generated Protobufs
-field name (for example, ``color-name`` becomes ``colorName``).
-*default* is the default value for the slot. *reader* is a Lisp slot
-reader function to use to get the value during serialization, as opposed
-to using ``slot-value``; this is meant to be used when aliasing an
-existing class.
-
+Fields take the form ``(slot &key index type name default reader writer)``.
+*slot* can be either a symbol giving the slot name or a list of the
+form ``(slot index)``. By default, the field indexes start at 1 and
+are incremented by 1 for each new field value. *type* is the type of
+the slot. For schema forward and backward compatibility, you should
+always use either the ``(slot index)`` form or supply ``:index``.
+
+*name* can be used to override the defaultly generated Protobufs field
+name (for example, a Lisp field called ``color-name``, by default,
+becomes ``color_name``). *default* is the default value for the
+slot. *reader* is the name of a Lisp slot reader function to use to
+get the value during serialization, as opposed to using
+``slot-value``; this is meant to be used when aliasing an existing
+class. *writer* can be similarly used to name a Lisp slot writer
+function.
+
+Note that the Protobufs does not support full Lisp type expressions in
+the types of fields. The following type expressions are supported:
+
+ - ``integer``, optionally with upper and lower bounds
+ - ``signed-byte``, which correspond to ``proto:int32`` or ``proto:int64``
+ - ``unsigned-byte``, which correspond to ``proto:uint32`` or ``proto:uint64``
+ - ``float`` and ``double-float``
+ - ``string``and ``character``
+ - ``(array (unsigned-byte 8))``, which corresponds to ``proto:byte-vector``
+ - ``boolean``
+ - ``(member ...)``, where all the members are symbols or keywords or ``nil``
+ - the name of a class that corresponds to another Protobufs message
+ - ``(proto:list-of <T>)``, where ``<T>`` is any of the above types
+ - ``(proto:vector-of <T>)``, where ``<T>`` is any of the above types
+ - ``(or <T> null)``, where ``<T>`` is any of the above types
+
+``member`` corresponds to a Protobufs ``enum``. ``(or <T> null)``
+corresponds to an optional field. ``proto:list-of`` corresponds to a
+repeated field, and the Lisp slot will be typed as a list. ``proto:vector-of``
+corresponds to a repeated field, and the Lisp slot will be typed as an
+adjustable array with a fill pointer. The other types correspond to
+the various Protobufs scalar field types.
+
+``proto:define-message`` can be used only within ``proto:define-schema``
+or ``proto:define-message``.
::
proto:define-extension (from to) [Macro]
Defines a field extension for the indexes from *from* to *to*.
+*from* and *to* are positive integers ranging from 1 to 2^29 - 1.
+*to* can also be the token ``max``, i.e., 2^29 - 1.
+
+Once an extension to a message has been defined, you can use
+``proto:define-extends`` to add new fields.
+
+``proto:define-extension`` can be used only within ``proto:define-message``.
+
+In non-Lisp implementations of Protobufs, you set and get the value
+of an extension using functions like ``SetExtension()`` and
+``GetExtension()``. For example, if you extended a ``Color`` message
+to have an ``opacity`` field, you would set the field using something
+like this::
+
+ Color color;
+ color.SetExtension(opacity, 0.5);
+
+In Common Lisp Protobufs, you can just use an ordinary slot accessor::
+
+ (let ((color (make-instance 'color)))
+ (setf (color-opacity color) 0.5))
+
+::
+ proto:define-extend (type (&key name conc-name [Macro]
+ options documentation)
+ &body fields)
+
+Defines a Protobuf ``extend``, that is, an extension to an existing
+message (and corresponding Lisp class) that has additional fields that
+were reserved by ``proto:define-extension``. *type* and *name* are as
+for ``proto:define-message``. Note that no new Lisp class is defined;
+the additional slots are implemented as getter and setter methods on
+a closed-over variable. The other options, such as *conc-name* and
+*alias-for* are take from the extended message.
+
+*options* is a property list whose keys and values are both strings.
+
+*documentation* is a documentation string that is preserved as a comment
+in the .proto file.
+
+The body *fields* consists only of fields, which take the same form as
+they do for ``proto:define-message``.
+
+``proto:define-extend`` can be used only within ``proto:define-schema``
+or ``proto:define-message``.
::
proto:define-service (type (&key name [Macro]
options documentation)
- &body rpc-specs)
+ &body method-specs)
-Defines a Protobufs service named *type* and corresponding Lisp
-defgenerics for all its RPCs. If *name* is not supplied, the Protobufs
-name of the enum is the camel-cased rendition of *type*; otherwise the
-Protobufs name is the string *name*.
+Defines a Protobufs service named *type* and corresponding Lisp generic
+functions for all its methods. If *name* is not supplied, the Protobufs
+name of the service is the camel-cased rendition of *type*; otherwise
+the Protobufs name is the string *name*.
*options* is a property list whose keys and values are both strings.
*documentation* is a documentation string that is preserved as a comment
- in the .proto file.
+in the .proto file.
+
+The body is a set of method specs of the form
+``(name (input-type [=>] output-type &key streams) &key options documentation)``.
+
+For each method spec, *name* is a symbol naming the RPC method.
+*input-type* and *output-type* give the input and output types of the method;
+they may either be symbols or a list of the form ``(type &key name)``.
+You can optionally include the symbol ``=>`` between the input and
+output types; this seems to improve readability.
+
+*streams* is also the name of a type, and provides a hook to RPC
+implementations that implement "streaming".
+
+``proto:define-service`` can only be used within ``proto:define-schema``.
+
+
+Protobufs types
+~~~~~~~~~~~~~~~
+
+The following types are defined in the ``protobufs`` package:
+
+ - ``proto:int32``, which corresponds to the Protobufs ``int32`` type
+ - ``proto:int64``, which corresponds to the Protobufs ``int64`` type
+ - ``proto:uint32``, which corresponds to the Protobufs ``uint32`` type
+ - ``proto:uint64``, which corresponds to the Protobufs ``uint64`` type
+ - ``proto:sint32``, which corresponds to the Protobufs ``sint32`` type
+ - ``proto:sint64``, which corresponds to the Protobufs ``sint64`` type
+ - ``proto:fixed32``, which corresponds to the Protobufs ``fixed32`` type
+ - ``proto:fixed64``, which corresponds to the Protobufs ``fixed64`` type
+ - ``proto:sfixed32``, which corresponds to the Protobufs ``sfixed32`` type
+ - ``proto:sfixed64``, which corresponds to the Protobufs ``sfixed32`` type
+ - ``proto:byte-vector``, which corresponds to the Protobufs ``bytes`` type
+ - ``proto:list-of``, which corresponds to a repeated field
+ - ``proto:vector-of``, which corresponds to a repeated field
-The body is a set of RPC specs of the form
-``(name (input-type output-type) &key options documentation)``.
-*name* is a symbol naming the RPC function. *input-type* and
-*output-type* may either be symbols or a list of the form ``(type &key name)``.
+The following existing Lisp type correspond to other Protobufs types:
+
+ - ``string`` is the Protobufs UTF-8 encoded ``string`` type
+ - ``boolean`` is the Protobufs ``bool`` type
+ - ``float`` is the Protobufs ``float`` type
+ - ``double-float`` is the Protobufs ``double`` type
+ - ``member`` of a set of keywords generates a Protobufs ``enum`` type
+
+Note that ``(or <T> null)`` corresponds to an optional field.
+
+
+Protobufs service stubs
+~~~~~~~~~~~~~~~~~~~~~~~
+
+When you use the ``proto:define-service`` macro to define a service
+with some methods, the macro defines "stubs" (CLOS generic functions)
+for each of the methods in the service. Each method named ``foo`` gets
+a client stub and a server stub whose signatures are, respectively::
+
+ call-foo (rpc-channel request &key callback) => response
+ foo-impl (rpc-channel request) => response
+
+These methods are interned in a different lisp package, ``XXX-RPC``,
+where ``XXX`` is the name of the lisp package into which the rest of
+the schema's symbols are interned. This is done so that message field
+accessors methods can't collide with the stubs.
+
+The type of *rpc-channel* is unspecified, but is meant to be a
+"channel" over which the RPC call will be done. The types of *request*
+and *response* are message classes that were defined via
+Protobufs. *callback* is a function of two arguments, the RPC channel
+and the response; it is intended for use by asynchronous RPC calls.
+
+For example, this fragment defines four stubs::
+
+ (proto:define-service color-wheel ()
+ (get-color (get-color-request color))
+ (add-color (add-color-request color)))
+
+The client stubs are ``call-get-color`` and ``call-add-color``, the
+server stubs are ``get-color-impl`` and ``add-color-impl``. An RPC
+library will implement a method for the client stub. You must fill in
+the server stub yourself; it will implement the desired functionality.
+
+The client stub also gets a single method defined for it that looks like
+something like this::
+
+ (defmethod call-foo (rpc-channel (request input-type) &key callback)
+ (let ((call (and *rpc-package* *rpc-call-function*)))
+ (funcall call rpc-channel method request :callback callback)))
+
+where *rpc-channel*, *request* and *callback* are as above.
+The special variables ``*rpc-package*`` and ``*rpc-call-function*``
+are filled in when the RPC package is loaded. *method* is the
+``proto:protobuf-method`` that describes the method; this is
+included so that the RPC implementation can determine what type
+of response object to create, what timeout to use, etc.
+
+It is beyond the scope of this Protobufs library to provide the RPC
+service; that is the domain of another library.
Serializing and deserializing
(often the name of a Lisp class) or a ``proto:protobuf-message`` object.
*type* defaults to the class of *object*
-*visited* is a hash table used to cache object sizes. If it is supplied,
-it will be cleared before it is used; otherwise, a fresh table will be
-created.
+The element type of *stream* must be ``(unsigned-byte 8)``.
-The returned values are a byte vector containing the serialized object
-and the number of bytes required to serialize the object. If the stream
-is ``nil``, the buffer is not actually written anywhere.
+*visited* is an ``eql`` hash table used to cache object sizes. If it is
+supplied, it will be cleared before it is used; otherwise, a fresh table
+will be created.
+The returned value is a byte vector containing the serialized object.
+If the stream is ``nil``, the buffer is not actually written anywhere.
::
message (often the name of a Lisp class) or a ``proto:protobuf-message``
object. *type* defaults to the class of *object*. The buffer is assumed
to be large enough to hold the serialized object; if it is not, an
-out-of-bounds condition may be signalled.
+out-of-bounds condition may be signaled.
-The object is serialized into the byte array given by *buffer* starting
-at the fixnum index *start* using the wire format.
+The object is serialized using the wire format into the byte array
+(i.e., a vector whose type is ``(unsigned-byte 8)``) given by *buffer*,
+starting at the fixnum index *start* .
-*visited* is a hash table used to cache object sizes.
+*visited* is an ``eql`` hash table used to cache object sizes.
The returned values are the modified buffer containing the serialized
-object and the number of bytes required to serialize the object.
+object and the index that points one past the last serialized byte in
+the buffer, which will be the number of bytes required to serialize the
+object if *start* was 0.
+Note that ``proto:serialize-object`` will not correctly serialize a
+set of objects that has cycles. You must resolve these yourself.
::
*type* is the Lisp name of a Protobufs message (usually the name of a
Lisp class) or a ``proto:protobuf-message``.
-The returned value is the deserialized object.
+The element type of *stream* must be ``(unsigned-byte 8)``.
+The returned value is the deserialized object.
::
The encoded bytes come from the byte array given by *buffer*, starting
at the fixnum index *start* up to the end of the buffer, given by *end*.
+*start* defaults to 0, *end*' defaults to the length of the buffer.
+
If a zero byte is encountered in in the "tag position" during
-deserialization, this is interpreted as an "end of object" marker.
+deserialization, this is interpreted as an "end of object" marker
+and deserialization stops.
The returned values are the deserialized object and the index into the
buffer at which the deserialization ended.
-
::
proto:object-size (object type &optional visited) [Generic function]
Lisp class) or a ``proto:protobuf-message``. *type* defaults to the
class of *object*
-*visited* is a hash table used to cache object sizes.
+*visited* is an ``eql`` hash table used to cache object sizes.
The returned value is the size of the serialized object in bytes.
If *suppress-line-breaks* is true, all the output is put on a single line.
-
::
proto:parse-text-format (type &key stream) [Function]
The returned value is the object.
-Python compatibility functions
-==============================
+Other API functions
+===================
-By popular demand, the Protobufs library provides an API that very
-similar to the API of the Python Protobufs library.
+Extensions functions
+--------------------
::
- proto:clear (object) [Generic function]
+proto:get-extension (object slot) [Generic function]
-Initializes all of the fields of *object* to their default values.
+Returns the value of the extended slot *slot* in the object *object*.
+
+Since you can just use the ordinary slot reader function, you should
+not need to call ``proto:get-extension``. It is included for compatibility
+with other Protobufs APIs.
+
+::
+
+proto:set-extension (object slot value) [Generic function]
+
+Sets the value of the extended slot *slot* in the object *object*
+to *value*.
+
+Since you can just use the ordinary slot writer function, you should
+not need to call ``proto:set-extension``. It is included for compatibility
+with other Protobufs APIs.
+
+::
+
+proto:has-extension (object slot) [Generic function]
+
+Returns true iff the object *object* has any value for the extended
+slot *slot*.
+
+::
+
+proto:clear-extension (object slot) [Generic function]
+
+Removes the value for the extended slot *slot* in the object *object*.
+
+
+Initialization functions
+------------------------
+
+::
+
+ proto:object-initialized-p (object type) [Generic function]
+
+Returns true iff all of the fields of *object* of type *type* are
+initialized, i.e., there are no fields whose value is unbound.
+
+::
+
+ proto:slot-initialized-p (object type slot) [Generic function]
+Returns true iff the field *slot* of *object* of type *type* is
+initialized, i.e., there are no fields whose value is unbound.
+
+::
+
+ proto:reinitialize-object (object type) [Generic function]
+
+Initializes all of the fields of *object* of type *type* to their
+default values.
+
+
+Python compatibility functions
+------------------------------
+
+By popular demand, the Protobufs library provides an API that is very
+similar to the API of the Python Protobufs library.
::
Returns true iff all of the fields of *object* are initialized, i.e.,
there are no fields whose value is unbound.
+::
+
+ proto:has-field (object slot) [Generic function]
+
+Returns true iff the field *slot* is initialized in *object*.
+
+::
+
+ proto:clear (object) [Generic function]
+
+Initializes all of the fields of *object* to their default values.
::
the index *start* and ending at *end*. *object* is an object whose Lisp
class corresponds to a Protobufs message.
-
::
proto:octet-size (object) [Generic function]
Returns the number of bytes required to serialize *object* using the
wire format. *object* is an object whose Lisp class corresponds to a
Protobufs message.
+
+
+Lisp-only extensions
+====================
+
+CL-Protobufs includes some Lisp-only extensions that have no
+counterpart in Protobufs, but which "ground out" to compatible
+Protobufs code.
+
+
+Type aliases
+------------
+
+::
+
+ proto:define-type-alias (type (&key name alias-for [Macro]
+ documentation)
+ &key lisp-type proto-type
+ serializer deserializer)
+
+Defines a Lisp type alias named *type* whose Lisp type is *lisp-type*
+and whose Protobufs type is *proto-type*. *lisp-type* must be a valid
+Lisp type expression; *proto-type* myst be a Protobufs primitive type
+(e.g., ``int32``, ``string``).
+
+*serializer* is a function of one argument that takes an object of
+type *lisp-type* and returns an object having the Protobufs primitive
+type *proto-type*. *deserializer* is a function of one argument that
+takes an object of type *proto-type* and returns an object having the
+type *lisp-type*.
+
+If *name* is not supplied, the Protobufs name of the type alias is the
+camel-cased rendition of *type*; otherwise the Protobufs name is the
+string *name*.
+
+If *alias-for* is given, no Lisp deftype for ``type`` is
+defined. Instead, the type alias is assumed to refer to a
+previously-defined Lisp type.
+
+For example, this Lisp schema::
+
+ (proto:define-schema revision-history
+ (:package revision-history)
+ (proto:define-type-alias date ()
+ :lisp-type integer
+ :proto-type string
+ :serializer integer-to-date
+ :deserializer date-to-integer)
+ (proto:define-message revision ()
+ (proto:define-message metadata ()
+ (author :type (or null string))
+ (revision :type (or null string))
+ (date :type (or null date)))
+ (name :type string)
+ (description :type string)))
+
+will generate this Protobufs schema::
+
+ message Revision {
+ message Metadata {
+ optional string author = 1;
+ optional string revision = 2;
+ // alias maps Lisp integer to Protobufs string
+ optional string date = 3;
+ }
+ required string name = 1;
+ required string description = 2;
+ }
projects / cl-protobufs.git / blobdiff