3 \setlength{\parindent}{0pt}
4 \setlength{\parskip}{6pt plus 2pt minus 1pt}
7 =========================
8 Protobufs for Common Lisp
9 =========================
12 :Description: Protobufs for Common Lisp
13 :Author: Scott McKay <swm@google.com>
14 :Date: $Date: 2012-04-10 14:18:00 -0500 (Tue, 10 Apr 2012) $
19 1.1 Implementation notes
21 2 Defining a Protobufs schema
22 2.1 .proto file to Lisp conversion
23 2.2 CLOS classes to .proto conversion
24 2.3 Using .proto files directly
25 2.4 Using the Protobufs macros
26 3 Serializing and deserializing
29 4 Python compatibility functions
35 The Protobufs library provides a way for Common Lisp programs to use
36 existing (or define new) Protobufs "schemas", and serialize and
37 deserialize objects to and from the Protobufs wire and text formats.
39 To use it, first load the ASDF declaration file ``protobufs/protobufs.asd``
40 and then use ASDF to load the library.
46 The Protobufs library defines a set of model classes that describes a
47 protobufs "schema". There is a class that describes one .proto file
48 (i.e., one "schema"), options, enums and enum values, messages and
49 fields, and services and methods.
51 The library provides the means to convert several kinds of inputs into
52 the Protobufs models, including:
54 - Parse an existing .proto file into a set of model objects.
55 - Convert a set of related CLOS classes into a set of model objects.
56 - Compile a ``proto:define-proto`` macro call into a set of model objects.
58 It also provides two ways to convert the model objects into outputs:
60 - Print a set of model objects using the standard Protobufs v2 syntax.
61 - Print a set of model objects using the Common Lisp syntax, defined below.
63 There are two formats for serialization and deserialization:
65 - The wire format, which is compact and fast.
66 - The text format, which is human readable.
68 Finally, there is a validator that takes an old version of a Protobufs
69 schema and a new version of the same schema and generates a set of
70 warnings that describes any incompatibilities between the old schema and
79 proto:protobuf [Class]
81 The class the represents a Protobufs schema, i.e., one .proto file.
82 It has slots for the name, options, enums, messages and services.
87 proto:protobuf-option [Class]
89 The class that represents a Protobufs option.
90 It has slots for a key and a value.
95 proto:protobuf-enum [Class]
97 The class that represents a Protobufs enum type.
98 It has slots for the enum name and its values.
103 proto:protobuf-enum-value [Class]
105 The class that represents one value in a Protobufs enum type.
106 It has slots for the value name and the value index.
110 proto:protobuf-message [Class]
112 The class that represents a Protobufs message.
113 It has slots for the name, options, nested enums and messages, and fields.
117 proto:protobuf-field [Class]
119 The class that represents one field in a Protobufs message.
120 It has slots for the name, type, index and options.
124 proto:protobuf-service [Class]
126 The class that represents a Protobufs service.
127 It has slots for the name, options and methods.
131 proto:protobuf-method [Class]
133 The class that represents one method description in a Protobufs service.
134 It has slots for the name, input type, output type and options.
137 Defining a Protobufs schema
138 ===========================
140 There are several ways to define a Protobufs schema: convert a .proto
141 file to a .lisp source file and then using the Lisp file; convert a set
142 of Lisp classes to a Protobufs model, and then use either the .lisp or
143 the .proto representation of the model; use a .proto file directly in as
144 ASDF system; or use the Protobufs macros in a Lisp source file.
147 .proto file to Lisp conversion
148 ------------------------------
150 If you have an existing .proto source file that you would like to
151 convert to Lisp classes (more precisely, to the macros defined by the
152 Protobufs library), you can use ``proto:parse-protobuf-from-file`` to read the
153 .proto file and then use ``proto:write-protobuf`` to write a new .lisp file.
157 proto:parse-protobuf-from-file (filename) [Function]
159 Parses the contents of the file given by *filename*, and returns the
160 Protobufs model (a set object objects rooted at ``proto:protobuf``)
161 corresponding to the parsed file. The name of the Protobufs schema is
162 generated automatically from the file name.
167 proto:parse-protobuf-from-stream (stream &key name class) [Function]
169 Parses the contents of the stream *stream*, and returns the Protobufs
170 schema corresponding to the parsed file. If *name* is supplied, it gives
171 the Protobufs name for the schema. If *class* is supplied, it gives the
177 proto:write-protobuf (protobuf &key stream type) [Function]
179 Pretty-prints the Protobufs schema *protobuf* onto the stream,
180 which defaults to ``*standard-output*``.
182 ``type`` can be either ``:proto`` or ``:lisp``.
185 CLOS classes to .proto conversion
186 ---------------------------------
188 If you have an existing set of CLOS classes that you would like to
189 convert to a Protobufs schema, you can use ``proto:generate-protobuf-schema-from-classes``.
191 Note that the Protobufs schema is an *approximation* of a good schema.
192 You should review it and, if necessary, change it (and probably the Lisp
193 classes as well) until you have a good Protobufs schema definition.
197 proto:generate-protobuf-schema-for-classes (classes [Function]
198 &key name package lisp-package)
200 Given a list of class names *classes*, this generates a Protobufs schema
201 for the classes, generating any necessary enum types that correspond to
202 Lisp ``member`` types.
204 *name* and *package* can be supplied to give the Protobufs name and package.
205 *lisp-package* can be supplied to give the name of the Lisp package, if it
206 is different from *package*.
211 proto:write-protobuf-schema-for-classes (classes [Function]
212 &key stream type name package lisp-package)
214 Given a list of class names *classes*, this generates a Protobufs schema
215 for the classes, generating enum types as necessary, and then
216 pretty-prints the result onto *stream*. *type* can be either ``:proto``
217 (the default) or ``:lisp``.
219 *name* and *package* can be supplied to give the Protobufs name and package.
220 *lisp-package* can be supplied to give the name of the Lisp package, if it
221 is different from *package*.
224 Using .proto files directly
225 ---------------------------
227 In addition to using the tools described above to convert between .proto
228 files and .lisp files, you can also use .proto files directly in ASDF
229 systems. Just use the ASDF module type ``:proto`` in your system, and
230 compile and load the system in the usual way. This will create both the
231 Protobufs model and the Lisp classes that correspond to the Protobufs
235 Using the Protobufs macros
236 --------------------------
238 You can define a Protobufs schema entirely within Lisp by using the
239 following macros. For example::
241 (proto:define-proto color-wheel
242 (:package color-wheel)
243 (proto:define-message color-wheel
244 (:conc-name color-wheel-)
246 (colors :type (proto:list-of color) :default ()))
247 (proto:define-message color
249 (name :type (or string null))
250 (r-value :type integer)
251 (g-value :type integer)
252 (b-value :type integer)
253 (proto:define-extension 1000 max))
254 (proto:define-extend color ()
255 ((opacity 1000) :type (or null integer)))
256 (proto:define-message get-color-request ()
257 (wheel :type color-wheel)
259 (proto:define-message add-color-request ()
260 (wheel :type color-wheel)
262 (proto:define-service color-wheel ()
263 (get-color (get-color-request color)
264 :options ("deadline" "1.0"))
265 (add-color (add-color-request color)
266 :options ("deadline" "1.0"))))
268 This will create the Protobufs model objects, Lisp classes and enum
269 types that correspond to the model. The .proto file of the same schema
276 import "net/proto2/proto/descriptor.proto";
278 extend proto2.MessageOptions {
279 optional string lisp_package = 195801;
280 optional string lisp_name = 195802;
281 optional string lisp_alias = 195803;
285 required string name = 1;
286 repeated Color colors = 2;
290 optional string name = 1;
291 required int64 rValue = 2;
292 required int64 gValue = 3;
293 required int64 bValue = 4;
294 extensions 1000 to max;
298 optional int64 opacity = 1000;
301 message GetColorRequest {
302 required ColorWheel wheel = 1;
303 required string name = 2;
306 message AddColorRequest {
307 required ColorWheel wheel = 1;
308 required Color color = 2;
312 rpc GetColor (GetColorRequest) returns (Color) {
313 option deadline = "1.0";
315 rpc AddColor (AddColorRequest) returns (Color) {
316 option deadline = "1.0";
323 proto:define-proto (type (&key name syntax import [Macro]
325 optimize options documentation)
328 Defines a Protobufs schema whose name is given by the symbol *type*,
329 corresponding to a .proto file of that name. If *name* is not supplied,
330 the Protobufs name of the schema is the camel-cased rendition of *type*
331 (e.g., ``color-wheel`` becomes ``ColorWheel``); otherwise the Protobufs
332 name is the string *name*.
334 *imports* is a list of pathname strings to be imported. This corresponds
335 to ``import`` in a .proto file.
337 *syntax* and *package* are strings that give the Protobufs syntax and
338 package name. *lisp-package* can be supplied to give the name of the
339 Lisp package, if it is different from *package*. *package* corresponds
340 to ``package`` in a .proto file. If you want to specify a Lisp package
341 in a .proto file, you can use ``option lisp_package``.
343 *optimize* can be either ``:space`` (the default) or ``:speed``. When it
344 is ``:space`` the serialization methods generated for each message are
345 compact, but slower; when it is ``:speed``, the serialization methods
346 will be much faster, but will take more space. This corresponds to
347 ``option optimize_for CODE_SIZE|SPEED`` in a .proto file.
349 *options* is a property list whose keys and values are both strings,
350 for example, ``:option ("java_package" "com.yoyodyne.overthruster")``.
351 The are passed along unchanged to a generated .proto file.
353 *documentation* is a documentation string that is preserved as a comment
356 *body* consists of any number of calls to ``proto:define-enum``,
357 ``proto:define-message``, ``proto:define-extend`` or ``proto:define-service``.
362 proto:define-enum (type (&key name conc-name alias-for [Macro]
363 options documentation)
366 Defines a Protobufs enum type and a corresponding Lisp deftype whose name
367 is given by the symbol *type*. If *name* is not supplied, the Protobufs
368 name of the enum is the camel-cased rendition of *type*; otherwise the
369 Protobufs name is the string *name*. If *conc-name* is given, it will
370 be used as the prefix for all of the enum value names. In a .proto file,
371 you can use ``option lisp_name`` to override the default name for the
374 If *alias-for* is given, no Lisp deftype is defined. Instead, the enum
375 will be used as an alias for an enum type that already exists in Lisp.
376 You can use ``option lisp_alias`` in a .proto file to give the Lisp
377 alias for an enum type.
379 *options* is a property list whose keys and values are both strings.
381 *documentation* is a documentation string that is preserved as a comment
384 *body* consists of the enum values, each of which is either a symbol
385 or a list of the form ``(name index)``. By default, the indexes start at
386 0 and are incremented by 1 for each new enum value.
391 proto:define-message (type (&key name conc-name alias-for [Macro]
392 options documentation)
395 Defines a Protobuf message and a corresponding Lisp defclass whose name
396 is given by the symbol *type*. If *name* is not supplied, the Protobufs
397 name of the class is the camel-cased rendition of *type*; otherwise the
398 Protobufs name is the string *name*. If *conc-name* is given, it will
399 be used as the prefix for all of the slot accessor names. In a .proto
400 file, you can use ``option lisp_name`` to override the default name
401 for the class in Lisp.
403 If *alias-for* is given, no Lisp defclass is defined. Instead, the
404 message will be used as an alias for a class that already exists in
405 Lisp. This feature is intended to be used to define messages that will
406 be serialized from existing Lisp classes; unless you get the slot names
407 or readers exactly right for each field, it will be the case that trying
408 to (de)serialize into a Lisp object won't work.
409 You can use ``option lisp_alias`` in a .proto file to give the Lisp
410 alias for the class corresponding to a message.
412 *options* is a property list whose keys and values are both strings.
414 *documentation* is a documentation string that is preserved as a comment
417 The body *fields* consists of fields, ``proto:define-enum``,
418 ``proto:define-message`` or ``proto:define-extension`` forms.
420 Fields take the form ``(slot &key type name default reader writer)``.
421 *slot* can be either a symbol giving the slot name or a list of the
422 form ``(slot index)``. By default, the field indexes start at 1 and
423 are incremented by 1 for each new field value. *type* is the type of
424 the slot. *name* can be used to override the defaultly generated
425 Protobufs field name (for example, ``color-name`` becomes
426 ``colorName``). *default* is the default value for the slot. *reader*
427 is a Lisp slot reader function to use to get the value during
428 serialization, as opposed to using ``slot-value``; this is meant to be
429 used when aliasing an existing class. *writer* can be similarly used
430 to give a Lisp slot writer function.
435 proto:define-extend (type (&key name [Macro]
436 options documentation)
439 Defines a Protobuf "extend", that is, an extension to an existing
440 message and Lisp class that has additional fields that were reserved
441 by ``proto:define-extension``. *type* and *name* are as for
442 ``proto:define-message``. Note that no new Lisp class is defined; the
443 additional slots are implemented as getter and setter methods with a
444 closed-over variable. The other options, such as *conc-name* and
445 *alias-for* are take from the extended message.
447 *options* is a property list whose keys and values are both strings.
449 *documentation* is a documentation string that is preserved as a comment
452 The body *fields* consists only of fields, which take the same form as
453 they do for ``proto:define-message``.
458 proto:define-extension (from to) [Macro]
460 Defines a field extension for the indexes from *from* to *to*.
461 *from* and *to* are positive integers ranging from 1 to 2^29 - 1.
462 *to* can also be the token ``max``, i.e., 2^29 - 1.
467 proto:define-service (type (&key name [Macro]
468 options documentation)
471 Defines a Protobufs service named *type* and corresponding Lisp
472 defgenerics for all its methods. If *name* is not supplied, the Protobufs
473 name of the enum is the camel-cased rendition of *type*; otherwise the
474 Protobufs name is the string *name*.
476 *options* is a property list whose keys and values are both strings.
478 *documentation* is a documentation string that is preserved as a comment
481 The body is a set of method specs of the form
482 ``(name (input-type output-type) &key options documentation)``.
483 *name* is a symbol naming the RPC method. *input-type* and
484 *output-type* may either be symbols or a list of the form ``(type &key name)``.
487 Serializing and deserializing
488 =============================
490 You can serialize from Lisp objects or deserialize into Lisp objects
491 using either the fast and compact Protobufs wire format, or the
492 human-readable text format.
500 proto:serialize-object-to-stream (object type [Function]
503 Serializes the object *object* of type *type* onto the stream *stream*
504 using the wire format. *type* is the Lisp name of a Protobufs message
505 (often the name of a Lisp class) or a ``proto:protobuf-message`` object.
506 *type* defaults to the class of *object*
508 *visited* is a hash table used to cache object sizes. If it is supplied,
509 it will be cleared before it is used; otherwise, a fresh table will be
512 The returned values are a byte vector containing the serialized object
513 and the number of bytes required to serialize the object. If the stream
514 is ``nil``, the buffer is not actually written anywhere.
519 proto:serialize-object (object type buffer [Generic function]
520 &optional start visited)
522 Serializes the object *object* of type *type* into the byte array
523 *buffer* using the wire format. *type* is the Lisp name of a Protobufs
524 message (often the name of a Lisp class) or a ``proto:protobuf-message``
525 object. *type* defaults to the class of *object*. The buffer is assumed
526 to be large enough to hold the serialized object; if it is not, an
527 out-of-bounds condition may be signalled.
529 The object is serialized into the byte array given by *buffer*, starting
530 at the fixnum index *start* using the wire format.
532 *visited* is a hash table used to cache object sizes.
534 The returned values are the modified buffer containing the serialized
535 object and the number of bytes required to serialize the object.
540 proto:deserialize-object-from-stream (type &key stream) [Function]
542 Deserializes an object of the given type *type* as a Protobuf object.
543 *type* is the Lisp name of a Protobufs message (usually the name of a
544 Lisp class) or a ``proto:protobuf-message``.
546 The returned value is the deserialized object.
551 proto:deserialize-object (type buffer &optional start end) [Generic function]
553 Deserializes an object of the given type *type* as a Protobufs object.
554 *type* is the Lisp name of a Protobufs message (usually the name of a
555 Lisp class) or a ``proto:protobuf-message``.
557 The encoded bytes come from the byte array given by *buffer*, starting
558 at the fixnum index *start* up to the end of the buffer, given by *end*.
559 If a zero byte is encountered in in the "tag position" during
560 deserialization, this is interpreted as an "end of object" marker.
562 The returned values are the deserialized object and the index into the
563 buffer at which the deserialization ended.
568 proto:object-size (object type &optional visited) [Generic function]
570 Computes the size in bytes of the object *object* of type *type*.
571 *type* is the Lisp name of a Protobufs message (usually the name of a
572 Lisp class) or a ``proto:protobuf-message``. *type* defaults to the
575 *visited* is a hash table used to cache object sizes.
577 The returned value is the size of the serialized object in bytes.
585 proto:print-text-format (object &optional type [Function]
586 &key stream suppress-line-breaks)
588 Prints the object *object* of type *type* onto the stream *stream* using
589 the textual format. *type* defaults to the class of *object*.
591 If *suppress-line-breaks* is true, all the output is put on a single line.
596 proto:parse-text-format (type &key stream) [Function]
598 Parses the textual format of an object of the given type *type*. *type*
599 is the Lisp name of a Protobufs message (usually the name of a Lisp
600 class) or a ``proto:protobuf-message``. The input is read from the
603 The returned value is the object.
606 Python compatibility functions
607 ==============================
609 By popular demand, the Protobufs library provides an API that very
610 similar to the API of the Python Protobufs library.
614 proto:clear (object) [Generic function]
616 Initializes all of the fields of *object* to their default values.
621 proto:is-initialized (object) [Generic function]
623 Returns true iff all of the fields of *object* are initialized, i.e.,
624 there are no fields whose value is unbound.
629 proto:serialize (object &optional buffer start end) [Generic function]
631 Serializes *object* into *buffer* using the wire format, starting at the
632 index *start* and going no further than *end*. *object* is an object
633 whose Lisp class corresponds to a Protobufs message.
637 proto:merge-from-array (object buffer &optional start end) [Generic function]
639 Deserializes the object encoded in *buffer* into *object*, starting at
640 the index *start* and ending at *end*. *object* is an object whose Lisp
641 class corresponds to a Protobufs message.
646 proto:octet-size (object) [Generic function]
648 Returns the number of bytes required to serialize *object* using the
649 wire format. *object* is an object whose Lisp class corresponds to a