3 \setlength{\parindent}{0pt}
4 \setlength{\parskip}{6pt plus 2pt minus 1pt}
7 =========================
8 Protobufs for Common Lisp
9 =========================
12 :Description: Protobufs for Common Lisp
13 :Author: Scott McKay <swm@google.com>
14 :Date: $Date: 2012-04-10 14:18:00 -0500 (Tue, 10 Apr 2012) $
19 1.1 Implementation notes
21 2 Defining a Protobufs schema
22 2.1 .proto file to Lisp conversion
23 2.2 CLOS classes to .proto conversion
24 2.3 Using .proto files directly
25 2.4 Using the Protobufs macros
26 3 Serializing and deserializing
29 4 Python compatibility functions
35 The Common Lisp Protobufs library provides a way for Common Lisp
36 programs to use existing (or define new) Protobufs "schemas", and
37 serialize and deserialize objects to and from the Protobufs wire and
40 To use it, first load the ASDF declaration file ``protobufs/protobufs.asd``
41 and then use ASDF to load the library named ``:protobufs``.
47 The Protobufs library defines a set of model classes that describes a
48 protobufs "schema" (i.e., one .proto file). There is a class that
49 describes each schema, options, enums and enum values, messages and
50 fields, and services and methods.
52 The library provides the means to convert several kinds of inputs into
53 the Protobufs models, including:
55 - Parse an existing .proto file into a set of model objects.
56 - Convert a set of related CLOS classes into a set of model objects.
57 - Compile a ``proto:define-proto`` macro call into a set of model objects.
59 It also provides two ways to convert the model objects into outputs:
61 - Print a set of model objects using the standard Protobufs v2 syntax.
62 - Print a set of model objects using the Common Lisp syntax, defined below.
64 There are two formats for serialization and deserialization:
66 - The wire format, which is compact and fast.
67 - The text format, which is human readable.
69 Finally, there is a validator that takes an old version of a Protobufs
70 schema and a new version of the same schema and generates a set of
71 warnings that describes any incompatibilities between the old schema and
80 proto:protobuf [Class]
82 The class the represents a Protobufs schema, i.e., one .proto file.
83 It has slots for the name, options, enums, messages and services. The
84 name is equal to the name of the .proto file, without the file type.
89 proto:protobuf-option [Class]
91 The class that represents a Protobufs option.
92 It has slots for a key and a value.
97 proto:protobuf-enum [Class]
99 The class that represents a Protobufs enum type.
100 It has slots for the enum name and its values.
105 proto:protobuf-enum-value [Class]
107 The class that represents one value in a Protobufs enum type.
108 It has slots for the value name and the value index.
112 proto:protobuf-message [Class]
114 The class that represents a Protobufs message.
115 It has slots for the name, options, nested enums and messages, and fields.
119 proto:protobuf-field [Class]
121 The class that represents one field in a Protobufs message.
122 It has slots for the name, type, index and options.
126 proto:protobuf-service [Class]
128 The class that represents a Protobufs service.
129 It has slots for the name, options and methods.
133 proto:protobuf-method [Class]
135 The class that represents one method description in a Protobufs service.
136 It has slots for the name, input type, output type and options.
139 Defining a Protobufs schema
140 ===========================
142 There are several ways to define a Protobufs schema: convert a .proto
143 file to a .lisp source file and then use the Lisp file; convert a set
144 of Lisp classes to a Protobufs model, and then use either the .lisp or
145 the .proto representation of the model; use a .proto file directly in
146 an ASDF system; or use the Protobufs macros in a Lisp source file.
149 .proto file to Lisp conversion
150 ------------------------------
152 If you have an existing .proto source file that you would like to
153 convert to Lisp classes (more precisely, to the macros defined by the
154 Protobufs library), you can use ``proto:parse-protobuf-from-file`` to
155 read the .proto file and then use ``proto:write-protobuf`` to write a
156 new .lisp file. (This is what that ASDF module type ``:proto`` does.)
160 proto:parse-protobuf-from-file (filename) [Function]
162 Parses the contents of the file given by *filename*, and returns the
163 Protobufs model (a set object objects rooted at ``proto:protobuf``)
164 corresponding to the parsed file. The name of the Protobufs schema is
165 generated automatically from the file name.
170 proto:parse-protobuf-from-stream (stream &key name class) [Function]
172 Parses the contents of the stream *stream*, and returns the Protobufs
173 schema corresponding to the parsed file. If *name* is supplied, it gives
174 the Protobufs name for the schema. If *class* is supplied, it gives the
180 proto:write-protobuf (protobuf &key stream type) [Function]
182 Pretty-prints the Protobufs schema *protobuf* onto the stream *stream*,
183 which defaults to ``*standard-output*``.
185 ``type`` can be either ``:proto`` or ``:lisp``.
188 CLOS classes to .proto conversion
189 ---------------------------------
191 If you have an existing set of CLOS classes that you would like to
192 convert to a Protobufs schema, you can use ``proto:generate-protobuf-schema-from-classes``.
194 Note that the Protobufs schema is an *approximation* of a good schema.
195 You should review it and, if necessary, change it (and probably the Lisp
196 classes as well) until you have a good Protobufs schema definition.
200 proto:generate-protobuf-schema-for-classes (classes [Function]
201 &key name package lisp-package
202 slot-filter type-filter enum-filter value-filter)
204 Given a list of class names *classes*, this generates a Protobufs schema
205 for the classes, generating any necessary enum types that correspond to
206 Lisp ``member`` types. The return value is the model rooted at ``proto:protobuf``.
208 *name* and *package* can be supplied to give the Protobufs name and
209 *package. lisp-package* can be supplied to give the name of the Lisp
210 *package, if it is different from *package*.
212 *slot-filter*, *type-filter*, *enum-filter* and *value-filter* are
213 filtering functions that can be used to weed out things from the Lisp
214 classes that should not be included in the Protobufs schema.
216 *slot-filter* is a function of two arguments, a list of all the slots
217 in the class and the slot currently being processed, and should return
218 ``t`` if the slot is to be kept or ``nil`` if it to be discarded. For
219 example, if there are internal implementation slots in a class that
220 need not appear in the Protobufs description, it can be used to filter
223 *type-filter* is a function of one argument, the type (of a slot); it
224 should return a "transformed" type if any is required. For example,
225 complex ``and`` and ``or`` types can't be directly represented in
226 Protobufs; this can be used to substitute something workable.
228 *enum-filter* is a function of one argument, a list of all the values
229 of a ``member`` type; it should return the transformed values. For
230 example, there maybe be some enumeration values that don't make sense;
231 they can be discarded by the filter.
233 *value-filter* is a function of one argument, the value of a slot
234 initform. It should transform the value into a scalar value suitable
240 proto:write-protobuf-schema-for-classes (classes [Function]
241 &key stream type name package lisp-package)
243 Given a list of class names *classes*, this generates a Protobufs schema
244 for the classes, generating enum types as necessary, and then
245 pretty-prints the result onto *stream*. *type* can be either ``:proto``
246 (the default) or ``:lisp``. The return value is the model rooted at ``proto:protobuf``.
248 *name* and *package* can be supplied to give the Protobufs name and
249 *package. lisp-package* can be supplied to give the name of the Lisp
250 *package, if it is different from *package*.
252 *slot-filter*, *type-filter*, *enum-filter* and *value-filter* are
253 as for ``proto:generate-protobuf-schema-for-classes``.
256 Using .proto files directly
257 ---------------------------
259 In addition to using the tools described above to convert between .proto
260 files and .lisp files, you can also use .proto files directly in ASDF
261 systems. Just use the ASDF module type ``:proto`` in your system, and
262 compile and load the system in the usual way. This will create both the
263 Protobufs model and the Lisp classes that correspond to the Protobufs
264 messages. (Note that it will also leave a .lisp file having the same
265 name as the .proto file in the file system.)
268 Using the Protobufs macros
269 --------------------------
271 You can define a Protobufs schema entirely within Lisp by using the
272 following macros. For example::
274 (proto:define-proto color-wheel
275 (:package com.google.colorwheel
276 :lisp-package color-wheel)
277 (proto:define-message color-wheel
278 (:conc-name color-wheel-)
280 (colors :type (proto:list-of color) :default ()))
281 (proto:define-message color
283 (name :type (or string null))
284 (r-value :type integer)
285 (g-value :type integer)
286 (b-value :type integer)
287 (proto:define-extension 1000 max))
288 (proto:define-extend color ()
289 ((opacity 1000) :type (or null integer)))
290 (proto:define-message get-color-request ()
291 (wheel :type color-wheel)
293 (proto:define-message add-color-request ()
294 (wheel :type color-wheel)
296 (proto:define-service color-wheel ()
297 (get-color (get-color-request color)
298 :options ("deadline" "1.0"))
299 (add-color (add-color-request color)
300 :options ("deadline" "1.0"))))
302 This will create the Protobufs model objects, Lisp classes and enum
303 types that correspond to the model. The .proto file of the same schema
308 package com.google.colorwheel;
310 import "net/proto2/proto/descriptor.proto";
312 extend proto2.MessageOptions {
313 optional string lisp_package = 195801;
314 optional string lisp_name = 195802;
315 optional string lisp_alias = 195803;
318 option (lisp_package) = "color-wheel";
321 required string name = 1;
322 repeated Color colors = 2;
326 optional string name = 1;
327 required int64 rValue = 2;
328 required int64 gValue = 3;
329 required int64 bValue = 4;
330 extensions 1000 to max;
334 optional int64 opacity = 1000;
337 message GetColorRequest {
338 required ColorWheel wheel = 1;
339 required string name = 2;
342 message AddColorRequest {
343 required ColorWheel wheel = 1;
344 required Color color = 2;
348 rpc GetColor (GetColorRequest) returns (Color) {
349 option deadline = "1.0";
351 rpc AddColor (AddColorRequest) returns (Color) {
352 option deadline = "1.0";
356 Note that Lisp types ``(or null <T>)`` turn into optional fields,
357 and Lisp types ``(proto:list-of <T>)`` turn into repeated fields.
362 proto:define-proto (type (&key name syntax import [Macro]
364 optimize options documentation)
367 Defines a Protobufs "schema" whose name is given by the symbol *type*,
368 corresponding to a .proto file of that name. By a "schema", we mean an
369 object that corresponds to the contents of one .proto file. If *name*
370 is not supplied, the Protobufs name of the schema is the camel-cased
371 rendition of *type* (e.g., ``color-wheel`` becomes ``ColorWheel``);
372 otherwise the Protobufs name is the string *name*.
374 *imports* is a list of pathname strings to be imported. This corresponds
375 to ``import`` in a .proto file. Note that ``proto:define-proto`` can
376 import both .proto files and .lisp files containing Protobufs macros,
377 but the generated .proto code will convert all of these to imports of
380 *syntax* and *package* are strings that give the Protobufs syntax and
381 package name. *lisp-package* can be supplied to give the name of the
382 Lisp package, if it is different from *package*. *package* corresponds
383 to ``package`` in a .proto file. If you want to specify a Lisp package
384 in a .proto file, you can use ``option (lisp_package)``.
386 *optimize* can be either ``:space`` (the default) or ``:speed``. When it
387 is ``:space`` the serialization methods generated for each message are
388 compact, but slower; when it is ``:speed``, the serialization methods
389 will be much faster, but will take more space. This corresponds to
390 ``option optimize_for = CODE_SIZE|SPEED`` in a .proto file.
392 *options* is a property list whose keys and values are both strings,
393 for example, ``:option ("java_package" "com.yoyodyne.overthruster")``.
394 They are passed along unchanged to the generated .proto file.
396 *documentation* is a documentation string that is preserved as a comment
399 *body* consists of any number of calls to ``proto:define-enum``,
400 ``proto:define-message``, ``proto:define-extend`` or ``proto:define-service``.
405 proto:define-enum (type (&key name conc-name alias-for [Macro]
406 options documentation)
409 Defines a Protobufs enum type and a corresponding Lisp deftype whose name
410 is given by the symbol *type*. If *name* is not supplied, the Protobufs
411 name of the enum is the camel-cased rendition of *type*; otherwise the
412 Protobufs name is the string *name*. If *conc-name* is given, it will
413 be used as the prefix for all of the enum value names. In a .proto file,
414 you can use ``option (lisp_name)`` to override the default name for the
417 If *alias-for* is given, no Lisp deftype is defined. Instead, the enum
418 will be used as an alias for an enum type that already exists in Lisp.
419 You can use ``option (lisp_alias)`` in a .proto file to give the Lisp
420 alias for an enum type.
422 *options* is a property list whose keys and values are both strings.
424 *documentation* is a documentation string that is preserved as a comment
427 *body* consists of the enum values, each of which is either a symbol
428 or a list of the form ``(name index)``. By default, the indexes start at
429 0 and are incremented by 1 for each new enum value.
431 This can only be used within ``proto:define-proto`` or ``proto:define-message``.
436 proto:define-message (type (&key name conc-name alias-for [Macro]
437 options documentation)
440 Defines a Protobuf message and a corresponding Lisp defclass whose name
441 is given by the symbol *type*. If *name* is not supplied, the Protobufs
442 name of the class is the camel-cased rendition of *type*; otherwise the
443 Protobufs name is the string *name*. If *conc-name* is given, it will
444 be used as the prefix for all of the slot accessor names. In a .proto
445 file, you can use ``option (lisp_name)`` to override the default name
446 for the class in Lisp.
448 If *alias-for* is given, no Lisp defclass is defined. Instead, the
449 message will be used as an alias for a class that already exists in
450 Lisp. This feature is intended to be used to define messages that will
451 be serialized from existing Lisp classes; unless you get the slot names,
452 readers and writers exactly right for each field, it will be the case
453 that trying to (de)serialize into a(n aliased) Lisp object won't work.
454 You can use ``option (lisp_alias)`` in a .proto file to give the Lisp
455 alias for the class corresponding to a message.
457 *options* is a property list whose keys and values are both strings.
459 *documentation* is a documentation string that is preserved as a comment
462 The body *fields* consists of fields, ``proto:define-enum``,
463 ``proto:define-message`` or ``proto:define-extension`` forms.
465 Fields take the form ``(slot &key type name default reader writer)``.
466 *slot* can be either a symbol giving the slot name or a list of the
467 form ``(slot index)``. By default, the field indexes start at 1 and
468 are incremented by 1 for each new field value. *type* is the type of
469 the slot. *name* can be used to override the defaultly generated
470 Protobufs field name (for example, ``color-name`` becomes
471 ``colorName``). *default* is the default value for the slot. *reader*
472 is a Lisp slot reader function to use to get the value during
473 serialization, as opposed to using ``slot-value``; this is meant to be
474 used when aliasing an existing class. *writer* can be similarly used
475 to give a Lisp slot writer function.
477 This can only be used within ``proto:define-proto`` or ``proto:define-message``.
482 proto:define-extend (type (&key name [Macro]
483 options documentation)
486 Defines a Protobuf "extend", that is, an extension to an existing
487 message (and corresponding Lisp class) that has additional fields that
488 were reserved by ``proto:define-extension``. *type* and *name* are as
489 for ``proto:define-message``. Note that no new Lisp class is defined;
490 the additional slots are implemented as getter and setter methods on
491 a closed-over variable. The other options, such as *conc-name* and
492 *alias-for* are take from the extended message.
494 *options* is a property list whose keys and values are both strings.
496 *documentation* is a documentation string that is preserved as a comment
499 The body *fields* consists only of fields, which take the same form as
500 they do for ``proto:define-message``.
502 This can only be used within ``proto:define-proto`` or ``proto:define-message``.
507 proto:define-extension (from to) [Macro]
509 Defines a field extension for the indexes from *from* to *to*.
510 *from* and *to* are positive integers ranging from 1 to 2^29 - 1.
511 *to* can also be the token ``max``, i.e., 2^29 - 1.
513 This can only be used within ``proto:define-message``.
518 proto:define-service (type (&key name [Macro]
519 options documentation)
522 Defines a Protobufs service named *type* and corresponding Lisp generic
523 functions for all its methods. If *name* is not supplied, the Protobufs
524 name of the service is the camel-cased rendition of *type*; otherwise
525 the Protobufs name is the string *name*.
527 *options* is a property list whose keys and values are both strings.
529 *documentation* is a documentation string that is preserved as a comment
532 The body is a set of method specs of the form
533 ``(name (input-type output-type) &key options documentation)``.
534 *name* is a symbol naming the RPC method. *input-type* and
535 *output-type* may either be symbols or a list of the form ``(type &key name)``.
537 This can only be used within ``proto:define-message``.
540 Serializing and deserializing
541 =============================
543 You can serialize from Lisp objects or deserialize into Lisp objects
544 using either the fast and compact Protobufs wire format, or the
545 human-readable text format.
553 proto:serialize-object-to-stream (object type [Function]
556 Serializes the object *object* of type *type* onto the stream *stream*
557 using the wire format. *type* is the Lisp name of a Protobufs message
558 (often the name of a Lisp class) or a ``proto:protobuf-message`` object.
559 *type* defaults to the class of *object*
561 The element type of *stream* must be ``(unsigned-byte 8)``.
563 *visited* is an ``eql``-hash table used to cache object sizes. If it is
564 supplied, it will be cleared before it is used; otherwise, a fresh table
567 The returned value is a byte vector containing the serialized object.
568 If the stream is ``nil``, the buffer is not actually written anywhere.
573 proto:serialize-object (object type buffer [Generic function]
574 &optional start visited)
576 Serializes the object *object* of type *type* into the byte array
577 *buffer* using the wire format. *type* is the Lisp name of a Protobufs
578 message (often the name of a Lisp class) or a ``proto:protobuf-message``
579 object. *type* defaults to the class of *object*. The buffer is assumed
580 to be large enough to hold the serialized object; if it is not, an
581 out-of-bounds condition may be signalled.
583 The object is serialized using the wire format into the byte array
584 (i.e., a vector whose type is ``(unsigned-byte 8)``) given by *buffer*,
585 starting at the fixnum index *start* .
587 *visited* is an ``eql``-hash table used to cache object sizes.
589 The returned values are the modified buffer containing the serialized
590 object and the index that points one past the last serialized byte in
591 the buffer, which will be the number of bytes required to serialize the
592 object if *start* was 0.
597 proto:deserialize-object-from-stream (type &key stream) [Function]
599 Deserializes an object of the given type *type* as a Protobuf object.
600 *type* is the Lisp name of a Protobufs message (usually the name of a
601 Lisp class) or a ``proto:protobuf-message``.
603 The element type of *stream* must be ``(unsigned-byte 8)``.
605 The returned value is the deserialized object.
610 proto:deserialize-object (type buffer &optional start end) [Generic function]
612 Deserializes an object of the given type *type* as a Protobufs object.
613 *type* is the Lisp name of a Protobufs message (usually the name of a
614 Lisp class) or a ``proto:protobuf-message``.
616 The encoded bytes come from the byte array given by *buffer*, starting
617 at the fixnum index *start* up to the end of the buffer, given by *end*.
618 If a zero byte is encountered in in the "tag position" during
619 deserialization, this is interpreted as an "end of object" marker
620 and deserialization stops.
622 The returned values are the deserialized object and the index into the
623 buffer at which the deserialization ended.
628 proto:object-size (object type &optional visited) [Generic function]
630 Computes the size in bytes of the object *object* of type *type*.
631 *type* is the Lisp name of a Protobufs message (usually the name of a
632 Lisp class) or a ``proto:protobuf-message``. *type* defaults to the
635 *visited* is an ``eql``-hash table used to cache object sizes.
637 The returned value is the size of the serialized object in bytes.
645 proto:print-text-format (object &optional type [Function]
646 &key stream suppress-line-breaks)
648 Prints the object *object* of type *type* onto the stream *stream* using
649 the textual format. *type* defaults to the class of *object*.
651 If *suppress-line-breaks* is true, all the output is put on a single line.
656 proto:parse-text-format (type &key stream) [Function]
658 Parses the textual format of an object of the given type *type*. *type*
659 is the Lisp name of a Protobufs message (usually the name of a Lisp
660 class) or a ``proto:protobuf-message``. The input is read from the
663 The returned value is the object.
666 Python compatibility functions
667 ==============================
669 By popular demand, the Protobufs library provides an API that is very
670 similar to the API of the Python Protobufs library.
674 proto:clear (object) [Generic function]
676 Initializes all of the fields of *object* to their default values.
681 proto:is-initialized (object) [Generic function]
683 Returns true iff all of the fields of *object* are initialized, i.e.,
684 there are no fields whose value is unbound.
689 proto:serialize (object &optional buffer start end) [Generic function]
691 Serializes *object* into *buffer* using the wire format, starting at the
692 index *start* and going no further than *end*. *object* is an object
693 whose Lisp class corresponds to a Protobufs message.
697 proto:merge-from-array (object buffer &optional start end) [Generic function]
699 Deserializes the object encoded in *buffer* into *object*, starting at
700 the index *start* and ending at *end*. *object* is an object whose Lisp
701 class corresponds to a Protobufs message.
706 proto:octet-size (object) [Generic function]
708 Returns the number of bytes required to serialize *object* using the
709 wire format. *object* is an object whose Lisp class corresponds to a