3 \setlength{\parindent}{0pt}
4 \setlength{\parskip}{6pt plus 2pt minus 1pt}
7 =========================
8 Protobufs for Common Lisp
9 =========================
12 :Description: Protobufs for Common Lisp
13 :Author: Scott McKay <swm@google.com>
14 :Date: $Date: 2012-05-07 14:58:00 -0500 (Mon, 7 May 2012) $
19 1.1 Implementation notes
21 2 Defining a Protobufs schema
22 2.1 .proto file to Lisp conversion
23 2.2 CLOS classes to .proto conversion
24 2.3 Using .proto files directly
25 2.4 Using the Protobufs macros
26 3 Serializing and deserializing
30 4.1 Extensions functions
31 4.2 Initialization functions
32 4.3 Python compatibility functions
38 The Common Lisp Protobufs library provides a way for Common Lisp
39 programs to use existing (or define new) Protobufs "schemas", and
40 serialize and deserialize objects to and from the Protobufs wire and
43 To use it, first load the ASDF declaration file ``protobufs/protobufs.asd``
44 and then use ASDF to load the library named ``:protobufs``.
50 The Protobufs library defines a set of model classes that describes a
51 protobufs "schema" (i.e., one .proto file). There is a class that
52 describes a schema, its options, enums and enum values, messages and
53 fields, and services and methods.
55 The library provides the means to convert several kinds of inputs into
56 the Protobufs models, including:
58 - Parse an existing .proto file into a set of model objects.
59 - Convert a set of related CLOS classes into a set of model objects.
60 - Compile a ``proto:define-schema`` macro call into a set of model objects.
62 It also provides two ways to convert the model objects into outputs:
64 - Print a set of model objects using the standard Protobufs v2 syntax.
65 - Print a set of model objects using the Common Lisp syntax, defined below.
67 There are two formats for serialization and deserialization:
69 - The wire format, which is compact and fast.
70 - The text format, which is human readable.
72 Finally, there is a validator that takes an old version of a Protobufs
73 schema and a new version of the same schema and generates a set of
74 warnings that describes any incompatibilities between the old schema and
83 proto:protobuf-schema [Class]
85 The class the represents a Protobufs schema, i.e., one .proto file.
86 It has slots for the name, options, enums, messages and services. The
87 name is equal to the name of the .proto file, without the file type.
91 proto:protobuf-option [Class]
93 The class that represents a Protobufs option.
94 It has slots for a key and a value.
98 proto:protobuf-enum [Class]
100 The class that represents a Protobufs enum type.
101 It has slots for the enum name and its values.
105 proto:protobuf-enum-value [Class]
107 The class that represents one value in a Protobufs enum type.
108 It has slots for the value name and the value index.
112 proto:protobuf-message [Class]
114 The class that represents a Protobufs message.
115 It has slots for the name, options, nested enums and messages, and fields.
119 proto:protobuf-field [Class]
121 The class that represents one field in a Protobufs message.
122 It has slots for the name, type, index and options.
126 proto:protobuf-service [Class]
128 The class that represents a Protobufs service.
129 It has slots for the name, options and methods.
133 proto:protobuf-method [Class]
135 The class that represents one method description in a Protobufs service.
136 It has slots for the name, input type, output type and options.
139 Defining a Protobufs schema
140 ===========================
142 There are several ways to define a Protobufs schema: convert a .proto
143 file to a .lisp source file and then use the Lisp file; convert a set
144 of Lisp classes to a Protobufs model, and then use either the .lisp or
145 the .proto representation of the model; use a .proto file directly in
146 an ASDF system; or use the Protobufs macros in a Lisp source file.
149 .proto file to Lisp conversion
150 ------------------------------
152 If you have an existing .proto source file that you would like to
153 convert to Lisp classes (more precisely, to the macros defined by the
154 Protobufs library), you can use ``proto:parse-schema-from-file`` to
155 read the .proto file and then use ``proto:write-schema`` to write a
156 new .lisp file. (This is what that ASDF module type ``:proto`` does.)
160 proto:parse-schema-from-file (filename) [Function]
162 Parses the contents of the file given by *filename*, and returns the
163 Protobufs model (a set object objects rooted at ``proto:protobuf-schema``)
164 corresponding to the parsed file. The name of the Protobufs schema is
165 generated automatically from the file name.
169 proto:parse-schema-from-stream (stream &key name class) [Function]
171 Parses the contents of the stream *stream*, and returns the Protobufs
172 schema corresponding to the parsed file. If *name* is supplied, it gives
173 the Protobufs name for the schema. If *class* is supplied, it gives the
178 proto:write-schema (schema &key stream type) [Function]
180 Pretty-prints the Protobufs schema *schema* onto the stream *stream*,
181 which defaults to ``*standard-output*``.
183 *type* can be either ``:proto`` or ``:lisp``.
186 CLOS classes to .proto conversion
187 ---------------------------------
189 If you have an existing set of CLOS classes that you would like to
190 convert to a Protobufs schema, you can use ``proto:generate-schema-from-classes``.
192 Note that the Protobufs schema is an *approximation* of a good schema.
193 You should review it and, if necessary, change it (and probably the Lisp
194 classes as well) until you have a good Protobufs schema definition.
198 proto:generate-schema-for-classes (classes [Function]
199 &key name package lisp-package
200 slot-filter type-filter enum-filter value-filter
201 alias-existing-classes)
203 Given a list of class names *classes*, this generates a Protobufs schema
204 for the classes, generating any necessary enum types that correspond to
205 Lisp ``member`` types. The return value is the model, rooted at an instance
206 of ``proto:protobuf-schema``.
208 *name* and *package* can be supplied to give the Protobufs name and
209 package. *lisp-package* can be supplied to give the name of the Lisp
210 package, if it is different from *package*. (Note that you should
211 still use ``in-package`` at the top of .lisp files, and it should
212 match the value of *lisp-package*.)
214 *slot-filter*, *type-filter*, *enum-filter* and *value-filter* are
215 filtering functions that can be used to weed out things from the Lisp
216 classes that should not be included in the Protobufs schema.
218 *slot-filter* is a function of two arguments, a list of all the slots
219 in the class and the slot currently being processed, and should return
220 true if the slot is to be kept or ``nil`` if it to be discarded. For
221 example, if there are internal implementation slots in a class that
222 need not appear in the Protobufs description, it can be used to filter
225 *type-filter* is a function of one argument, the type (of a slot); it
226 should return a "transformed" type if any is required. For example,
227 complex ``and`` and ``or`` types can't be directly represented in
228 Protobufs; this can be used to substitute something workable.
230 *enum-filter* is a function of one argument, a list of all the values
231 of a ``member`` type; it should return the transformed values. For
232 example, there maybe be some enumeration values that don't make sense;
233 they can be discarded by the filter.
235 *value-filter* is a function of one argument, the value of a slot
236 initform. It should transform the value into a scalar value suitable
239 If *alias-existing-classes* is true (the default), the generated
240 code will include ``:alias-for`` so that there will be no clash
241 with the existing Lisp class.
245 proto:write-schema-for-classes (classes [Function]
246 &key stream type name package lisp-package
247 slot-filter type-filter enum-filter value-filter
248 alias-existing-classes)
250 Given a list of class names *classes*, this generates a Protobufs schema
251 for the classes, generating enum types as necessary, and then
252 pretty-prints the result onto *stream*. *type* can be either ``:proto``
253 (the default) or ``:lisp``; it controls which format the generated
254 code will be printed in. The return value is the model, rooted at an
255 instance of ``proto:protobuf-schema``.
257 *name* and *package* can be supplied to give the Protobufs name and
258 package. *lisp-package* can be supplied to give the name of the Lisp
259 package, if it is different from *package*.
261 *slot-filter*, *type-filter*, *enum-filter* and *value-filter* are
262 as for ``proto:generate-schema-for-classes``.
264 *alias-existing-classes* is as for ``proto:generate-schema-for-classes``.
267 Using .proto files directly
268 ---------------------------
270 In addition to using the tools described above to convert between .proto
271 files and .lisp files, you can also use .proto files directly in ASDF
272 systems. Just use the ASDF module type ``:proto`` in your system, and
273 compile and load the system in the usual way. This will create both the
274 Protobufs model and the Lisp classes that correspond to the Protobufs
275 messages. (Note that it will also leave a .lisp file having the same
276 name as the .proto file in the file system.)
279 Using the Protobufs macros
280 --------------------------
282 You can define a Protobufs schema entirely within Lisp by using the
283 following macros. For example::
285 (proto:define-schema color-wheel
286 (:package com.google.colorwheel
287 :lisp-package color-wheel)
288 (proto:define-message color-wheel
289 (:conc-name color-wheel-)
291 (colors :type (proto:list-of color) :default ()))
292 (proto:define-message color
294 (name :type (or string null))
295 (r-value :type integer)
296 (g-value :type integer)
297 (b-value :type integer)
298 (proto:define-extension 1000 max))
299 (proto:define-extend color ()
300 ((opacity 1000) :type (or null integer)))
301 (proto:define-message get-color-request ()
302 (wheel :type color-wheel)
304 (proto:define-message add-color-request ()
305 (wheel :type color-wheel)
307 (proto:define-service color-wheel ()
308 (get-color (get-color-request color)
309 :options ("deadline" "1.0"))
310 (add-color (add-color-request color)
311 :options ("deadline" "1.0"))))
313 This will create the Protobufs model objects, Lisp classes and enum
314 types that correspond to the model. The .proto file of the same schema
319 package com.google.colorwheel;
321 import "net/proto2/proto/descriptor.proto";
323 extend proto2.MessageOptions {
324 optional string lisp_package = 195801;
325 optional string lisp_name = 195802;
326 optional string lisp_alias = 195803;
329 option (lisp_package) = "color-wheel";
332 required string name = 1;
333 repeated Color colors = 2;
337 optional string name = 1;
338 required int64 rValue = 2;
339 required int64 gValue = 3;
340 required int64 bValue = 4;
341 extensions 1000 to max;
345 optional int64 opacity = 1000;
348 message GetColorRequest {
349 required ColorWheel wheel = 1;
350 required string name = 2;
353 message AddColorRequest {
354 required ColorWheel wheel = 1;
355 required Color color = 2;
359 rpc GetColor (GetColorRequest) returns (Color) {
360 option deadline = "1.0";
362 rpc AddColor (AddColorRequest) returns (Color) {
363 option deadline = "1.0";
367 Note that Lisp types ``(or null <T>)`` turn into optional fields,
368 and Lisp types ``(proto:list-of <T>)`` turn into repeated fields.
372 proto:define-schema (type (&key name syntax import [Macro]
374 optimize options documentation)
377 Defines a Protobufs "schema" whose name is given by the symbol *type*,
378 corresponding to a .proto file of that name. By a "schema", we mean an
379 object that corresponds to the contents of one .proto file. If *name*
380 is not supplied, the Protobufs name of the schema is the camel-cased
381 rendition of *type* (e.g., ``color-wheel`` becomes ``ColorWheel``);
382 otherwise the Protobufs name is the string *name*.
384 *imports* is a list of pathname strings to be imported. This corresponds
385 to ``import`` in a .proto file. Note that ``proto:define-schema`` can
386 import both .proto files and .lisp files containing Protobufs macros,
387 but the generated .proto code will convert all of these to imports of
390 *syntax* and *package* are strings that give the Protobufs syntax and
391 package name. *lisp-package* can be supplied to give the name of the
392 Lisp package, if it is different from *package*. *package* corresponds
393 to ``package`` in a .proto file. If you want to specify a Lisp package
394 in a .proto file, you can use ``option (lisp_package)``.
396 *optimize* can be either ``:space`` (the default) or ``:speed``. When it
397 is ``:space`` the serialization methods generated for each message are
398 compact, but slower; when it is ``:speed``, the serialization methods
399 will be much faster, but will take more space. This corresponds to
400 ``option optimize_for = CODE_SIZE|SPEED`` in a .proto file.
402 *options* is a property list whose keys and values are both strings,
403 for example, ``:option ("java_package" "com.yoyodyne.overthruster")``.
404 They are passed along unchanged to the generated .proto file.
406 *documentation* is a documentation string that is preserved as a comment
409 *body* consists of any number of calls to ``proto:define-enum``,
410 ``proto:define-message``, ``proto:define-extend`` or ``proto:define-service``.
414 proto:define-enum (type (&key name conc-name alias-for [Macro]
415 options documentation)
418 Defines a Protobufs enum type and a corresponding Lisp deftype whose name
419 is given by the symbol *type*. If *name* is not supplied, the Protobufs
420 name of the enum is the camel-cased rendition of *type*; otherwise the
421 Protobufs name is the string *name*. If *conc-name* is given, it will
422 be used as the prefix for all of the enum value names. In a .proto file,
423 you can use ``option (lisp_name)`` to override the default name for the
426 If *alias-for* is given, no Lisp deftype is defined. Instead, the enum
427 will be used as an alias for an enum type that already exists in Lisp.
428 You can use ``option (lisp_alias)`` in a .proto file to give the Lisp
429 alias for an enum type.
431 *options* is a property list whose keys and values are both strings.
433 *documentation* is a documentation string that is preserved as a comment
436 *body* consists of the enum values, each of which is either a symbol
437 or a list of the form ``(name index)``. By default, the indexes start at
438 0 and are incremented by 1 for each new enum value.
440 ``proto:define-enum`` can be used only within ``proto:define-schema``
441 or ``proto:define-message``.
445 proto:define-message (type (&key name conc-name alias-for [Macro]
446 options documentation)
449 Defines a Protobuf message and a corresponding Lisp defclass whose name
450 is given by the symbol *type*. If *name* is not supplied, the Protobufs
451 name of the class is the camel-cased rendition of *type*; otherwise the
452 Protobufs name is the string *name*. If *conc-name* is given, it will
453 be used as the prefix for all of the slot accessor names. In a .proto
454 file, you can use ``option (lisp_name)`` to override the default name
455 for the class in Lisp.
457 If *alias-for* is given, no Lisp defclass is defined. Instead, the
458 message will be used as an alias for a class that already exists in
459 Lisp. This feature is intended to be used to define messages that will
460 be serialized from existing Lisp classes; unless you get the slot names,
461 readers and writers exactly right for each field, it will be the case
462 that trying to (de)serialize into a(n aliased) Lisp object won't work.
463 You can use ``option (lisp_alias)`` in a .proto file to give the Lisp
464 alias for the class corresponding to a message.
466 *options* is a property list whose keys and values are both strings.
468 *documentation* is a documentation string that is preserved as a comment
471 The body *fields* consists of fields, ``proto:define-enum``,
472 ``proto:define-message`` or ``proto:define-extension`` forms.
474 Fields take the form ``(slot &key type name default reader writer)``.
475 *slot* can be either a symbol giving the slot name or a list of the
476 form ``(slot index)``. By default, the field indexes start at 1 and
477 are incremented by 1 for each new field value. *type* is the type of
478 the slot. *name* can be used to override the defaultly generated
479 Protobufs field name (for example, ``color-name`` becomes
480 ``colorName``). *default* is the default value for the slot. *reader*
481 is a Lisp slot reader function to use to get the value during
482 serialization, as opposed to using ``slot-value``; this is meant to be
483 used when aliasing an existing class. *writer* can be similarly used
484 to give a Lisp slot writer function.
486 Note that the Protobufs does not support full Lisp type expressions in
487 the types of fields. The following type expressions are supported:
489 - ``integer``, optionally with upper and lower bounds
490 - ``signed-byte``, which correspond to ``proto:int32`` or ``proto:int64``
491 - ``unsigned-byte``, which correspond to ``proto:uint32`` or ``proto:uint64``
492 - ``float`` and ``double-float``
493 - ``string``and ``character``
494 - ``(array (unsigned-byte 8))``, which corresponds to ``proto:byte-vector``
496 - ``(member ...)``, where all the members are symbols or keywords or ``nil``
497 - the name of a class that corresponds to another Protobufs message
498 - ``(proto:list-of <T>)``, where ``<T>`` is any of the above types
499 - ``(or <T> null)``, where ``<T>`` is any of the above types
501 ``member`` corresponds to a Protobufs ``enum``, ``proto:list-of`` to
502 a repeated field, and ``(or <T> null)`` to an optional field. The other
503 types correspond to the various Protobufs scalar field types.
505 ``proto:define-message`` can be used only within ``proto:define-schema``
506 or ``proto:define-message``.
510 proto:define-extension (from to) [Macro]
512 Defines a field extension for the indexes from *from* to *to*.
513 *from* and *to* are positive integers ranging from 1 to 2^29 - 1.
514 *to* can also be the token ``max``, i.e., 2^29 - 1.
516 Once an extension to a message has been defined, you can use
517 ``proto:define-extends`` to add new fields.
519 ``proto:define-extension`` can be used only within ``proto:define-message``.
521 In non-Lisp implementations of Protobufs, you set and get the value
522 of an extension using functions like ``SetExtension()`` and
523 ``GetExtension()``. For example, if you extended a ``Color`` message
524 to have an ``opacity`` field, you would set the field using something
528 color.SetExtension(opacity, 0.5);
530 In Common Lisp Protobufs, you can just use an ordinary slot accessor::
532 (let ((color (make-instance 'color)))
533 (setf (color-opacity color) 0.5))
537 proto:define-extend (type (&key name [Macro]
538 options documentation)
541 Defines a Protobuf ``extend``, that is, an extension to an existing
542 message (and corresponding Lisp class) that has additional fields that
543 were reserved by ``proto:define-extension``. *type* and *name* are as
544 for ``proto:define-message``. Note that no new Lisp class is defined;
545 the additional slots are implemented as getter and setter methods on
546 a closed-over variable. The other options, such as *conc-name* and
547 *alias-for* are take from the extended message.
549 *options* is a property list whose keys and values are both strings.
551 *documentation* is a documentation string that is preserved as a comment
554 The body *fields* consists only of fields, which take the same form as
555 they do for ``proto:define-message``.
557 ``proto:define-extend`` can be used only within ``proto:define-schema``
558 or ``proto:define-message``.
562 proto:define-service (type (&key name [Macro]
563 options documentation)
566 Defines a Protobufs service named *type* and corresponding Lisp generic
567 functions for all its methods. If *name* is not supplied, the Protobufs
568 name of the service is the camel-cased rendition of *type*; otherwise
569 the Protobufs name is the string *name*.
571 *options* is a property list whose keys and values are both strings.
573 *documentation* is a documentation string that is preserved as a comment
576 The body is a set of method specs of the form
577 ``(name (input-type output-type) &key options documentation)``.
578 *name* is a symbol naming the RPC method. *input-type* and
579 *output-type* may either be symbols or a list of the form ``(type &key name)``.
581 ``proto:define-service`` can only be used within ``proto:define-schema``.
584 Serializing and deserializing
585 =============================
587 You can serialize from Lisp objects or deserialize into Lisp objects
588 using either the fast and compact Protobufs wire format, or the
589 human-readable text format.
597 proto:serialize-object-to-stream (object type [Function]
600 Serializes the object *object* of type *type* onto the stream *stream*
601 using the wire format. *type* is the Lisp name of a Protobufs message
602 (often the name of a Lisp class) or a ``proto:protobuf-message`` object.
603 *type* defaults to the class of *object*
605 The element type of *stream* must be ``(unsigned-byte 8)``.
607 *visited* is an ``eql`` hash table used to cache object sizes. If it is
608 supplied, it will be cleared before it is used; otherwise, a fresh table
611 The returned value is a byte vector containing the serialized object.
612 If the stream is ``nil``, the buffer is not actually written anywhere.
616 proto:serialize-object (object type buffer [Generic function]
617 &optional start visited)
619 Serializes the object *object* of type *type* into the byte array
620 *buffer* using the wire format. *type* is the Lisp name of a Protobufs
621 message (often the name of a Lisp class) or a ``proto:protobuf-message``
622 object. *type* defaults to the class of *object*. The buffer is assumed
623 to be large enough to hold the serialized object; if it is not, an
624 out-of-bounds condition may be signaled.
626 The object is serialized using the wire format into the byte array
627 (i.e., a vector whose type is ``(unsigned-byte 8)``) given by *buffer*,
628 starting at the fixnum index *start* .
630 *visited* is an ``eql`` hash table used to cache object sizes.
632 The returned values are the modified buffer containing the serialized
633 object and the index that points one past the last serialized byte in
634 the buffer, which will be the number of bytes required to serialize the
635 object if *start* was 0.
637 Note that ``proto:serialize-object`` will not correctly serialize a
638 set of objects that has cycles. You must resolve these yourself.
642 proto:deserialize-object-from-stream (type &key stream) [Function]
644 Deserializes an object of the given type *type* as a Protobuf object.
645 *type* is the Lisp name of a Protobufs message (usually the name of a
646 Lisp class) or a ``proto:protobuf-message``.
648 The element type of *stream* must be ``(unsigned-byte 8)``.
650 The returned value is the deserialized object.
654 proto:deserialize-object (type buffer &optional start end) [Generic function]
656 Deserializes an object of the given type *type* as a Protobufs object.
657 *type* is the Lisp name of a Protobufs message (usually the name of a
658 Lisp class) or a ``proto:protobuf-message``.
660 The encoded bytes come from the byte array given by *buffer*, starting
661 at the fixnum index *start* up to the end of the buffer, given by *end*.
662 *start* defaults to 0, *end*' defaults to the length of the buffer.
664 If a zero byte is encountered in in the "tag position" during
665 deserialization, this is interpreted as an "end of object" marker
666 and deserialization stops.
668 The returned values are the deserialized object and the index into the
669 buffer at which the deserialization ended.
673 proto:object-size (object type &optional visited) [Generic function]
675 Computes the size in bytes of the object *object* of type *type*.
676 *type* is the Lisp name of a Protobufs message (usually the name of a
677 Lisp class) or a ``proto:protobuf-message``. *type* defaults to the
680 *visited* is an ``eql`` hash table used to cache object sizes.
682 The returned value is the size of the serialized object in bytes.
690 proto:print-text-format (object &optional type [Function]
691 &key stream suppress-line-breaks)
693 Prints the object *object* of type *type* onto the stream *stream* using
694 the textual format. *type* defaults to the class of *object*.
696 If *suppress-line-breaks* is true, all the output is put on a single line.
700 proto:parse-text-format (type &key stream) [Function]
702 Parses the textual format of an object of the given type *type*. *type*
703 is the Lisp name of a Protobufs message (usually the name of a Lisp
704 class) or a ``proto:protobuf-message``. The input is read from the
707 The returned value is the object.
718 proto:get-extension (object slot) [Generic function]
720 Returns the value of the extended slot *slot* in the object *object*.
722 Since you can just use the ordinary slot reader function, you should
723 not need to call ``proto:get-extension``. It is included for compatibility
724 with other Protobufs APIs.
728 proto:set-extension (object slot value) [Generic function]
730 Sets the value of the extended slot *slot* in the object *object*
733 Since you can just use the ordinary slot writer function, you should
734 not need to call ``proto:set-extension``. It is included for compatibility
735 with other Protobufs APIs.
739 proto:has-extension (object slot) [Generic function]
741 Returns true iff the object *object* has any value for the extended
746 proto:clear-extension (object slot) [Generic function]
748 Removes the value for the extended slot *slot* in the object *object*.
751 Initialization functions
752 ------------------------
756 proto:object-initialized-p (object type) [Generic function]
758 Returns true iff all of the fields of *object* of type *type* are
759 initialized, i.e., there are no fields whose value is unbound.
763 proto:slot-initialized-p (object type slot) [Generic function]
765 Returns true iff the field *slot* of *object* of type *type* is
766 initialized, i.e., there are no fields whose value is unbound.
770 proto:reinitialize-object (object type) [Generic function]
772 Initializes all of the fields of *object* of type *type* to their
776 Python compatibility functions
777 ------------------------------
779 By popular demand, the Protobufs library provides an API that is very
780 similar to the API of the Python Protobufs library.
784 proto:is-initialized (object) [Generic function]
786 Returns true iff all of the fields of *object* are initialized, i.e.,
787 there are no fields whose value is unbound.
791 proto:has-field (object slot) [Generic function]
793 Returns true iff the field *slot* is initialized in *object*.
797 proto:clear (object) [Generic function]
799 Initializes all of the fields of *object* to their default values.
803 proto:serialize (object &optional buffer start end) [Generic function]
805 Serializes *object* into *buffer* using the wire format, starting at the
806 index *start* and going no further than *end*. *object* is an object
807 whose Lisp class corresponds to a Protobufs message.
811 proto:merge-from-array (object buffer &optional start end) [Generic function]
813 Deserializes the object encoded in *buffer* into *object*, starting at
814 the index *start* and ending at *end*. *object* is an object whose Lisp
815 class corresponds to a Protobufs message.
819 proto:octet-size (object) [Generic function]
821 Returns the number of bytes required to serialize *object* using the
822 wire format. *object* is an object whose Lisp class corresponds to a