Scott McKay [Wed, 12 Sep 2012 19:09:46 +0000 (19:09 +0000)]
Fix some user-reported bugs
Testing : precheckin --full --strict-errors
Reviewer: Sergey V, Shaun M
JTB impact: No
Ops impact: No
Change to config : No
Change to XML schema : No
Change to DB schema : No
Change to transport (timeouts, headers) : No
Any change (or new use) of OAQs : No
Change to inter-component transactions : No
Depends on any other checkin / bug : No
Tests that will verify:
The CL-Protobufs tests
Description:
I had made a "policy decision" that the package created
by importing a .proto file would (:use :common-lisp). In
practice, this turned out to be a mistake. Sergey suggested
a fix that I think is correct, so in it goes.
Shaun noticed that the optimized 'serialize-object' and
'object-size' methods didn't quite implement what they were
supposed to for optional boolean fields whose value was never
supplied. Fix the optimized methods to implement the same
(correct) semantics as the unoptimized ones.
Scott McKay [Mon, 10 Sep 2012 17:38:33 +0000 (17:38 +0000)]
Add a more readable syntax for enum and field indices
Testing : precheckin --full --strict-errors
Reviewer: Fare (please)
JTB impact: No
Ops impact: No
Change to config : No
Change to XML schema : No
Change to DB schema : No
Change to transport (timeouts, headers) : No
Any change (or new use) of OAQs : No
Change to inter-component transactions : No
Depends on any other checkin / bug : No
Tests that will verify:
I extended the CL-Protobufs tests
Description:
Add a more readable syntax for enum and field indices.
For define-enum, it was (name value).
Allow (name :index value).
For define-message, it was ((name index) ...).
Allow (name :index index ...).
Complain if both forms appear in the same field.
Scott McKay [Fri, 7 Sep 2012 15:22:28 +0000 (15:22 +0000)]
Add a few missing features that aren't in any documentation.
Testing : precheckin --full --strict-errors
Reviewer: Fare (please)
JTB impact: No
Ops impact: No
Change to config : No
Change to XML schema : No
Change to DB schema : No
Change to transport (timeouts, headers) : No
Any change (or new use) of OAQs : No
Change to inter-component transactions : No
Depends on any other checkin / bug : No
Tests that will verify:
I extended the CL-Protobufs examples
Description:
Add a few missing features that aren't in any documentation.
String literals can look like "foo"<whitespace>"bar".
- Fix 'parse-string' to handle this.
Option values can be complex structures, not just atoms.
- If 'parse-proto-option' sees a '{' character after the '=',
it should use 'parse-text-format' to read the option value.
- Fix the 'protobuf-option' printer to call 'print-text-format'
for complex option values.
- Minor refactoring to the text format parser and printer in
order to support the above.
Scott McKay [Tue, 4 Sep 2012 18:36:57 +0000 (18:36 +0000)]
Add stub support for streaming RPC APIs.
Testing : precheckin --full --strict-errors
Reviewer: Fare (please)
JTB impact: No
Ops impact: No
Change to config : No
Change to XML schema : No
Change to DB schema : No
Change to transport (timeouts, headers) : No
Any change (or new use) of OAQs : No
Change to inter-component transactions : No
Depends on any other checkin / bug : No
Tests that will verify:
I extended the CL-Protobufs tests
Description:
Add support for a 'streams' type in Protobufs RPC methods.
- Add a few slots to 'protobufs-method'
- Add '&key streams' to the method arglists in 'define-service'
- Extend the parser to handle "streams" in addition to the
'streams_type' option
- Extend the printer
- Change a test to use a 'streams' type
- While we're in the neighborhood, allow an optional '=>'
between the input and output methods in 'define-service';
it makes it a bit more readable
- Update the documentation
Scott McKay [Fri, 31 Aug 2012 15:15:49 +0000 (15:15 +0000)]
Add type aliases to CL-Protobufs
Testing : precheckin --full --strict-errors
Reviewer: Fare (please)
JTB impact: No
Ops impact: No
Change to config : No
Change to XML schema : No
Change to DB schema : No
Change to transport (timeouts, headers) : No
Any change (or new use) of OAQs : No
Change to inter-component transactions : No
Depends on any other checkin / bug : No
Tests that will verify:
The CL-Protobufs tests
Description:
Add type aliases to CL-Protobufs. It's a Lisp-only
feature that allows you define a new Protobufs type in
terms of a Lisp type, a serializer and a deserializer.
- Add a new model class, 'protobuf-type-alias'.
- Extend schemas and messages so that they can hold
a set of type aliases; add 'find-type-alias'.
- Add 'define-type-alias'.
- Factor out 'lisp-type-to-protobuf-type' from
'clos-type-to-protobuf-type', I need it for the
'define-type-alias' macro.
- Fix the schema printer to print 'define-type-alias'
for .lisp schemas, and to add a comment describing
the type alias in .proto schemas.
- Extend the wire format (de)serializer to know
about type aliases; it just (de)serializes the
Lisp type as the Protobufs primitive type using
the provided (de)serializer functions. There's
zero cost to this if you don't use aliases.
- Update the optimized serialization generation.
- Similarly extend the text format (de)serializer
to know about type aliases.
- Change an example to use a type alias for 'date'.
- Update the documentation.
Scott McKay [Thu, 28 Jun 2012 20:42:55 +0000 (20:42 +0000)]
It turns out that being able to get from method -> service -> schema
makes my CL-Stubby client code simpler. So, fill in the 'parent'
slot for more than just messages. No cost at runtime, so WTH.
Passes 'precheckin' with the new Protobufs unit tests in place.
CL-Stubby tests pass in Google3 environment.
Scott McKay [Wed, 20 Jun 2012 17:39:41 +0000 (17:39 +0000)]
Random things discovered while working on Stubby support...
- Make the 'find-xxx' support searching "relative to" another namespace.
- There was a bug in the non-optimized deserializer when deserializing
a repeated slot into a vector; create a stretchy vector on demand
if it's needed.
- 'define-extends' should wrap 'eval-when' around the generated 'defsetf'
forms so that they are visible at compile time.
- Fix a formatting bug in the export list in the Lisp printer.
- Straighten of the ASDF declaration for the tests.
- Add the Protobufs test suite to QRes, to keep things honest.
Passes 'precheckin' with the new Protobufs unit tests in place.
Scott McKay [Mon, 18 Jun 2012 15:00:19 +0000 (15:00 +0000)]
Still better support for qualified names:
- Add 'make-qualified-name' function that knows how to prepend
message and package names.
- All the places that create messages, enums, fields, etc, now
generate a qualified name.
- 'find-qualified-name' now thoroughly checks the qualified name.
Passes 'precheckin' with the new Protobufs unit tests in place.
Scott McKay [Mon, 11 Jun 2012 17:13:55 +0000 (17:13 +0000)]
While implementing the Fortuneteller stubby server, I ran into
some things that needed to be addressed:
- We need to do a more thorough search of namespaces so that
qualified references to imported messages, etc, work properly.
It's still not 100%, but it covers most of what I can find.
- Name mappers now do their best to preserve qualified names.
- 'find-xxx' methods search the qualified names.
- Fix the importation process to cooperate better with ASDF, i.e.,
don't bother re-importing things that have already compiled
as part of an ASDF compile/load operation.
- When generating .lisp from .proto, if no package has been
declared, do it. And fill it with nice exports, too. This
won't be exactly right, but it'll make it easier to create
a proper package declaration, if you so desire.
- Do a better job of generating client- and server-side stubs.
Passes 'precheckin' with the new Protobufs unit tests in place.
And more to the point, the Fortuneseeker server now works as a
Stubby client using my new Google3 http2 and rpc2 libraries.
Scott McKay [Tue, 5 Jun 2012 17:46:46 +0000 (17:46 +0000)]
SBCL kindly offered suggestions for improvement:
- Use 'defgeneric' with ':method' for things that are meant to
have just a single method.
- Lose some unreachable code in the parser.
- Weak hash tables in SBCL need to use ':weakness :value'.
- The wire-level stuff can use some more type declarations.
- Fix a package problem in one of the examples.
Passes 'precheckin' with the new Protobufs unit tests in place.
Scott McKay [Wed, 30 May 2012 20:58:28 +0000 (20:58 +0000)]
More support for Stubby and use in Google3 in general:
- More sophisticated and easier to use ASDF support.
- Add search paths to the :protobuf-file ASDF module
Passes 'precheckin' with the new Protobufs unit tests in place.
Scott McKay [Fri, 25 May 2012 20:24:29 +0000 (20:24 +0000)]
Get some stuff working better for Stubby purposes.
- Better implementation of 'uncamel-case' gives better names.
- Add an index slot to 'protobuf-method' so that we can more
easily do automatic service registration.
- Give a better type to the index slot in 'protobuf-message'
and 'protobuf-enum'.
- Parser and macros now assign an index to each service method.
- Define 'find-service' and 'find-method' for use by Stubby.
Passes 'precheckin' with the new Protobufs unit tests in place.
Scott McKay [Fri, 25 May 2012 14:46:50 +0000 (14:46 +0000)]
Fix a few more things discovered by the tests:
- 'reinitialize-slot' didn't quite work.
- Add a geodata example that uses vectors for repeated fields,
which uncovered a bug in the optimized deserializers.
- Importing the geodata CLOS classes revealed a bug in default
handling when the default is provided only in 'defclass'.
- Fix the knock-on bug in deserialization and the optimized
(de)serialization caused the above.
- Add tighter types in 'decode-uint32' and 'decode-uint64'.
Passes 'precheckin'. Even with the new unit tests in place.
Scott McKay [Wed, 23 May 2012 18:17:11 +0000 (18:17 +0000)]
Add 'vector-of' type, at Shaun Morris's request. This required
some additions to the macrology and to the (de)serialization
code. No runtime performance cost to being able to have things
both ways.
Fix a bug that caused unprocessed options to get dropped.
Improve some of the unit tests.
Passes 'precheckin'. Even with the new unit tests in place.
Scott McKay [Tue, 22 May 2012 20:16:41 +0000 (20:16 +0000)]
Add more stuff to the test suite, and fix what it uncovered.
- Behavior of ':default xxx'/[default=xxx] was not stable,
so add an "empty default" marker and respect it during
parsing and printing.
- 'find-option' now returns a third "found-p" value, useful
for the better handling of defaults.
- Don't print slot readers/writers if they have the same name
as what would be (silently) generated by the macrology.
- Don't print 'import descriptor.proto' lines if there are
no 'lisp_xxx' options in the generated code.
- Add 'clear-field' to the API; add the functions it needs.
- Be more careful about how :conc-name interacts with the
generation of readers/writers.
- When parsing from a .proto file, force the conc-name to
be "" unless requested explicitly not to. This has the
effect of generating readers/writers for all the fields
of imported Protobufs messages, which I think is probably
the desired behavior.
- Add 'make-byte-vector' for convenience.
Passes 'precheckin'. Even with the new unit tests in place.
Scott McKay [Fri, 18 May 2012 21:04:02 +0000 (21:04 +0000)]
Now that Protobufs has a test suite, it found a few things to fix.
- Don't generate warnings for anonymous enums, they're harmless
and ubiquitous.
- 'member' types where are all the members are keywords, is a symbol,
which got converted to a Protobufs 'string'. Wrong.
- If a field has an enum type, the (Lisp) default value should be
treated as a keyword.
- When parsing a .proto file or using the 'define-xxx' macros, any
options that are handled specially should be trimmed from the
options list so that they don't get printed twice.
- Add 'remove-options' for the above.
- Make the options printer be more type-aware.
- Clean up examples.lisp
- 'schemas-equal' can ignore the schema name/class if they're null.
Passes 'precheckin'. Even with the new unit tests in place.
Update the Quake Protobufs schema generator to use the new names.
The Protobufs documentation doesn't say so, but enums can be packed.
- Update the wire-level protocol to (de)serialize packed enums.
- Update the serialization code to use the new wire functions.
Scott McKay [Wed, 16 May 2012 17:01:06 +0000 (17:01 +0000)]
Tie up a few loose ends with 'unittests.proto'; there are still a few
edge case problems, but it's good enough to continue the Stubby work.
The critical .proto files (descriptor.proto, etc) all appear to work.
More precise mapping of (mainly integer) types from .lisp to .proto files
so that we can a truly exact model once we compile the .lisp file. It may
not make any difference, but better safe than sorry.
- Introduce Lisp type 'int32', 'int64', 'uint32', etc
- Make Lisp->Protobufs recognize the new Lisp types
- Make Protobufs schema printers recognize the new types
Fix up the handling of imports and namespaces.
- Always use truenames to record schemas
- Fix typo in 'find-enum' that caused it never to search other schemas
- Record the schema in *all-protobufs* at exactly the right time,
and simplify 'make-load-form' methods
A few tweaks to (de)serialization.
- Make the bodies for empty messages not generate compiler warnings
- Don't make ':packed t' be the default, the user has to say so
- The user documentation doesn't say so, but booleans can be packed.
Scott McKay [Mon, 14 May 2012 23:22:59 +0000 (23:22 +0000)]
A few more model fixes to fully support google/protobuf/unittest.proto
(I love this file, it's got one of everything.)
- The printer should not print extended fields in any message
that further extends an extended message.
- If parsing or importing a .proto file needs a Lisp package that
does not exist, go ahead and create it.
- Importing now tracks all the schemas imported by the "parent",
this so that we can search namespaces thoroughly.
- It looks like 'define-extend' can accept groups. For a deprecated
feature, it sure is a pain in the neck.
- Make 'find-message' and 'find-enum' search all namespaces.
- Print escaped strings, just for completeness.
- Annotate a couple of common options with their types.
Passes 'precheckin'. Passes my by-hand tests and is epsilon away from
handling the worst .proto file around.
Scott McKay [Sun, 13 May 2012 15:17:06 +0000 (15:17 +0000)]
Get google/protobuf/unittest.proto working, insofar as it parses
completely and produces what appears to be a correct model. Next
up, ensure (de)serialization works with this complex beast, which
sets the stage for running the (very complete) Google tests.
- Be much more careful to preserve type information in option
values and field defaults, both in the macros and the parser.
Make the slots in their model objects be untyped.
- This makes it easier for the Proto and Lisp printers to produce
exactly the right output, so fix them, too. Be extra careful
with booleans.
- Fix the printer to recognize more of the standard options so
that their values get printed with the correct type.
- Learn to parse more complicated numeric and integer constants.
- Learn to parse escaped strings.
- Learn to parse comma-separated options.
- Learn to parse extensions with no 'to' specifier.
- Fix the places that used (deprecated) :single instead of :float.
- Proto enum values always map to Lisp keywords, so don't futz
with packages for them.
- Add some more doc strings.
- Add some more examples-that-will-become-tests.
Scott McKay [Fri, 11 May 2012 20:17:48 +0000 (20:17 +0000)]
Now that I've started importing the Google tests for Protos,
a few more bugs are showing up:
- Add basic handling for namespaces
- A typo in my own tests caused me to miss the fact that
negative signed integers did not deserialize correctly
(thanks, Shaun!)
- Just dike out the rest of the Quux dependencies, since it's
pretty clear non-QRes folks will be using it
Scott McKay [Thu, 10 May 2012 21:14:58 +0000 (21:14 +0000)]
Fix some problems reported by Shaun Morris:
- Looks like Stubby will need 'merge-from-array', make room for that.
- Text format deserializer didn't correctly parse negative integers.
- Simply 'deserialize-object' a bit
- Fix a bug in the 'deserialize-object' that caused it not to work
correctly on extended fields.
- Fix a sloppy bug in 'make-tag' that only showed up in SBCL.
- Add a few more examples, soon to be part of a test suite.
Scott McKay [Thu, 10 May 2012 16:13:14 +0000 (16:13 +0000)]
D'oh, I didn't check in 'defgeneric' in the previous commit,
which caused a unit test to fail. Sorry about that.
Make 'make-load-form' for protobuf schemas and messages update
*all-protobufs* and *all-messages*, since 'initialize-instance'
doesn't get called at load time (something I did not know!).
This time it passes 'precheckin --full', and I'll be sure to
commit all the files.
Scott McKay [Tue, 8 May 2012 18:10:35 +0000 (18:10 +0000)]
Fix some corner initialization corner cases.
In particular, uninitialized optional boolean fields
whose Lisp type is (or (member t nil) null) now work.
Implement 'object-initialized-p' and 'slot-initialized-p'
and their Python lookalikes, 'is-initialized' and 'has-field'.
Update the documentation.
Passes 'precheckin --full+', which makes sense since none
of this is called in normal use yet.
Scott McKay [Fri, 4 May 2012 20:00:39 +0000 (20:00 +0000)]
Well, it turns out that the Protobufs 'group' feature,
which has been deprecated for years, is still in wide
use, e.g., in Chubby's bnsresolver.proto
So, implement support for groups:
- Add a model class for it
- Add .proto and .lisp printers
- Add a 'define-group' macro
- Make the .proto parser know how to parse them
Passes 'precheckin --full+', which makes sense since none
of this is used in normal use yet.
Scott McKay [Wed, 25 Apr 2012 17:53:53 +0000 (17:53 +0000)]
Merge $SVN/branches/qres/swm/borgify-1/qres back into the trunk.
Major changes include:
- Simplification of 'define-db-enumeration',
and fixing clients to use simpler name conventions (no more {})
- Removal of unused Quake functionality (courtesy of Alejandro),
including memcached, Oracle O-R record loaders, Quake-OCI c code
- Addition of MySQL schema generator
- Addition of Protobufs support
(see $Q/lisp/quux/protobugs/cl-protobufs.rst)
Passes 'precheckin --full+'. Changes mutually review by swm and asedeno.
projects / cl-protobufs.git / log