Generate integer encoders and decoders with thought towards the size of a FIXNUM
Rather than hard-code assumptions about the size of a FIXNUM, check
the size of FIXNUM at compile time and generate code appropriate for
the number of bits we're working with, with FIXNUM optimizations
in place if appropriate.
"A common pattern is to define extensions inside the scope of the
extension's field type – for example, here's an extension to Foo of
type Baz, where the extension is defined as part of Baz"
Add a proper input-files method for proto-to-lisp,
which shall fix the issue that causes of these actions never being operation-done-p.
However this does NOT handle recursive import dependencies,
which requires further work.
Tested: QRes doesn't always recompile the proto and all dependees anymore.
Reviewer: asedeno
Ben Wagner [Tue, 20 Nov 2012 18:43:00 +0000 (13:43 -0500)]
fix cross-package and forward references in cl-protobufs
* Previously, if a field in a .proto file referenced a message in
another proto file using a different lisp package, the cl-protobufs
library would silently fail to serialize the field. A similar
problem would occur if a message defined later in the file used the
lisp_name option to override the name generated by cl-protobufs.
This change fixes these issues and others.
* Add conditions that are signaled when encountering an undefined
type.
* Delay assigning lisp classes/types to fields and methods until all
possible forward references have been parsed.
* This allows the class slot to be unbound, so check for that case
in print-object methods.
* Add a test for forward references to messages that override the
lisp name.
* Add a test for references to messages and enums defined in
another proto file with a different lisp package.
* Change color-wheel-stability test, because it used "string" as
the input type for an rpc, which seems to be disallowed (although
I haven't found this documented anywhere).
* Signal errors during parsing for undefined types.
* Add a test for these errors. Add assert-error macro to qtest.
* Signal a condition if we are unable to find the definition for a
field's type during serialization, deserialization, determining an
object's serialized size, printing text format, parsing text format,
or generating code for one of the above.
* Remove logic in find-qualified-name that indirects through lisp
packages. Proto packages and lisp packages do not necessarily map
1-to-1.
* Always use the schema's lisp package for any symbols generated when
parsing proto files.
* When generating lisp code using write-schema-as, set the package to
the package used in the generated file, so that ~s will print the
package prefix in the correct circumstances.
* Remove broken proto1 "streams" parsing ("returns" comes before
"streams" in every example I've found); replace with proto2 syntax.
* In process-imports, the call to find-schema using a pathname was not
giving the expected result. Sidestep this issue by using the same
logic to find the schema as is used earlier in the function.
Improve schema generation performance on large schemas
Avoid SUBTYPEP during macroexpansion, it can be very expensive on SBCL.
Actually recognize LIST-OF:LIST-OF constructs.
In DEFINE-SCHEMA, don't nest the DEFMETHODs inside the LET,
this also can cause SBCL's control flow analysis to blow up.
Update a few basic Lisp libraries.
Testing: [trunk r567444] precheckin --parallel 11 PASSED
Reviewers: dcrawford, asedeno
Rationale: there is a problem in quicklisp due to recent asdf package changes,
that requires urgent fix to cl-protobufs,
so I'm updating these for the sake of issuing
a cl-protobufs fix to quicklisp ASAP.
I require these updates for some further changes I'm making to QUUX, anyway,
so it's just a matter of timing.
Scott McKay [Tue, 18 Sep 2012 16:04:09 +0000 (16:04 +0000)]
Speed up unoptimized serialization
Testing : precheckin --full --strict-errors
Reviewer: Fare
JTB impact: No
Ops impact: No
Change to config : No
Change to XML schema : No
Change to DB schema : No
Change to transport (timeouts, headers) : No
Any change (or new use) of OAQs : No
Change to inter-component transactions : No
Depends on any other checkin / bug : No
Tests that will verify:
The CL-Protobufs tests
Description:
The unoptimized serialization code was slower than
it needed to be and was a Niagara falls of consing.
The problem is that (map () (curry ...)) is poorly
optimized, which is a shame because it's such a nice
coding style. :-P
The fix is to replace 'map' with iteration (sigh) in
a few critical places.
- Add 'doseq', which chooses between 'dolist' or 'dovector'.
- Make 'serialize-object', 'deserialize-object' and 'object-size'
use 'doseq' instead of using 'map'.
- Ditto for 'print-text-format'.
- Fix the 'serialize-packed' and 'packed-size' optimizers to
use 'dolist' or 'dovector' based on whether the field is
a vector field. This makes the optimized code faster, too.
- Make the optimized serializer generators pass in 'vectorp'
so that the wire-format optimizers can do a better job.
Tsunami framework + use tsunami in customer profiles
Testing : precheckin --full (both against Oracle and against megastore for customers)
Reviewer: eschwartz
JTB impact: n
Ops impact: n
Security impact: n
Change to config : n
Change to XML schema : Y (XMLREQ bug 120820)
Change to DB schema : Y (DBREQ bug 119801)
Change to transport (timeouts, headers) : n
Any change (or new use) of OAQs : n
Change to inter-component transactions : sort of (use megastore in development-only environments)
Depends on any other checkin / bug : n
Tests that will verify : tsunami and megastore unit tests + existing customer profile tests
Tsunami is a new layer that sits between business logic code and the
persistence logic. The main manifestation of this layer is
define-domain-class macro that defines an interface to a domain
class. You can then define how to perist in various implementations,
e.g. using define-quake-db-class-bindings for quake and
define-proto-binding & define-megastore-proto-binding for megastore.
Tsunami framework is still a work in progress, and there is a bunch of
outstanding tasks/issues.
As the first application, I refactored customer profile code to use
the tsunami framework. This enables storing of customer profiles in
megastore (or other data stores in the future), while everything else
is stored in Oracle.
Here is the list of other miscellaneous/harder-to-understand changes:
- I changed @id attribute of ContactInfo objects to use the actual
numeric ID, rather than passing it through encode-record-locator (in
qres-dev schema only). This required changing XML schema type from
typeBasicLocatorString to typeBasicID.
- Drop quake/relationships.lisp. Use a more straightforward way to
associated ADO agencies to customer profiles.
- Customer usernames are now stored as lowercase strings (this means
that I don't need a separate field in megastore protobuf to
implement case-insensitive index).
- agency-id permission attribute is now the agency-key (serialized to
string), not the agency's record ID. This way you don't need to load
agency to do permission check when all you have is the agency-key
- moved agency-key from core/agencies.lisp to
lisp/core/agency-types.lisp, so that agency-key type can be used in
permissions
- add when* test handler that takes a random Lisp expression (as
opposed to the existing when that takes a variable name and looks at
whether it is empty)
- remove cc-deleted-error -- there is no such thing as deleting
cc-info
- refactored cc-information, so that it is used as a value object (ie
a mostly-immutable object with no identity) in the application. When
saving to Oracle database, it is saved in the central
tbl_cc_information table with appropriately populated ownership
columns (pointing to the customer, journey,
accountable_doc_container, etc)
- drop unneeded tbl_cc_transaction columns that were caching cc_info
values
Scott McKay [Wed, 12 Sep 2012 19:09:46 +0000 (19:09 +0000)]
Fix some user-reported bugs
Testing : precheckin --full --strict-errors
Reviewer: Sergey V, Shaun M
JTB impact: No
Ops impact: No
Change to config : No
Change to XML schema : No
Change to DB schema : No
Change to transport (timeouts, headers) : No
Any change (or new use) of OAQs : No
Change to inter-component transactions : No
Depends on any other checkin / bug : No
Tests that will verify:
The CL-Protobufs tests
Description:
I had made a "policy decision" that the package created
by importing a .proto file would (:use :common-lisp). In
practice, this turned out to be a mistake. Sergey suggested
a fix that I think is correct, so in it goes.
Shaun noticed that the optimized 'serialize-object' and
'object-size' methods didn't quite implement what they were
supposed to for optional boolean fields whose value was never
supplied. Fix the optimized methods to implement the same
(correct) semantics as the unoptimized ones.
Scott McKay [Mon, 10 Sep 2012 17:38:33 +0000 (17:38 +0000)]
Add a more readable syntax for enum and field indices
Testing : precheckin --full --strict-errors
Reviewer: Fare (please)
JTB impact: No
Ops impact: No
Change to config : No
Change to XML schema : No
Change to DB schema : No
Change to transport (timeouts, headers) : No
Any change (or new use) of OAQs : No
Change to inter-component transactions : No
Depends on any other checkin / bug : No
Tests that will verify:
I extended the CL-Protobufs tests
Description:
Add a more readable syntax for enum and field indices.
For define-enum, it was (name value).
Allow (name :index value).
For define-message, it was ((name index) ...).
Allow (name :index index ...).
Complain if both forms appear in the same field.
Scott McKay [Fri, 7 Sep 2012 15:22:28 +0000 (15:22 +0000)]
Add a few missing features that aren't in any documentation.
Testing : precheckin --full --strict-errors
Reviewer: Fare (please)
JTB impact: No
Ops impact: No
Change to config : No
Change to XML schema : No
Change to DB schema : No
Change to transport (timeouts, headers) : No
Any change (or new use) of OAQs : No
Change to inter-component transactions : No
Depends on any other checkin / bug : No
Tests that will verify:
I extended the CL-Protobufs examples
Description:
Add a few missing features that aren't in any documentation.
String literals can look like "foo"<whitespace>"bar".
- Fix 'parse-string' to handle this.
Option values can be complex structures, not just atoms.
- If 'parse-proto-option' sees a '{' character after the '=',
it should use 'parse-text-format' to read the option value.
- Fix the 'protobuf-option' printer to call 'print-text-format'
for complex option values.
- Minor refactoring to the text format parser and printer in
order to support the above.
Scott McKay [Tue, 4 Sep 2012 18:36:57 +0000 (18:36 +0000)]
Add stub support for streaming RPC APIs.
Testing : precheckin --full --strict-errors
Reviewer: Fare (please)
JTB impact: No
Ops impact: No
Change to config : No
Change to XML schema : No
Change to DB schema : No
Change to transport (timeouts, headers) : No
Any change (or new use) of OAQs : No
Change to inter-component transactions : No
Depends on any other checkin / bug : No
Tests that will verify:
I extended the CL-Protobufs tests
Description:
Add support for a 'streams' type in Protobufs RPC methods.
- Add a few slots to 'protobufs-method'
- Add '&key streams' to the method arglists in 'define-service'
- Extend the parser to handle "streams" in addition to the
'streams_type' option
- Extend the printer
- Change a test to use a 'streams' type
- While we're in the neighborhood, allow an optional '=>'
between the input and output methods in 'define-service';
it makes it a bit more readable
- Update the documentation
Scott McKay [Fri, 31 Aug 2012 15:15:49 +0000 (15:15 +0000)]
Add type aliases to CL-Protobufs
Testing : precheckin --full --strict-errors
Reviewer: Fare (please)
JTB impact: No
Ops impact: No
Change to config : No
Change to XML schema : No
Change to DB schema : No
Change to transport (timeouts, headers) : No
Any change (or new use) of OAQs : No
Change to inter-component transactions : No
Depends on any other checkin / bug : No
Tests that will verify:
The CL-Protobufs tests
Description:
Add type aliases to CL-Protobufs. It's a Lisp-only
feature that allows you define a new Protobufs type in
terms of a Lisp type, a serializer and a deserializer.
- Add a new model class, 'protobuf-type-alias'.
- Extend schemas and messages so that they can hold
a set of type aliases; add 'find-type-alias'.
- Add 'define-type-alias'.
- Factor out 'lisp-type-to-protobuf-type' from
'clos-type-to-protobuf-type', I need it for the
'define-type-alias' macro.
- Fix the schema printer to print 'define-type-alias'
for .lisp schemas, and to add a comment describing
the type alias in .proto schemas.
- Extend the wire format (de)serializer to know
about type aliases; it just (de)serializes the
Lisp type as the Protobufs primitive type using
the provided (de)serializer functions. There's
zero cost to this if you don't use aliases.
- Update the optimized serialization generation.
- Similarly extend the text format (de)serializer
to know about type aliases.
- Change an example to use a type alias for 'date'.
- Update the documentation.
Scott McKay [Thu, 28 Jun 2012 20:42:55 +0000 (20:42 +0000)]
It turns out that being able to get from method -> service -> schema
makes my CL-Stubby client code simpler. So, fill in the 'parent'
slot for more than just messages. No cost at runtime, so WTH.
Passes 'precheckin' with the new Protobufs unit tests in place.
CL-Stubby tests pass in Google3 environment.
Scott McKay [Wed, 20 Jun 2012 17:39:41 +0000 (17:39 +0000)]
Random things discovered while working on Stubby support...
- Make the 'find-xxx' support searching "relative to" another namespace.
- There was a bug in the non-optimized deserializer when deserializing
a repeated slot into a vector; create a stretchy vector on demand
if it's needed.
- 'define-extends' should wrap 'eval-when' around the generated 'defsetf'
forms so that they are visible at compile time.
- Fix a formatting bug in the export list in the Lisp printer.
- Straighten of the ASDF declaration for the tests.
- Add the Protobufs test suite to QRes, to keep things honest.
Passes 'precheckin' with the new Protobufs unit tests in place.
Scott McKay [Mon, 18 Jun 2012 15:00:19 +0000 (15:00 +0000)]
Still better support for qualified names:
- Add 'make-qualified-name' function that knows how to prepend
message and package names.
- All the places that create messages, enums, fields, etc, now
generate a qualified name.
- 'find-qualified-name' now thoroughly checks the qualified name.
Passes 'precheckin' with the new Protobufs unit tests in place.
Scott McKay [Mon, 11 Jun 2012 17:13:55 +0000 (17:13 +0000)]
While implementing the Fortuneteller stubby server, I ran into
some things that needed to be addressed:
- We need to do a more thorough search of namespaces so that
qualified references to imported messages, etc, work properly.
It's still not 100%, but it covers most of what I can find.
- Name mappers now do their best to preserve qualified names.
- 'find-xxx' methods search the qualified names.
- Fix the importation process to cooperate better with ASDF, i.e.,
don't bother re-importing things that have already compiled
as part of an ASDF compile/load operation.
- When generating .lisp from .proto, if no package has been
declared, do it. And fill it with nice exports, too. This
won't be exactly right, but it'll make it easier to create
a proper package declaration, if you so desire.
- Do a better job of generating client- and server-side stubs.
Passes 'precheckin' with the new Protobufs unit tests in place.
And more to the point, the Fortuneseeker server now works as a
Stubby client using my new Google3 http2 and rpc2 libraries.
Scott McKay [Tue, 5 Jun 2012 17:46:46 +0000 (17:46 +0000)]
SBCL kindly offered suggestions for improvement:
- Use 'defgeneric' with ':method' for things that are meant to
have just a single method.
- Lose some unreachable code in the parser.
- Weak hash tables in SBCL need to use ':weakness :value'.
- The wire-level stuff can use some more type declarations.
- Fix a package problem in one of the examples.
Passes 'precheckin' with the new Protobufs unit tests in place.
Scott McKay [Wed, 30 May 2012 20:58:28 +0000 (20:58 +0000)]
More support for Stubby and use in Google3 in general:
- More sophisticated and easier to use ASDF support.
- Add search paths to the :protobuf-file ASDF module
Passes 'precheckin' with the new Protobufs unit tests in place.
Scott McKay [Fri, 25 May 2012 20:24:29 +0000 (20:24 +0000)]
Get some stuff working better for Stubby purposes.
- Better implementation of 'uncamel-case' gives better names.
- Add an index slot to 'protobuf-method' so that we can more
easily do automatic service registration.
- Give a better type to the index slot in 'protobuf-message'
and 'protobuf-enum'.
- Parser and macros now assign an index to each service method.
- Define 'find-service' and 'find-method' for use by Stubby.
Passes 'precheckin' with the new Protobufs unit tests in place.
Scott McKay [Fri, 25 May 2012 14:46:50 +0000 (14:46 +0000)]
Fix a few more things discovered by the tests:
- 'reinitialize-slot' didn't quite work.
- Add a geodata example that uses vectors for repeated fields,
which uncovered a bug in the optimized deserializers.
- Importing the geodata CLOS classes revealed a bug in default
handling when the default is provided only in 'defclass'.
- Fix the knock-on bug in deserialization and the optimized
(de)serialization caused the above.
- Add tighter types in 'decode-uint32' and 'decode-uint64'.
Passes 'precheckin'. Even with the new unit tests in place.
Scott McKay [Wed, 23 May 2012 18:17:11 +0000 (18:17 +0000)]
Add 'vector-of' type, at Shaun Morris's request. This required
some additions to the macrology and to the (de)serialization
code. No runtime performance cost to being able to have things
both ways.
Fix a bug that caused unprocessed options to get dropped.
Improve some of the unit tests.
Passes 'precheckin'. Even with the new unit tests in place.
Scott McKay [Tue, 22 May 2012 20:16:41 +0000 (20:16 +0000)]
Add more stuff to the test suite, and fix what it uncovered.
- Behavior of ':default xxx'/[default=xxx] was not stable,
so add an "empty default" marker and respect it during
parsing and printing.
- 'find-option' now returns a third "found-p" value, useful
for the better handling of defaults.
- Don't print slot readers/writers if they have the same name
as what would be (silently) generated by the macrology.
- Don't print 'import descriptor.proto' lines if there are
no 'lisp_xxx' options in the generated code.
- Add 'clear-field' to the API; add the functions it needs.
- Be more careful about how :conc-name interacts with the
generation of readers/writers.
- When parsing from a .proto file, force the conc-name to
be "" unless requested explicitly not to. This has the
effect of generating readers/writers for all the fields
of imported Protobufs messages, which I think is probably
the desired behavior.
- Add 'make-byte-vector' for convenience.
Passes 'precheckin'. Even with the new unit tests in place.
Scott McKay [Fri, 18 May 2012 21:04:02 +0000 (21:04 +0000)]
Now that Protobufs has a test suite, it found a few things to fix.
- Don't generate warnings for anonymous enums, they're harmless
and ubiquitous.
- 'member' types where are all the members are keywords, is a symbol,
which got converted to a Protobufs 'string'. Wrong.
- If a field has an enum type, the (Lisp) default value should be
treated as a keyword.
- When parsing a .proto file or using the 'define-xxx' macros, any
options that are handled specially should be trimmed from the
options list so that they don't get printed twice.
- Add 'remove-options' for the above.
- Make the options printer be more type-aware.
- Clean up examples.lisp
- 'schemas-equal' can ignore the schema name/class if they're null.
Passes 'precheckin'. Even with the new unit tests in place.
Update the Quake Protobufs schema generator to use the new names.
The Protobufs documentation doesn't say so, but enums can be packed.
- Update the wire-level protocol to (de)serialize packed enums.
- Update the serialization code to use the new wire functions.
Scott McKay [Wed, 16 May 2012 17:01:06 +0000 (17:01 +0000)]
Tie up a few loose ends with 'unittests.proto'; there are still a few
edge case problems, but it's good enough to continue the Stubby work.
The critical .proto files (descriptor.proto, etc) all appear to work.
More precise mapping of (mainly integer) types from .lisp to .proto files
so that we can a truly exact model once we compile the .lisp file. It may
not make any difference, but better safe than sorry.
- Introduce Lisp type 'int32', 'int64', 'uint32', etc
- Make Lisp->Protobufs recognize the new Lisp types
- Make Protobufs schema printers recognize the new types
Fix up the handling of imports and namespaces.
- Always use truenames to record schemas
- Fix typo in 'find-enum' that caused it never to search other schemas
- Record the schema in *all-protobufs* at exactly the right time,
and simplify 'make-load-form' methods
A few tweaks to (de)serialization.
- Make the bodies for empty messages not generate compiler warnings
- Don't make ':packed t' be the default, the user has to say so
- The user documentation doesn't say so, but booleans can be packed.
Scott McKay [Mon, 14 May 2012 23:22:59 +0000 (23:22 +0000)]
A few more model fixes to fully support google/protobuf/unittest.proto
(I love this file, it's got one of everything.)
- The printer should not print extended fields in any message
that further extends an extended message.
- If parsing or importing a .proto file needs a Lisp package that
does not exist, go ahead and create it.
- Importing now tracks all the schemas imported by the "parent",
this so that we can search namespaces thoroughly.
- It looks like 'define-extend' can accept groups. For a deprecated
feature, it sure is a pain in the neck.
- Make 'find-message' and 'find-enum' search all namespaces.
- Print escaped strings, just for completeness.
- Annotate a couple of common options with their types.
Passes 'precheckin'. Passes my by-hand tests and is epsilon away from
handling the worst .proto file around.
Scott McKay [Sun, 13 May 2012 15:17:06 +0000 (15:17 +0000)]
Get google/protobuf/unittest.proto working, insofar as it parses
completely and produces what appears to be a correct model. Next
up, ensure (de)serialization works with this complex beast, which
sets the stage for running the (very complete) Google tests.
- Be much more careful to preserve type information in option
values and field defaults, both in the macros and the parser.
Make the slots in their model objects be untyped.
- This makes it easier for the Proto and Lisp printers to produce
exactly the right output, so fix them, too. Be extra careful
with booleans.
- Fix the printer to recognize more of the standard options so
that their values get printed with the correct type.
- Learn to parse more complicated numeric and integer constants.
- Learn to parse escaped strings.
- Learn to parse comma-separated options.
- Learn to parse extensions with no 'to' specifier.
- Fix the places that used (deprecated) :single instead of :float.
- Proto enum values always map to Lisp keywords, so don't futz
with packages for them.
- Add some more doc strings.
- Add some more examples-that-will-become-tests.
Scott McKay [Fri, 11 May 2012 20:17:48 +0000 (20:17 +0000)]
Now that I've started importing the Google tests for Protos,
a few more bugs are showing up:
- Add basic handling for namespaces
- A typo in my own tests caused me to miss the fact that
negative signed integers did not deserialize correctly
(thanks, Shaun!)
- Just dike out the rest of the Quux dependencies, since it's
pretty clear non-QRes folks will be using it
Scott McKay [Thu, 10 May 2012 21:14:58 +0000 (21:14 +0000)]
Fix some problems reported by Shaun Morris:
- Looks like Stubby will need 'merge-from-array', make room for that.
- Text format deserializer didn't correctly parse negative integers.
- Simply 'deserialize-object' a bit
- Fix a bug in the 'deserialize-object' that caused it not to work
correctly on extended fields.
- Fix a sloppy bug in 'make-tag' that only showed up in SBCL.
- Add a few more examples, soon to be part of a test suite.
projects / cl-protobufs.git / log