Don't assume that returning no options meant there was no body in the
method declaration. An empty body may have been there
instead. Distinguish that scenario with a second return value from
PARSE-PROTO-METHOD-OPTIONS and use that value in PARSE-PROTO-METHOD
determine whether or not to look for a semicolon.
printer.lisp: consume more arguments when appropriate
Here we should have been consuming both the thing we would be printing
and the indentation we would use on the next line, but in the case
where the printed thing was nil, we would only consume it, and not the
subsequent indentation. It worked out here, but the mistake could only
be made once, and only at the end.
Robert Brown's protobuf package falls back on the java_package option
as the lisp package name when there isn't a package declaration in the
schema. Emulate that behavior for better compatibility between code
generated by the packages.
Generate integer encoders and decoders with thought towards the size of a FIXNUM
Rather than hard-code assumptions about the size of a FIXNUM, check
the size of FIXNUM at compile time and generate code appropriate for
the number of bits we're working with, with FIXNUM optimizations
in place if appropriate.
"A common pattern is to define extensions inside the scope of the
extension's field type – for example, here's an extension to Foo of
type Baz, where the extension is defined as part of Baz"
Add a proper input-files method for proto-to-lisp,
which shall fix the issue that causes of these actions never being operation-done-p.
However this does NOT handle recursive import dependencies,
which requires further work.
Tested: QRes doesn't always recompile the proto and all dependees anymore.
Reviewer: asedeno
Ben Wagner [Tue, 20 Nov 2012 18:43:00 +0000 (13:43 -0500)]
fix cross-package and forward references in cl-protobufs
* Previously, if a field in a .proto file referenced a message in
another proto file using a different lisp package, the cl-protobufs
library would silently fail to serialize the field. A similar
problem would occur if a message defined later in the file used the
lisp_name option to override the name generated by cl-protobufs.
This change fixes these issues and others.
* Add conditions that are signaled when encountering an undefined
type.
* Delay assigning lisp classes/types to fields and methods until all
possible forward references have been parsed.
* This allows the class slot to be unbound, so check for that case
in print-object methods.
* Add a test for forward references to messages that override the
lisp name.
* Add a test for references to messages and enums defined in
another proto file with a different lisp package.
* Change color-wheel-stability test, because it used "string" as
the input type for an rpc, which seems to be disallowed (although
I haven't found this documented anywhere).
* Signal errors during parsing for undefined types.
* Add a test for these errors. Add assert-error macro to qtest.
* Signal a condition if we are unable to find the definition for a
field's type during serialization, deserialization, determining an
object's serialized size, printing text format, parsing text format,
or generating code for one of the above.
* Remove logic in find-qualified-name that indirects through lisp
packages. Proto packages and lisp packages do not necessarily map
1-to-1.
* Always use the schema's lisp package for any symbols generated when
parsing proto files.
* When generating lisp code using write-schema-as, set the package to
the package used in the generated file, so that ~s will print the
package prefix in the correct circumstances.
* Remove broken proto1 "streams" parsing ("returns" comes before
"streams" in every example I've found); replace with proto2 syntax.
* In process-imports, the call to find-schema using a pathname was not
giving the expected result. Sidestep this issue by using the same
logic to find the schema as is used earlier in the function.
Improve schema generation performance on large schemas
Avoid SUBTYPEP during macroexpansion, it can be very expensive on SBCL.
Actually recognize LIST-OF:LIST-OF constructs.
In DEFINE-SCHEMA, don't nest the DEFMETHODs inside the LET,
this also can cause SBCL's control flow analysis to blow up.
Update a few basic Lisp libraries.
Testing: [trunk r567444] precheckin --parallel 11 PASSED
Reviewers: dcrawford, asedeno
Rationale: there is a problem in quicklisp due to recent asdf package changes,
that requires urgent fix to cl-protobufs,
so I'm updating these for the sake of issuing
a cl-protobufs fix to quicklisp ASAP.
I require these updates for some further changes I'm making to QUUX, anyway,
so it's just a matter of timing.
Scott McKay [Tue, 18 Sep 2012 16:04:09 +0000 (16:04 +0000)]
Speed up unoptimized serialization
Testing : precheckin --full --strict-errors
Reviewer: Fare
JTB impact: No
Ops impact: No
Change to config : No
Change to XML schema : No
Change to DB schema : No
Change to transport (timeouts, headers) : No
Any change (or new use) of OAQs : No
Change to inter-component transactions : No
Depends on any other checkin / bug : No
Tests that will verify:
The CL-Protobufs tests
Description:
The unoptimized serialization code was slower than
it needed to be and was a Niagara falls of consing.
The problem is that (map () (curry ...)) is poorly
optimized, which is a shame because it's such a nice
coding style. :-P
The fix is to replace 'map' with iteration (sigh) in
a few critical places.
- Add 'doseq', which chooses between 'dolist' or 'dovector'.
- Make 'serialize-object', 'deserialize-object' and 'object-size'
use 'doseq' instead of using 'map'.
- Ditto for 'print-text-format'.
- Fix the 'serialize-packed' and 'packed-size' optimizers to
use 'dolist' or 'dovector' based on whether the field is
a vector field. This makes the optimized code faster, too.
- Make the optimized serializer generators pass in 'vectorp'
so that the wire-format optimizers can do a better job.
Tsunami framework + use tsunami in customer profiles
Testing : precheckin --full (both against Oracle and against megastore for customers)
Reviewer: eschwartz
JTB impact: n
Ops impact: n
Security impact: n
Change to config : n
Change to XML schema : Y (XMLREQ bug 120820)
Change to DB schema : Y (DBREQ bug 119801)
Change to transport (timeouts, headers) : n
Any change (or new use) of OAQs : n
Change to inter-component transactions : sort of (use megastore in development-only environments)
Depends on any other checkin / bug : n
Tests that will verify : tsunami and megastore unit tests + existing customer profile tests
Tsunami is a new layer that sits between business logic code and the
persistence logic. The main manifestation of this layer is
define-domain-class macro that defines an interface to a domain
class. You can then define how to perist in various implementations,
e.g. using define-quake-db-class-bindings for quake and
define-proto-binding & define-megastore-proto-binding for megastore.
Tsunami framework is still a work in progress, and there is a bunch of
outstanding tasks/issues.
As the first application, I refactored customer profile code to use
the tsunami framework. This enables storing of customer profiles in
megastore (or other data stores in the future), while everything else
is stored in Oracle.
Here is the list of other miscellaneous/harder-to-understand changes:
- I changed @id attribute of ContactInfo objects to use the actual
numeric ID, rather than passing it through encode-record-locator (in
qres-dev schema only). This required changing XML schema type from
typeBasicLocatorString to typeBasicID.
- Drop quake/relationships.lisp. Use a more straightforward way to
associated ADO agencies to customer profiles.
- Customer usernames are now stored as lowercase strings (this means
that I don't need a separate field in megastore protobuf to
implement case-insensitive index).
- agency-id permission attribute is now the agency-key (serialized to
string), not the agency's record ID. This way you don't need to load
agency to do permission check when all you have is the agency-key
- moved agency-key from core/agencies.lisp to
lisp/core/agency-types.lisp, so that agency-key type can be used in
permissions
- add when* test handler that takes a random Lisp expression (as
opposed to the existing when that takes a variable name and looks at
whether it is empty)
- remove cc-deleted-error -- there is no such thing as deleting
cc-info
- refactored cc-information, so that it is used as a value object (ie
a mostly-immutable object with no identity) in the application. When
saving to Oracle database, it is saved in the central
tbl_cc_information table with appropriately populated ownership
columns (pointing to the customer, journey,
accountable_doc_container, etc)
- drop unneeded tbl_cc_transaction columns that were caching cc_info
values
Scott McKay [Wed, 12 Sep 2012 19:09:46 +0000 (19:09 +0000)]
Fix some user-reported bugs
Testing : precheckin --full --strict-errors
Reviewer: Sergey V, Shaun M
JTB impact: No
Ops impact: No
Change to config : No
Change to XML schema : No
Change to DB schema : No
Change to transport (timeouts, headers) : No
Any change (or new use) of OAQs : No
Change to inter-component transactions : No
Depends on any other checkin / bug : No
Tests that will verify:
The CL-Protobufs tests
Description:
I had made a "policy decision" that the package created
by importing a .proto file would (:use :common-lisp). In
practice, this turned out to be a mistake. Sergey suggested
a fix that I think is correct, so in it goes.
Shaun noticed that the optimized 'serialize-object' and
'object-size' methods didn't quite implement what they were
supposed to for optional boolean fields whose value was never
supplied. Fix the optimized methods to implement the same
(correct) semantics as the unoptimized ones.
Scott McKay [Mon, 10 Sep 2012 17:38:33 +0000 (17:38 +0000)]
Add a more readable syntax for enum and field indices
Testing : precheckin --full --strict-errors
Reviewer: Fare (please)
JTB impact: No
Ops impact: No
Change to config : No
Change to XML schema : No
Change to DB schema : No
Change to transport (timeouts, headers) : No
Any change (or new use) of OAQs : No
Change to inter-component transactions : No
Depends on any other checkin / bug : No
Tests that will verify:
I extended the CL-Protobufs tests
Description:
Add a more readable syntax for enum and field indices.
For define-enum, it was (name value).
Allow (name :index value).
For define-message, it was ((name index) ...).
Allow (name :index index ...).
Complain if both forms appear in the same field.
Scott McKay [Fri, 7 Sep 2012 15:22:28 +0000 (15:22 +0000)]
Add a few missing features that aren't in any documentation.
Testing : precheckin --full --strict-errors
Reviewer: Fare (please)
JTB impact: No
Ops impact: No
Change to config : No
Change to XML schema : No
Change to DB schema : No
Change to transport (timeouts, headers) : No
Any change (or new use) of OAQs : No
Change to inter-component transactions : No
Depends on any other checkin / bug : No
Tests that will verify:
I extended the CL-Protobufs examples
Description:
Add a few missing features that aren't in any documentation.
String literals can look like "foo"<whitespace>"bar".
- Fix 'parse-string' to handle this.
Option values can be complex structures, not just atoms.
- If 'parse-proto-option' sees a '{' character after the '=',
it should use 'parse-text-format' to read the option value.
- Fix the 'protobuf-option' printer to call 'print-text-format'
for complex option values.
- Minor refactoring to the text format parser and printer in
order to support the above.
Scott McKay [Tue, 4 Sep 2012 18:36:57 +0000 (18:36 +0000)]
Add stub support for streaming RPC APIs.
Testing : precheckin --full --strict-errors
Reviewer: Fare (please)
JTB impact: No
Ops impact: No
Change to config : No
Change to XML schema : No
Change to DB schema : No
Change to transport (timeouts, headers) : No
Any change (or new use) of OAQs : No
Change to inter-component transactions : No
Depends on any other checkin / bug : No
Tests that will verify:
I extended the CL-Protobufs tests
Description:
Add support for a 'streams' type in Protobufs RPC methods.
- Add a few slots to 'protobufs-method'
- Add '&key streams' to the method arglists in 'define-service'
- Extend the parser to handle "streams" in addition to the
'streams_type' option
- Extend the printer
- Change a test to use a 'streams' type
- While we're in the neighborhood, allow an optional '=>'
between the input and output methods in 'define-service';
it makes it a bit more readable
- Update the documentation
Scott McKay [Fri, 31 Aug 2012 15:15:49 +0000 (15:15 +0000)]
Add type aliases to CL-Protobufs
Testing : precheckin --full --strict-errors
Reviewer: Fare (please)
JTB impact: No
Ops impact: No
Change to config : No
Change to XML schema : No
Change to DB schema : No
Change to transport (timeouts, headers) : No
Any change (or new use) of OAQs : No
Change to inter-component transactions : No
Depends on any other checkin / bug : No
Tests that will verify:
The CL-Protobufs tests
Description:
Add type aliases to CL-Protobufs. It's a Lisp-only
feature that allows you define a new Protobufs type in
terms of a Lisp type, a serializer and a deserializer.
- Add a new model class, 'protobuf-type-alias'.
- Extend schemas and messages so that they can hold
a set of type aliases; add 'find-type-alias'.
- Add 'define-type-alias'.
- Factor out 'lisp-type-to-protobuf-type' from
'clos-type-to-protobuf-type', I need it for the
'define-type-alias' macro.
- Fix the schema printer to print 'define-type-alias'
for .lisp schemas, and to add a comment describing
the type alias in .proto schemas.
- Extend the wire format (de)serializer to know
about type aliases; it just (de)serializes the
Lisp type as the Protobufs primitive type using
the provided (de)serializer functions. There's
zero cost to this if you don't use aliases.
- Update the optimized serialization generation.
- Similarly extend the text format (de)serializer
to know about type aliases.
- Change an example to use a type alias for 'date'.
- Update the documentation.
Scott McKay [Thu, 28 Jun 2012 20:42:55 +0000 (20:42 +0000)]
It turns out that being able to get from method -> service -> schema
makes my CL-Stubby client code simpler. So, fill in the 'parent'
slot for more than just messages. No cost at runtime, so WTH.
Passes 'precheckin' with the new Protobufs unit tests in place.
CL-Stubby tests pass in Google3 environment.
Scott McKay [Wed, 20 Jun 2012 17:39:41 +0000 (17:39 +0000)]
Random things discovered while working on Stubby support...
- Make the 'find-xxx' support searching "relative to" another namespace.
- There was a bug in the non-optimized deserializer when deserializing
a repeated slot into a vector; create a stretchy vector on demand
if it's needed.
- 'define-extends' should wrap 'eval-when' around the generated 'defsetf'
forms so that they are visible at compile time.
- Fix a formatting bug in the export list in the Lisp printer.
- Straighten of the ASDF declaration for the tests.
- Add the Protobufs test suite to QRes, to keep things honest.
Passes 'precheckin' with the new Protobufs unit tests in place.
Scott McKay [Mon, 18 Jun 2012 15:00:19 +0000 (15:00 +0000)]
Still better support for qualified names:
- Add 'make-qualified-name' function that knows how to prepend
message and package names.
- All the places that create messages, enums, fields, etc, now
generate a qualified name.
- 'find-qualified-name' now thoroughly checks the qualified name.
Passes 'precheckin' with the new Protobufs unit tests in place.
Scott McKay [Mon, 11 Jun 2012 17:13:55 +0000 (17:13 +0000)]
While implementing the Fortuneteller stubby server, I ran into
some things that needed to be addressed:
- We need to do a more thorough search of namespaces so that
qualified references to imported messages, etc, work properly.
It's still not 100%, but it covers most of what I can find.
- Name mappers now do their best to preserve qualified names.
- 'find-xxx' methods search the qualified names.
- Fix the importation process to cooperate better with ASDF, i.e.,
don't bother re-importing things that have already compiled
as part of an ASDF compile/load operation.
- When generating .lisp from .proto, if no package has been
declared, do it. And fill it with nice exports, too. This
won't be exactly right, but it'll make it easier to create
a proper package declaration, if you so desire.
- Do a better job of generating client- and server-side stubs.
Passes 'precheckin' with the new Protobufs unit tests in place.
And more to the point, the Fortuneseeker server now works as a
Stubby client using my new Google3 http2 and rpc2 libraries.
Scott McKay [Tue, 5 Jun 2012 17:46:46 +0000 (17:46 +0000)]
SBCL kindly offered suggestions for improvement:
- Use 'defgeneric' with ':method' for things that are meant to
have just a single method.
- Lose some unreachable code in the parser.
- Weak hash tables in SBCL need to use ':weakness :value'.
- The wire-level stuff can use some more type declarations.
- Fix a package problem in one of the examples.
Passes 'precheckin' with the new Protobufs unit tests in place.
Scott McKay [Wed, 30 May 2012 20:58:28 +0000 (20:58 +0000)]
More support for Stubby and use in Google3 in general:
- More sophisticated and easier to use ASDF support.
- Add search paths to the :protobuf-file ASDF module
Passes 'precheckin' with the new Protobufs unit tests in place.
Scott McKay [Fri, 25 May 2012 20:24:29 +0000 (20:24 +0000)]
Get some stuff working better for Stubby purposes.
- Better implementation of 'uncamel-case' gives better names.
- Add an index slot to 'protobuf-method' so that we can more
easily do automatic service registration.
- Give a better type to the index slot in 'protobuf-message'
and 'protobuf-enum'.
- Parser and macros now assign an index to each service method.
- Define 'find-service' and 'find-method' for use by Stubby.
Passes 'precheckin' with the new Protobufs unit tests in place.
Scott McKay [Fri, 25 May 2012 14:46:50 +0000 (14:46 +0000)]
Fix a few more things discovered by the tests:
- 'reinitialize-slot' didn't quite work.
- Add a geodata example that uses vectors for repeated fields,
which uncovered a bug in the optimized deserializers.
- Importing the geodata CLOS classes revealed a bug in default
handling when the default is provided only in 'defclass'.
- Fix the knock-on bug in deserialization and the optimized
(de)serialization caused the above.
- Add tighter types in 'decode-uint32' and 'decode-uint64'.
Passes 'precheckin'. Even with the new unit tests in place.
Scott McKay [Wed, 23 May 2012 18:17:11 +0000 (18:17 +0000)]
Add 'vector-of' type, at Shaun Morris's request. This required
some additions to the macrology and to the (de)serialization
code. No runtime performance cost to being able to have things
both ways.
Fix a bug that caused unprocessed options to get dropped.
Improve some of the unit tests.
Passes 'precheckin'. Even with the new unit tests in place.
projects / cl-protobufs.git / log