Don't override file extension for imports; assume the string that was
used to form the pathname is accurate. This makes base-path and
proto-file the same, so fold them both into proto-file.
Change how protobuf schemas and classes are recorded and found
* FIND-SCHEMA no longer works on a string or keyword. Both of these
were based on the PATHNAME-NAME of the protobuf, which has a high
probability of collision. (e.g., common.proto -> "COMMON", :COMMON)
Instead, look up schemas based on package and name. For instance,
the protobuf foo.proto, declaring "package FooPackage;", can be
found with: (FIND-SCHEMA 'FOO-PACKAGE::FOO)
* Pathname-based schema lookup no longer ignores file type.
* DO-PROCESS-IMPORT is now responsible for deciding whether or not to
process an import. It does so by searching for the .proto to be
imported, and upon finding it, checks to see if we alredy have a
schema for that pathname. Additionally, it now returns the pathname.
* PROCESS-IMPORTS and PROCESS-IMPORTS-FROM-FILE now defer the work of
figuring out whether or not an import has been done to
DO-PROCESS-IMPORT. PROCESS-IMPORTS uses the return value of
DO-PROCESS-IMPORT to update the PROTO-IMPORTED-SCHEMAS of the schema
being processed.
* ASDF::PROTOBUF-MANGLE-NAME: Takes a pathname, returns a string to be
used as a filename that represents the original path.
e.g., #P"/foo/bar/baz.quux" becomes "foo-bar-baz-quux"
* ASDF::PROTOBUF-LISPIZE-PATHNAME: Takes a pathname, returns a new
pathname with the name mangled as described above, and with a lisp
type.
Scott McKay [Sun, 3 Mar 2013 09:52:37 +0000 (15:22 +0530)]
In the spirit of making CL-Protobufs represent exemplary modern Common Lisp code:
- Use 'defparameter' instead of 'defvar' where appropriate.
- Fix 'defvar' doc strings to distinguish between globals and "thread locals".
- Avoid using 'nconc'. introduce a new 'appendf' macro instead.
- Add a comment lamenting the fact that exporting something like 'proto-options'
also exports the writer '(setf proto-options)'. Fixed in Dylan.
Don't assume that returning no options meant there was no body in the
method declaration. An empty body may have been there
instead. Distinguish that scenario with a second return value from
PARSE-PROTO-METHOD-OPTIONS and use that value in PARSE-PROTO-METHOD
determine whether or not to look for a semicolon.
printer.lisp: consume more arguments when appropriate
Here we should have been consuming both the thing we would be printing
and the indentation we would use on the next line, but in the case
where the printed thing was nil, we would only consume it, and not the
subsequent indentation. It worked out here, but the mistake could only
be made once, and only at the end.
Robert Brown's protobuf package falls back on the java_package option
as the lisp package name when there isn't a package declaration in the
schema. Emulate that behavior for better compatibility between code
generated by the packages.
Generate integer encoders and decoders with thought towards the size of a FIXNUM
Rather than hard-code assumptions about the size of a FIXNUM, check
the size of FIXNUM at compile time and generate code appropriate for
the number of bits we're working with, with FIXNUM optimizations
in place if appropriate.
"A common pattern is to define extensions inside the scope of the
extension's field type – for example, here's an extension to Foo of
type Baz, where the extension is defined as part of Baz"
Add a proper input-files method for proto-to-lisp,
which shall fix the issue that causes of these actions never being operation-done-p.
However this does NOT handle recursive import dependencies,
which requires further work.
Tested: QRes doesn't always recompile the proto and all dependees anymore.
Reviewer: asedeno
Ben Wagner [Tue, 20 Nov 2012 18:43:00 +0000 (13:43 -0500)]
fix cross-package and forward references in cl-protobufs
* Previously, if a field in a .proto file referenced a message in
another proto file using a different lisp package, the cl-protobufs
library would silently fail to serialize the field. A similar
problem would occur if a message defined later in the file used the
lisp_name option to override the name generated by cl-protobufs.
This change fixes these issues and others.
* Add conditions that are signaled when encountering an undefined
type.
* Delay assigning lisp classes/types to fields and methods until all
possible forward references have been parsed.
* This allows the class slot to be unbound, so check for that case
in print-object methods.
* Add a test for forward references to messages that override the
lisp name.
* Add a test for references to messages and enums defined in
another proto file with a different lisp package.
* Change color-wheel-stability test, because it used "string" as
the input type for an rpc, which seems to be disallowed (although
I haven't found this documented anywhere).
* Signal errors during parsing for undefined types.
* Add a test for these errors. Add assert-error macro to qtest.
* Signal a condition if we are unable to find the definition for a
field's type during serialization, deserialization, determining an
object's serialized size, printing text format, parsing text format,
or generating code for one of the above.
* Remove logic in find-qualified-name that indirects through lisp
packages. Proto packages and lisp packages do not necessarily map
1-to-1.
* Always use the schema's lisp package for any symbols generated when
parsing proto files.
* When generating lisp code using write-schema-as, set the package to
the package used in the generated file, so that ~s will print the
package prefix in the correct circumstances.
* Remove broken proto1 "streams" parsing ("returns" comes before
"streams" in every example I've found); replace with proto2 syntax.
* In process-imports, the call to find-schema using a pathname was not
giving the expected result. Sidestep this issue by using the same
logic to find the schema as is used earlier in the function.
Improve schema generation performance on large schemas
Avoid SUBTYPEP during macroexpansion, it can be very expensive on SBCL.
Actually recognize LIST-OF:LIST-OF constructs.
In DEFINE-SCHEMA, don't nest the DEFMETHODs inside the LET,
this also can cause SBCL's control flow analysis to blow up.
Update a few basic Lisp libraries.
Testing: [trunk r567444] precheckin --parallel 11 PASSED
Reviewers: dcrawford, asedeno
Rationale: there is a problem in quicklisp due to recent asdf package changes,
that requires urgent fix to cl-protobufs,
so I'm updating these for the sake of issuing
a cl-protobufs fix to quicklisp ASAP.
I require these updates for some further changes I'm making to QUUX, anyway,
so it's just a matter of timing.
Scott McKay [Tue, 18 Sep 2012 16:04:09 +0000 (16:04 +0000)]
Speed up unoptimized serialization
Testing : precheckin --full --strict-errors
Reviewer: Fare
JTB impact: No
Ops impact: No
Change to config : No
Change to XML schema : No
Change to DB schema : No
Change to transport (timeouts, headers) : No
Any change (or new use) of OAQs : No
Change to inter-component transactions : No
Depends on any other checkin / bug : No
Tests that will verify:
The CL-Protobufs tests
Description:
The unoptimized serialization code was slower than
it needed to be and was a Niagara falls of consing.
The problem is that (map () (curry ...)) is poorly
optimized, which is a shame because it's such a nice
coding style. :-P
The fix is to replace 'map' with iteration (sigh) in
a few critical places.
- Add 'doseq', which chooses between 'dolist' or 'dovector'.
- Make 'serialize-object', 'deserialize-object' and 'object-size'
use 'doseq' instead of using 'map'.
- Ditto for 'print-text-format'.
- Fix the 'serialize-packed' and 'packed-size' optimizers to
use 'dolist' or 'dovector' based on whether the field is
a vector field. This makes the optimized code faster, too.
- Make the optimized serializer generators pass in 'vectorp'
so that the wire-format optimizers can do a better job.
Tsunami framework + use tsunami in customer profiles
Testing : precheckin --full (both against Oracle and against megastore for customers)
Reviewer: eschwartz
JTB impact: n
Ops impact: n
Security impact: n
Change to config : n
Change to XML schema : Y (XMLREQ bug 120820)
Change to DB schema : Y (DBREQ bug 119801)
Change to transport (timeouts, headers) : n
Any change (or new use) of OAQs : n
Change to inter-component transactions : sort of (use megastore in development-only environments)
Depends on any other checkin / bug : n
Tests that will verify : tsunami and megastore unit tests + existing customer profile tests
Tsunami is a new layer that sits between business logic code and the
persistence logic. The main manifestation of this layer is
define-domain-class macro that defines an interface to a domain
class. You can then define how to perist in various implementations,
e.g. using define-quake-db-class-bindings for quake and
define-proto-binding & define-megastore-proto-binding for megastore.
Tsunami framework is still a work in progress, and there is a bunch of
outstanding tasks/issues.
As the first application, I refactored customer profile code to use
the tsunami framework. This enables storing of customer profiles in
megastore (or other data stores in the future), while everything else
is stored in Oracle.
Here is the list of other miscellaneous/harder-to-understand changes:
- I changed @id attribute of ContactInfo objects to use the actual
numeric ID, rather than passing it through encode-record-locator (in
qres-dev schema only). This required changing XML schema type from
typeBasicLocatorString to typeBasicID.
- Drop quake/relationships.lisp. Use a more straightforward way to
associated ADO agencies to customer profiles.
- Customer usernames are now stored as lowercase strings (this means
that I don't need a separate field in megastore protobuf to
implement case-insensitive index).
- agency-id permission attribute is now the agency-key (serialized to
string), not the agency's record ID. This way you don't need to load
agency to do permission check when all you have is the agency-key
- moved agency-key from core/agencies.lisp to
lisp/core/agency-types.lisp, so that agency-key type can be used in
permissions
- add when* test handler that takes a random Lisp expression (as
opposed to the existing when that takes a variable name and looks at
whether it is empty)
- remove cc-deleted-error -- there is no such thing as deleting
cc-info
- refactored cc-information, so that it is used as a value object (ie
a mostly-immutable object with no identity) in the application. When
saving to Oracle database, it is saved in the central
tbl_cc_information table with appropriately populated ownership
columns (pointing to the customer, journey,
accountable_doc_container, etc)
- drop unneeded tbl_cc_transaction columns that were caching cc_info
values
projects / cl-protobufs.git / log