Commit graph

117 commits

Author SHA1 Message Date
Richard Newman
df58de52f4
Correctly parse and unescape quotes etc. inside EDN strings. (#434) (#589) 2018-03-15 07:13:27 -07:00
Richard Newman
1817ce7c0b Performance and cleanup. r=emily
* Use fixed-size arrays for bootstrap datoms, not vecs.
* Wide-ranging cleanup.

    This commit:
    - Deletes some dead code.
    - Marks some functions only used by tests as cfg(test).
    - Adds pub(crate) to a bunch of functions.
    - Cleans up a few other nits.
2018-03-06 09:03:00 -08:00
Richard Newman
9b23cf3945
Speed up EDN parser (fixes #445) (#581) r=nalexander
Fixes from @kevinmehall.

* Prefer character sets over backtracking in the EDN parser.
* Avoid duplicate effort when parsing floats in the EDN parser.
* Clean up duplicate position tracking code.

This turns out to have little performance impact, but makes the grammar
much cleaner.

* Fix EDN work to pass tests with correct numeric precedence.
2018-03-05 20:33:51 -08:00
Grisha Kruglov
84f29676e8
"Unchanged server" uploader flow (#543) r=rnewman
* Remove unused struct from tx_processor

* Derive serialize & deserialize for TypedValue

* First pass of uploader flow + feedback
2018-02-09 09:55:19 -08:00
Richard Newman
2614f498be Ergonomics improvements, including a kw macro. (#537) r=emily
* Add TypedValue::instant(micros).
* Add From<f64> for TypedValue.
* Add lookup_values_for_attribute to Conn.
* Add q_explain to Queryable.
* Expose an iterator over FindSpec's columns.
* Export edn from mentat crate. Export QueryExecutionResult.
* Implement Display for Variable and Element.
* Introduce a `kw` macro.

    This allows you to write:

    ```rust
    kw!(:foo/bar)
    ```

    instead of

    ```rust
    NamespacedKeyword::new("foo", "bar")
    ```

    … and it's more efficient, too.

Add `mentat::open`, eliminate use of `mentat_db` in some places.
2018-02-01 09:27:23 -08:00
Thom
9740cafdbd Automatically remove trailing whitespace from text files. (#527) r=rnewman
This was done using the following shell script:

```
find . -type f -not -path "*target*" \
       '(' -name '*.rs' -o -name '*.md' -o -name '*.toml' ')' -print0 | \
    xargs -0 sed -i '' -E 's/[[:space:]]*$//'
```

Which is admittedly imperfect, but manages to hit everything that was a problem in this repo.
2018-01-19 21:21:04 -06:00
Grisha Kruglov
c61bc79b99 Sync metadata schema and SyncMetadataClient. (#502) r=rnewman 2017-12-13 14:19:05 -06:00
Richard Newman
c600152d78
Update some dependencies. (#492) r=etoop
* Update some dependencies.

* Update rusqlite to 0.12.

* Update error-chain to a forked version that implements Sync.

* Fix some compiler warnings.

* Remove unused imports in tests.

* Parse errors no longer naturally print with the expected symbol.
2017-11-21 16:24:08 +00:00
Richard Newman
aa5f569df5 There are one million microseconds in a second, not one hundred thousand. (#480) r=fluffyemily 2017-06-16 08:00:14 -07:00
Nick Alexander
59a710f80f Review comments: another test, add unreversed(). 2017-06-08 10:30:31 -07:00
Nick Alexander
d1ac752de6 Parse without copying; parse keyword maps using macros.
This is a big commit, but it breaks into two conceptual pieces.  The
first is to "parse without copying".  We replace a stream of an owned
collection of edn::ValueAndSpan and instead have a stream of a
borrowed collection of &edn::ValueAndSpan references.  (Generally,
this is represented as an iterator over a slice, but it can be over
other things too.)  Cloning such iterators is constant time, which
improves on cloning an owned collection of edn::ValueAndSpan, which is
linear time in the length of the collection and additional time
depending on the complexity of the EDN values.

The second conceptual piece is to parse keyword maps using a special
parser and a macro to build the parser implementations.  Before, we
created a new edn::ValueAndSpan::Map to represent a keyword map in
vector form; since we're working with &edn::ValueAndSpan references
now, we can't create an &edn::ValueAndSpan reference with an
appropriate lifetime.  Therefore we generalize the concept of
iteration slightly and turn keyword maps in map form into linear
iterators by flattening the value maps.  This is a potentially
obscuring transformation, so we have to take care to protect against
some failure cases.  (See the comments and the tests in the code.)

After these changes, parsing using `combine` is linear time (and
reasonably fast).
2017-05-18 10:17:13 -07:00
Richard Newman
523d5ea5f1 Bump dependency versions. r=bgrins. (#441) 2017-05-03 12:53:16 -07:00
Richard Newman
daca8def57 UUIDs and instants. Fixes #44, #45, #426, #427. (#438) r=nalexander
* Pre: unused import in translate.rs.

* Part 2: take a dependency on rusqlite for query arguments.

* Part 1: flatten V2 schema into V1. Add UUID and URI.

Bump expected ident and bootstrap datom count in tests.

* Part 5: parse edn::Value::Uuid.

* Part 3: extend ValueType and TypedValue to include Uuid.

* Part 4: add Uuid to query arguments.

* Part 6: extend db to support Uuid.

* Part 8: add a tx-parser test for #f NaN and #uuid.

* Part 7: parse and algebrize UUIDs in queries.

* Part 1: parse #inst in EDN and throughout query engine.

* Part 3: handle instants in db.

* Part 2: instants never matches integers in queries.

* Part 4: use DateTime for tx_instants.

* Add a test for adding and querying UUIDs and instants.

* Review comments.
2017-04-28 20:11:55 -07:00
Nick Alexander
5369f03464 Improve parsing of nested edn::ValueAndSpan streams. r=rnewman (#393)
* Pre: Expose more in edn.

* Pre: Make it easier to work with ValueAndSpan.

with_spans() is a temporary hack, needed only because I don't care to
parse the bootstrap assertions from text right now.

* Part 1a: Add `value_and_span` for parsing nested `edn::ValueAndSpan` instances.

I wasn't able to abstract over `edn::Value` and `edn::ValueAndSpan`;
there are multiple obstacles.  I chose to roll with
`edn::ValueAndSpan` since it exposes the additional span information
that we will want to form good error messages in the future.

* Part 1b: Add keyword_map() parsing an `edn::Value::Vector` into an `edn::Value::map`.

* Part 1c: Add `Log`/`.log(...)` for logging parser progress.

This is a terrible hack, but it sure helps to debug complicated nested
parsers.  I don't even know what a principled approach would look
like; since our parser combinators are so frequently expressed in
code, it's hard to imagine a data-driven interpreter that can help
debug things.

* Part 2: Use `value_and_span` apparatus in tx-parser/.

I break an abstraction boundary by returning a value column
`edn::ValueAndSpan` rather than just an `edn::Value`.  That is, the
transaction processor shouldn't care where the `edn::Value` it is
processing arose -- even we care to track that information we should
bake it into the `Entity` type.  We do this because we need to
dynamically parse the value column to support nested maps, and parsing
requires a full `edn::ValueAndSpan`.  Alternately, we could cheat and
fake the spans when parsing nested maps, but that's potentially
expensive.

* Part 3: Use `value_and_span` apparatus in query-parser/.

* Part 4: Use `value_and_span` apparatus in root crate.

* Review comment: Make Span and SpanPosition Copy.

* Review comment: nits.

* Review comment: Make `or` be `or_exactly`.

I baked the eof checking directly into the parser, rather than using
the skip and eof parsers.  I also took the time to restore some tests
that were mistakenly commented out.

* Review comment: Extract and use def_matches_* macros.

* Review comment: .map() as late as possible.
2017-04-06 10:06:28 -07:00
Nick Alexander
4b874deae1 Lookup refs, nested vector values, map notation. Fixes #180, fixes #183, fixes #284. (#382) r=rnewman
* Pre: Fix error in parser macros.

* Pre: Make test unwrapping more verbose.

* Pre: Make lookup refs be (lookup-ref a v) in the entity position.

This has the advantage of being explicit in all situations and
unambiguous at parse-time.  This choice agrees with the Clojure
implementation but not with Datomic.  Datomic treats [a v] as a lookup
ref, is ambiguous at parse-time, and is disambiguated in ways I do not
understand at transaction time.  We mooted making lookup refs [[a v]]
and outlawing nested value vectors in transactions, but after
implementing that approach I decided it was better to handle lookup
refs at parse time and therefore outlawing nested value vectors is not
necessary.

* Handle lookup refs in the entity and value columns. Fixes #183.

* Pre 0a: Use a stack instead of into_iter.

* Pre 0b: Dedent.

* Pre 0c: Handle `e` after `v`.

This allows to use the original `e` while handling `v`.

* Explode value lists for :db.cardinality/many attributes. Fixes #284.

* Parse and accept map notation. Fixes #180.

* Pre: Modernize add() and retract() into one add_or_retract().

* Pre: Add is_collection and is_atom to edn::Value.

* Pre: Differentiate atoms from lookup-refs in value position.

Initially, I expected to accept arbitrary edn::Value instances in the
value position, and to differentiate in the transactor.  However, the
implementation quickly became a two-stage parser, since we always
wanted to parse the resulting value position into some other known
thing using the tx-parser.  To save calls into the parser and to allow
the parser to move forward with a smaller API surface, I push as much
of this parsing as possible into the initial parse.

* Pre: Modernize entities().

* Pre: Quote edn::Value::Text in Display.

* Review comment: Add and use edn::Value::into_atom.

* Review comment: Use skip(eof()) throughout.

* Review comment: VecDeque instead of Vec.

* Review comment: Part 0: Rename TempId to TempIdHandle.

* Review comment: Part 1: Differentiate internal and external tempids.

This breaks an abstraction boundary by pushing the Internal/External
split up to the Entity level in tx/ and tx-parser/.  This just makes
it easier to explode Entity map notation instances into Entity
instances, taking an existing External tempid :db/id or generating a
new Internal tempid as appropriate.  To do this without breaking the
abstraction boundary would require adding flexibility to the
transaction processor: we'd need to be able to turn Entity instances
into some internal enum and handle the two cases independently.  It
wouldn't be too hard, but this reduces the combinatorial type
explosion.
2017-03-27 16:30:04 -07:00
Richard Newman
85f3b79f75 Support a limited set of '.'-prefixed non-keyword symbols. (#352) r=nalexander
This commit allows `.` and `...` to parse correctly as `PlainSymbol`.

Tests in edn, query-translator, and the top level have been added.
2017-03-06 15:01:19 -08:00
Nick Alexander
dcd9bcb1ce Extract partial storage abstraction; use error-chain throughout. Fixes #328. r=rnewman (#341)
* Pre: Drop unneeded tx0 from search results.

* Pre: Don't require a schema in some of the DB code.

The idea is to separate the transaction applying code, which is
schema-aware, from the concrete storage code, which is just concerned
with getting bits onto disk.

* Pre: Only reference Schema, not DB, in debug module.

This is part of a larger separation of the volatile PartitionMap,
which is modified every transaction, from the stable Schema, which is
infrequently modified.

* Pre: Fix indentation.

* Extract part of DB to new SchemaTypeChecking trait.

* Extract part of DB to new PartitionMapping trait.

* Pre: Don't expect :db.part/tx partition to advance when tx fails.

This fails right now, because we allocate tx IDs even when we shouldn't.

* Sketch a db interface without DB.

* Add ValueParseError; use error-chain in tx-parser.

This can be simplified when
https://github.com/Marwes/combine/issues/86 makes it to a published
release, but this unblocks us for now.  This converts the `combine`
error type `ParseError<&'a [edn::Value]>` to a type with owned
`Vec<edn::Value>` collections, re-using `edn::Value::Vector` for
making them `Display`.

* Pre: Accept Borrow<Schema> instead of just &Schema in debug module.

This makes it easy to use Rc<Schema> or Arc<Schema> without inserting
&* sigils throughout the code.

* Use error-chain in query-parser.

There are a few things to point out here:

- the fine grained error types have been flattened into one crate-wide
  error type; it's pretty easy to regain the granularity as needed.

- edn::ParseError is automatically lifted to
  mentat_query_parser::errors::Error;

- we use mentat_parser_utils::ValueParser to maintain parsing error
  information from `combine`.

* Patch up top-level.

* Review comment: Only `borrow()` once.
2017-02-24 15:33:48 -08:00
Victor Porof
7fc2a22d68 Implement a basic edn matcher, r=ncalexan (#271) (#338)
Signed-off-by: Victor Porof <victor.porof@gmail.com>
2017-02-23 09:11:34 +01:00
Victor Porof
bf707acbc3 Lint for the clippy gods in the edn crate (#340)
Signed-off-by: Victor Porof <victor.porof@gmail.com>
2017-02-22 18:11:05 +01:00
Victor Porof
0d3b8e4b29 Avoid code duplication for common Value trait implementations, r=ncalexan
Signed-off-by: Victor Porof <victor.porof@gmail.com>
2017-02-22 08:36:45 +01:00
Victor Porof
1b26e23d02 Implement edn pretty printing using pretty.rs. Fixes #195. (#245)
* Implement pretty printing

Signed-off-by: Victor Porof <victor.porof@gmail.com>

* Rewrite pretty printing.

This does a few things.  First, it use pretty.rs directly, without the
layer of macro obfuscation.  The code is significantly simpler as a
result.

Second, it tightens the layout, using pretty.rs to group nested
layouts that fit on a single line.  This is Clojure's EDN style, more
or less.

Third, it drops "special format" support for queries.  This wasn't
completely implemented; if we want it, we can newtype
Query(edn::Value) and figure out how to really implement this idea.

* Rename to reflect functionality.

* Make write interface more Rust-like.

There isn't a clear standard in the stdlib, but a function that takes
ownership of a writer and then returns it back is definitely not
Rust-like.  That's what a (mutable) reference is for.

* Review comment: Use as_ref to avoid cloning strings.

* Post: Fix tests to use `without_spans()`.
2017-02-21 11:48:08 -08:00
Richard Newman
a10f68fdb7 Mark every project as being part of the workspace. r=nalexander
This allows `cargo test --all` to work.
2017-02-20 11:04:08 -08:00
Jordan Santell
a59f9583ac Store Idents as NamespacedKeywords, rather than Strings. Fixes #291. (#300)
r=ncalexander
2017-02-17 13:55:36 -08:00
Victor Porof
896d7f8f88 Add a span component to edn::Value, r=ncalexan
Signed-off-by: Victor Porof <victor.porof@gmail.com>
2017-02-17 18:31:26 +01:00
Joe Walker
89949fb451 Update README for edn; r=me 2017-02-16 18:32:36 +00:00
Jordan Santell
4f5c94891a Add octal, hexadecimal, and arbitrary base integers to the EDN parser. Fixes #277. r=rnewman (#286) 2017-02-10 16:03:35 -08:00
Victor Porof
49f91b05b0 Expose EDN into_ methods that consume the edn value, r=jwalker. Fixes 256
Signed-off-by: Victor Porof <victor.porof@gmail.com>
2017-02-10 14:58:38 +01:00
Richard Newman
4366f6d61f Expose EDN as_ methods that return copied values, not references. 2017-02-09 10:21:35 -08:00
Victor Porof
42580539b8 Properly handle whitespace for Infinity and NaN, r=rnewman (#246)
Signed-off-by: Victor Porof <vporof@mozilla.com>
2017-02-09 18:13:44 +01:00
Victor Porof
c585715224 Don't depend on num and ordered-float in the db and query crates, r=ncalexan (#223)
Signed-off-by: Victor Porof <victor.porof@gmail.com>
2017-02-08 12:19:16 +01:00
Victor Porof
4d83aafa2a Ensure printing/parsing edn strings is isomorphic
Signed-off-by: Victor Porof <vporof@mozilla.com>
2017-02-04 08:47:48 +01:00
Victor Porof
a627f532f0 Relax whitespace rules for edn vectors, lists, sets and maps
Signed-off-by: Victor Porof <vporof@mozilla.com>
2017-02-04 08:45:31 +01:00
Victor Porof
419db388da Relax whitespace rules for Infinity and NaN
Signed-off-by: Victor Porof <vporof@mozilla.com>
2017-02-04 08:45:02 +01:00
Jordan Santell
0b20d7691b Parse and display EDN values for NaN, +Infinity and -Infinity. Fixes #232 (#238) r=victorporof 2017-02-03 10:14:23 -08:00
Victor Porof
c038c11017 Consolidate edn peg rules to better parse keywords and symbols, r=ncalexan. Fixes #219 2017-02-03 09:08:24 +01:00
Victor Porof
9ee0ac8e00 Unify and generalize keywords and symbols parsing
Signed-off-by: Victor Porof <vporof@mozilla.com>
2017-02-03 09:06:42 +01:00
Victor Porof
72da5722ae Update rustpeg to latest version and follow new syntax and formatting rules
Signed-off-by: Victor Porof <vporof@mozilla.com>
2017-02-03 09:06:42 +01:00
Victor Porof
611fbe2eef Properly print null edn values as "nil", to allow for isomorphic write/parse
Signed-off-by: Victor Porof <vporof@mozilla.com>
2017-02-03 09:06:42 +01:00
Richard Newman
5b770a54cd Parse basic :find and :where clauses. (#211) r=nalexander
* Make Variable::from_symbol public.
* Implement basic parsing of queries.
* Use pinned dependencies the hard way to fix Travis.
* Bump ordered-float dependency to 0.4.0.
* Error coercions to use ?, and finishing the find interface.
2017-02-02 18:32:00 -08:00
Victor Porof
707ce36236 Don't use single-character string constants in the is_backward function
See https://github.com/Manishearth/rust-clippy/wiki#single_char_pattern for further info

Signed-off-by: Victor Porof <vporof@mozilla.com>
2017-02-02 19:55:29 +01:00
Victor Porof
9a5ece8c89 Handle more edn::Value types for printing, precursor for #195
Signed-off-by: Victor Porof <vporof@mozilla.com>
2017-02-02 17:22:58 +01:00
Victor Porof
2ecda0a2bd Avoid needless reborrows and simplify Ord implementation for edn::Value 2017-02-02 17:22:58 +01:00
Victor Porof
a685d6c541 Move edn test functions into a submodule
Signed-off-by: Victor Porof <vporof@mozilla.com>
2017-02-02 17:22:58 +01:00
Victor Porof
cc56cec11a Add note about linked lists data type choice for edn::Value 2017-02-02 17:22:58 +01:00
Victor Porof
00048d1955 Remove edn::Pair struct since it's not used anywhere
Signed-off-by: Victor Porof <vporof@mozilla.com>
2017-02-02 17:22:58 +01:00
Victor Porof
4e9e8ed837 Use idiomatic enumerate method on interators instead of iterating over indices
Signed-off-by: Victor Porof <vporof@mozilla.com>
2017-02-02 10:59:03 +01:00
Victor Porof
4b2c7870c0 Wrap code indicated by the "_" in documentation as suggested by rustdoc best practices
Signed-off-by: Victor Porof <vporof@mozilla.com>
2017-02-02 10:59:03 +01:00
Victor Porof
17bc85fe27 Remove return statements from edn parser tests
Signed-off-by: Victor Porof <vporof@mozilla.com>
2017-02-02 10:59:03 +01:00
Victor Porof
25474980b1 Add rustdoc comments for to_symbol and to_keyword functions
Signed-off-by: Victor Porof <vporof@mozilla.com>
2017-02-02 10:59:03 +01:00
Victor Porof
8f68f68378 Use idiomatic map_or_else calls to Option<T> instead of double returns
Signed-off-by: Victor Porof <vporof@mozilla.com>
2017-02-02 10:59:03 +01:00
Victor Porof
85da91a0ab Add helper functions constructing OrderedFloat and BigInt to edn crate, r=ncalexan,rnewman. Fixes #198
Signed-off-by: Victor Porof <vporof@mozilla.com>
2017-02-02 10:58:08 +01:00
Richard Newman
a9929249eb Use Into<Option<>> trick for to_keyword and to_symbol. 2017-02-01 17:46:53 -08:00
Richard Newman
0b3387f8b9 Minor EDN cleanup. (#217) r=jsantell
* to_reverse -> to_reversed.
* Add PlainSymbol::plain_name for examining $x and ?y.
* Fix comment.
2017-02-01 14:34:51 -08:00
Richard Newman
932a42866c Fix comments in EDN crate. 2017-01-30 17:45:02 -08:00
Jordan Santell
0a21b5dca4 Add NamespacedKeyword::{is_forward, is_backward, to_reverse} for
inspection in NamespacedKewyord for usage by query and transaction parsers. Fixes #202
2017-01-30 14:39:53 -08:00
Jordan Santell
359d356dd9 Merge pull request #213 from mozilla/199
Add Into<String> to symbol::* constructors. Fixes #199
2017-01-30 13:35:12 -08:00
Jordan Santell
d116fd7bff Add is_$type and as_$type methods to edn::Value types and add tests. Fixes #197 2017-01-30 09:35:33 -08:00
Jordan Santell
18279fdd3c Add Into<String> to symbol::* constructors. Fixes #199 2017-01-28 22:53:40 -08:00
Richard Newman
c6fa14c0c8 Rudimentary printing of EDN values. (#209) r=jsantell
* Add a little From helper for edn::parse::ParseError. Not used yet.

* Ignore more things.

* Partly implement Display for edn::Value.
2017-01-28 14:18:17 -08:00
Nick Alexander
81af295948 Start installing SQL schema. (#171) r=rnewman
* Start installing the SQLite store and bootstrapping the datom store.

* Review comment: Decomplect V2_IDENTS.

* Review comment: Decomplect V2_PARTS.

* Review comment: Pre: Expose Clojure's merge on Value instances.

* Review comment: Decomplect V2_SYMBOLIC_SCHEMA.

* Review comment: Decomplect V1_STATEMENTS.

* Review comment: Prefer ? to try!.

* Review comment: Fix typos; format; add TODOs.

* Review comment: Assert that Mentat `Schema` is valid upon creation.

* Review comment: Improve conversion to and from SQL values.

This patch factors the fundamental SQL conversion maps
between (rusqlite::Value, value_type_tag) and (edn::Value, ValueType)
through a new Mentat TypedValue.  (A future patch might rename this
fundamental type mentat::Value.)

To make certain conversion functions infallible, I removed
placeholders for :db.type/{instant,uuid,uri}.  (We could panic
instead, but there's no need to do that right now.)

* Review comment: Always uses bundled SQLite in rusqlite.

This avoids (runtime) failures in Travis CI due to old SQLite
versions.  See 432966ac77.

* Review comment: Move semantics in `from_sql_value_pair`.

* Review comment: DB_EXCISE_BEFORE_T instead of ...BEFORET (no underscore).

* Review comment: Move overview notes to the Wiki.
2017-01-25 16:13:56 -08:00
Richard Newman
2592506288 Implement parsing of simple :find expressions. (#196) r=nalexander
* Test the mentat_query directory on Travis.

* Export common types from edn.

This allows you to write

  use edn::{PlainSymbol,Keyword};

instead of

  use edn:🔣:{PlainSymbol,Keyword};

* Add an edn::Value::is_keyword predicate.

* Clean up query, preparing for query-parser.

* Make EDN keywords and symbols take Into<String> arguments.

* Implement parsing of simple :find lists.

* Rustfmt query-parser. Split find and query.

* Review comment: values_to_variables now returns a NotAVariableError on failure.

* Review comment: rename gimme to to_parsed_value.

* Review comment: add comments.
2017-01-25 14:06:19 -08:00
Nick Alexander
ab041291fb edn: Bound values by optional whitespace; treat comma as whitespace. 2017-01-18 08:34:27 -08:00
Nick Alexander
247035cc9b edn: Allow comments.
EDN supports only one type of comment: initiated by ; and lasting
until the end of the current line or the end of the input stream.
2017-01-18 08:34:27 -08:00
Nick Alexander
b11b9b909c Add tx{-parser} crates; start parsing transactions. (#164) r=rnewman
This depends on edn and uses the combine parser combinator library.
2017-01-12 16:08:29 -08:00
Richard Newman
a152e60040 Read EDN keywords and symbols as rich types. Fixes #154. r=nalexander 2017-01-12 09:09:48 -08:00
Joe Walker
c4735119c4 Implement a basic EDN parser. (#149) r=rnewman,bgrins,nalexander
The parser mostly works and has a decent test suite. It parses all the
queries issued by the Tofino UAS, with some caveats. Known flaws:

* No support for tagged elements, comments, discarded elements or "'".
* Incomplete support for escaped characters in strings and the range of
  characters that are allowed in keywords and symbols.
* Possible whitespace handling problems.
2017-01-11 13:03:04 -08:00
Richard Newman
476f04e27b Implement a rudimentary Keyword struct and the beginnings of ident/entid. 2017-01-09 12:31:56 -08:00