Commit graph

53 commits

Author SHA1 Message Date
Nick Alexander
a8073056f2 Part 3: Move query into edn.
It's unfortunate to squash two crates together like this, but it's the
best option.
2018-06-04 15:04:37 -07:00
Nick Alexander
a4a8892309 Part 3a: Move file to preserve blame. 2018-06-04 14:56:56 -07:00
Nick Alexander
1d8d94f887 Part 2: Turn (type-function ?var) into (type ?var type-keyword).
This is more general (the parser doesn't encode the set of known
types), and avoids a dependency on `ValueType`.
2018-06-04 14:52:51 -07:00
Richard Newman
60cb5d2432
Pull improvements (#682) r=nalexander
* Parse and handle aliased pull attributes.
* Allow :db/id to be mentioned as a pull attribute.
* Clean up comment.
* Remove unused function.
2018-05-13 14:15:36 -07:00
Richard Newman
3dc68bcd38 Combine NamespacedKeyword and Keyword. (#689) r=nalexander
* Make properties on NamespacedKeyword/NamespacedSymbol private

* Use only a single String for NamespacedKeyword/NamespacedSymbol

* Review comments.

* Remove unsafe code in namespaced_name.

Benchmarking shows approximately zero change.

* Allow the types of ns and name to differ when constructing a NamespacedName.

* Make symbol namespaces optional.

* Normalize names of keyword/symbol constructors.

This will make the subsequent refactor much less painful.

* Use expect not unwrap.

* Merge Keyword and NamespacedKeyword.
2018-05-11 09:52:17 -07:00
Richard Newman
e21156a754
Implement simple pull expressions (#638) r=nalexander
* Refactor AttributeCache populator code for use from pull.

* Pre: add to_value_rc to Cloned.

* Pre: add From<StructuredMap> for Binding.

* Pre: clarify Store::open_empty.

* Pre: StructuredMap cleanup.

* Pre: clean up a doc test.

* Split projector crate. Pass schema to projector.

* CLI support for printing bindings.

* Add and use ConjoiningClauses::derive_types_from_find_spec.

* Define pull types.

* Implement pull on top of the attribute cache layer.

* Add pull support to the projector.

* Parse pull expressions.

* Add simple pull support to connection objects.

* Tests for pull.

* Compile with Rust 1.25.

The only choice involved in this commit is that of replacing the
anonymous lifetime '_ with a named lifetime for the cache; since we're
accepting a Known, which includes the cache in question, I think it's
clear that we expect the function to apply to any given cache
lifetime.

* Review comments.

* Bail on unnamed attribute.

* Make assert_parse_failure_contains safe to use.

* Rework query parser to report better errors for pull.

* Test for mixed wildcard and simple attribute.
2018-05-04 12:56:00 -07:00
Richard Newman
f979044ba1
Refactor value type boxing. (#659) r=nalexander
* Pre: eliminate some occurrences of Rc, largely through the magic of Into.
* Pre: introduce FromRc to convert between refcounted types.
* Introduce ValueRc as an abstraction over Rc/Arc choice.
* Move Cloned to core.
* Move CString-creation methods to TypedValue.
* Finish transition.
2018-04-25 14:23:27 -07:00
Richard Newman
a74a2deffc
Introduce RelResult rather than Vec<Vec<TypedValue>>. (#639) r=nalexander
* Pre: clean up core/src/lib.rs.
* Pre: use indexmap 1.0 in db and query-projector.
* Change rel results to be a RelResult instance, not a Vec<Vec<TypedValue>>.

This avoids memory fragmentation and improves locality by using a single
heap-allocated vector for all bindings, rather than a separate
heap-allocated vector for each row.

We hide this abstraction behind the `RelResult` type, which tracks the
stride length (width) of each row.

* Don't allocate temporary vectors when projecting RelResults.
2018-04-24 15:04:00 -07:00
Richard Newman
16a66517e4
Err, don't panic, on unbound variable (#518) (#590) r=emily
* Pre: switch a 'panic' to an 'unreachable'.
* Make candidate_type_column fallible. (#518)
2018-03-15 11:39:24 -07:00
Richard Newman
833ff92436
Simple aggregates. (#584) r=emily
* Pre: use debugcli in VSCode.
* Pre: wrap subqueries in parentheses in output SQL.
* Pre: add ExistingColumn.

This lets us make reference to columns by name, rather than only
pointing to qualified aliases.

* Pre: add Into for &str to TypedValue.
* Pre: add Store.transact.
* Pre: cleanup.
* Parse and algebrize simple aggregates. (#312)
* Follow-up: print aggregate columns more neatly in the CLI.
* Useful ValueTypeSet helpers.
* Allow for entity inequalities.
* Add 'differ', which is a ref-specialized not-equals.
* Add 'unpermute', a function for getting unique, distinct pairs from bindings.
* Review comments.
* Add 'the' pseudo-aggregation operator.

This allows for a corresponding value to be returned when a query
includes one 'min' or 'max' aggregate.
2018-03-12 15:18:50 -07:00
Richard Newman
f42ae35b70 Update cache on write. (#566) r=emily
* Use the cache to make constant queries super fast.
* Fix translate tests to match: we no longer generate SQL for many of them!
* Accumulate additions and removals into the cache.
    * Make attribute cache clone-on-write; store it in Metadata.
    * Allow caching of fulltext attributes, interning strings.
2018-03-06 09:01:20 -08:00
Richard Newman
2614f498be Ergonomics improvements, including a kw macro. (#537) r=emily
* Add TypedValue::instant(micros).
* Add From<f64> for TypedValue.
* Add lookup_values_for_attribute to Conn.
* Add q_explain to Queryable.
* Expose an iterator over FindSpec's columns.
* Export edn from mentat crate. Export QueryExecutionResult.
* Implement Display for Variable and Element.
* Introduce a `kw` macro.

    This allows you to write:

    ```rust
    kw!(:foo/bar)
    ```

    instead of

    ```rust
    NamespacedKeyword::new("foo", "bar")
    ```

    … and it's more efficient, too.

Add `mentat::open`, eliminate use of `mentat_db` in some places.
2018-02-01 09:27:23 -08:00
Thom Chiovoloni
98502eb68f Implement type annotations in queries. (#526) r=rnewman 2018-01-29 14:37:53 -08:00
Richard Newman
e8ec59e464
Implement a simple direct lookup API. Fixes #111 (#503) r=grisha
* Add some helpers and refactor how queries are run (once).
* Implement lookup_value_for_attribute.
* Add a multi-value test for lookup_value_for_attribute.
2017-12-11 11:08:10 -08:00
Richard Newman
df90c366af
Partial work from simple aggregates work (#497) r=nalexander
* Pre: make FindQuery, FindSpec, and Element non-Clone.
* Pre: make query translator return a Result.
* Pre: make projection return a Result.
* Pre: refactor query parser in preparation for parsing aggregates.
* Pre: rename PredicateFn -> QueryFunction.
* Pre: expose more about bound variables from CC.
* Pre: move ValueTypeSet to core.
2017-11-30 15:02:07 -08:00
Richard Newman
c600152d78
Update some dependencies. (#492) r=etoop
* Update some dependencies.

* Update rusqlite to 0.12.

* Update error-chain to a forked version that implements Sync.

* Fix some compiler warnings.

* Remove unused imports in tests.

* Parse errors no longer naturally print with the expected symbol.
2017-11-21 16:24:08 +00:00
Richard Newman
eaf3e7fc4b Extend inequalities to Instants. (#439) r=fluffyemily,nalexander 2017-06-16 11:57:44 -07:00
Nick Alexander
79fa0994b3 Part 3: Handle ground. (#469) r=nalexander,rnewman
This version removes nalexander's lovely matrix code. It turned out
that scalar and tuple bindings are sufficiently different from coll
and rel -- they can directly apply as values in the query -- that
there was no point in jumping through hoops to turn those single
values into a matrix.

Furthermore, I've standardized us on a Vec<TypedValue>
representation for rectangular matrices, which should be much
more efficient, but would have required rewriting that code.

Finally, coll and rel are sufficiently different from each other
-- coll doesn't require processing nested collections -- that
my attempts to share code between them fell somewhat flat. I had
lots of nice ideas about zipping together cycles and such, but
ultimately I ended up with relatively straightforward, if a bit
repetitive, code.

The next commit will demonstrate the value of this work -- tests
that exercised scalar and tuple grounding now collapse down to
the simplest possible SQL.
2017-06-09 20:18:31 -07:00
Nick Alexander
b9cbf92205 Part 1: Parse functions in where clauses. 2017-06-09 20:16:39 -07:00
Richard Newman
c6e933c396 Pre: make rule_vars return unique vars. 2017-06-09 20:16:39 -07:00
Nick Alexander
9fe31d443d Pre: Accept EDN vectors in FnArg arguments.
Datomic accepts mostly-arbitrary EDN, and it is actually used: for
example, the following are all valid, and all mean different things:
* `(ground 1 ?x)`
* `(ground [1 2 3] [?x ?y ?z])`
* `(ground [[1 2 3] [4 5 6]] [[?x ?y ?z]])`

We could probably introduce new syntax that expresses these patterns
while avoiding collection arguments, but I don't see one right now.
I've elected to support only vectors for simplicity; I'm hoping to
avoid parsing edn::Value in the query-algebrizer.
2017-06-09 20:16:36 -07:00
Nick Alexander
08534a1a3a Pre: Handle SrcVar. 2017-06-09 20:16:36 -07:00
Nick Alexander
d1ac752de6 Parse without copying; parse keyword maps using macros.
This is a big commit, but it breaks into two conceptual pieces.  The
first is to "parse without copying".  We replace a stream of an owned
collection of edn::ValueAndSpan and instead have a stream of a
borrowed collection of &edn::ValueAndSpan references.  (Generally,
this is represented as an iterator over a slice, but it can be over
other things too.)  Cloning such iterators is constant time, which
improves on cloning an owned collection of edn::ValueAndSpan, which is
linear time in the length of the collection and additional time
depending on the complexity of the EDN values.

The second conceptual piece is to parse keyword maps using a special
parser and a macro to build the parser implementations.  Before, we
created a new edn::ValueAndSpan::Map to represent a keyword map in
vector form; since we're working with &edn::ValueAndSpan references
now, we can't create an &edn::ValueAndSpan reference with an
appropriate lifetime.  Therefore we generalize the concept of
iteration slightly and turn keyword maps in map form into linear
iterators by flattening the value maps.  This is a potentially
obscuring transformation, so we have to take care to protect against
some failure cases.  (See the comments and the tests in the code.)

After these changes, parsing using `combine` is linear time (and
reasonably fast).
2017-05-18 10:17:13 -07:00
Richard Newman
daca8def57 UUIDs and instants. Fixes #44, #45, #426, #427. (#438) r=nalexander
* Pre: unused import in translate.rs.

* Part 2: take a dependency on rusqlite for query arguments.

* Part 1: flatten V2 schema into V1. Add UUID and URI.

Bump expected ident and bootstrap datom count in tests.

* Part 5: parse edn::Value::Uuid.

* Part 3: extend ValueType and TypedValue to include Uuid.

* Part 4: add Uuid to query arguments.

* Part 6: extend db to support Uuid.

* Part 8: add a tx-parser test for #f NaN and #uuid.

* Part 7: parse and algebrize UUIDs in queries.

* Part 1: parse #inst in EDN and throughout query engine.

* Part 3: handle instants in db.

* Part 2: instants never matches integers in queries.

* Part 4: use DateTime for tx_instants.

* Add a test for adding and querying UUIDs and instants.

* Review comments.
2017-04-28 20:11:55 -07:00
Emily Toop
bd389d2f0d Parse and Algebrize not & not-join. (#302) (Closes #303, #389, #422 ) r=rnewman
* Part 1 - Parse `not` and `not-join`

* Part 2 - Validate `not` and `not-join` pre-algebrization

* Address review comments rnewman.
* Remove `WhereNotClause` and populate `NotJoin` with `WhereClause`.
* Fix validation for `not` and `not-join`, removing tests that were invalid.
* Address rustification comments.

* Rebase against `rust` branch.

* Part 3 - Add required types for NotJoin.
* Implement `PartialEq` for
`ConjoiningClauses` so `ComputedTable` can be included inside `ColumnConstraint::NotExists`

* Part 4 - Implement `apply_not_join`

* Part 5 - Call `apply_not_join` from inside `apply_clause`

* Part 6 - Translate `not-join` into `NOT EXISTS` SQL

* Address review comments.

* Rename `projected` to `unified` to better describe the fact that we are not projecting any variables.
* Check for presence of each unified var in either `column_bindings` or `input_bindings` and bail if not there.
* Copy over `input_bindings` for each var in `unified`.
* Only copy over the first `column_binding` for each variable in `unified` rather than the whole list.
* Update tests.

* Address review comments.

* Make output from Debug for NotExists more useful

* Clear up misunderstanding. Any single failing clause in the not will cause the entire not to be considered empty

* Address review comments.

* Remove Limit requirement from cc_to_exists.
* Use Entry.or_insert instead of matching on the entry to add to column_bindings.
* Move addition of value_bindings to before apply_clauses on template.
* Tidy up tests with some variable reuse.
* Addressed nits,

* Address review comments.

* Move addition of column_bindings to above apply_clause.
* Update tests.

* Add test to ensure that unbound vars fail

* Improve test for unbound variable to check for correct variable and error

* address nits
2017-04-28 10:44:11 +01:00
Richard Newman
bc63744aba Add :limit to queries (#420) r=nalexander
* Pre: put query parts in alphabetical order.
* Pre: rename 'input' to 'query' in translate tests.
* Part 1: parse :limit.
* Part 2: validate and escape variable parameters in SQL.
* Part 3: algebrize and translate limits.
2017-04-19 16:16:19 -07:00
Richard Newman
60c082b61e Part 4: pass inputs through algebrizing and execution. (#418)
This also adds a test that an `UnboundVariables` error is raised if a
variable mentioned in the `:in` clause isn't bound.
2017-04-18 13:19:50 -07:00
Richard Newman
a9a82ea1a7 Part 1: parse :in.
We also at this point switch from using `Vec<Variable>` to
`BTreeSet<Variable>`. This allows us to guarantee no duplicates later;
we'll reject duplicates at parse time.
2017-04-18 13:19:50 -07:00
Richard Newman
35d73d5541 Implement :order. (#415) (#416) r=nalexander
This adds an `:order` keyword to `:find`.

If present, the results of the query will be an ordered set, rather than
an unordered set; rows will appear in an ordered defined by each
`:order` entry.

Each can be one of three things:

- A var, `?x`, meaning "order by ?x ascending".
- A pair, `(asc ?x)`, meaning "order by ?x ascending".
- A pair, `(desc ?x)`, meaning "order by ?x descending".

Values will be ordered in this sequence for asc, and in reverse for desc:

1. Entity IDs, in ascending numerical order.
2. Booleans, false then true.
3. Timestamps, in ascending numerical order.
4. Longs and doubles, intermixed, in ascending numerical order.
5. Strings, in ascending lexicographic order.
6. Keywords, in ascending lexicographic order, considering the entire
   ns/name pair as a single string separated by '/'.

Subcommits:

Pre: make bound_value public.
Pre: generalize ErrorKind::UnboundVariable for use in order.
Part 1: parse (direction, var) pairs.
Part 2: parse :order clause into FindQuery.
Part 3: include order variables in algebrized query.

We add order variables to :with, so we can reuse its type tag projection
logic, and so that we can phrase ordering in terms of variables rather
than datoms columns.

Part 4: produce SQL for order clauses.
2017-04-17 11:30:31 -07:00
Richard Newman
d8075aa07d Part 3: finish expansion and translation of complex or.
This commit turns complex `or` -- `or`s in which not all variables are
unified, or in which not all arms are the same shape -- into a
computed table.

We do this by building a template CC that shares some state with the
destination CC, applying each arm of the `or` to a copy of the template
as if it were a standalone query, then building a projection list and
creating a `ComputedTable::Union`. This is pushed into the destination
CC's `computed_tables` list.

Finally, the variables projected from the UNION are bound in the
destination CC, so that unification occurs, and projection of the
outermost query can use bindings established by the `or-join`.

This commit includes projection of type codes from heterogeneous `UNION`
arms: we compute a list of variables for which a definite type is
unknown in at least one arm, and force all arms to project either a type
tag column or a fixed type. It's important that each branch of a UNION
project the same columns in the same order, hence the projection of
fixed values.

The translator is similarly extended to project the type tag column name
or the known value_type_tag to support this.

Review comment: clarify union type extraction.
2017-04-12 19:21:45 -07:00
Richard Newman
0639c94468 Part 2: implement simple or. 2017-04-07 12:46:25 -07:00
Nick Alexander
5369f03464 Improve parsing of nested edn::ValueAndSpan streams. r=rnewman (#393)
* Pre: Expose more in edn.

* Pre: Make it easier to work with ValueAndSpan.

with_spans() is a temporary hack, needed only because I don't care to
parse the bootstrap assertions from text right now.

* Part 1a: Add `value_and_span` for parsing nested `edn::ValueAndSpan` instances.

I wasn't able to abstract over `edn::Value` and `edn::ValueAndSpan`;
there are multiple obstacles.  I chose to roll with
`edn::ValueAndSpan` since it exposes the additional span information
that we will want to form good error messages in the future.

* Part 1b: Add keyword_map() parsing an `edn::Value::Vector` into an `edn::Value::map`.

* Part 1c: Add `Log`/`.log(...)` for logging parser progress.

This is a terrible hack, but it sure helps to debug complicated nested
parsers.  I don't even know what a principled approach would look
like; since our parser combinators are so frequently expressed in
code, it's hard to imagine a data-driven interpreter that can help
debug things.

* Part 2: Use `value_and_span` apparatus in tx-parser/.

I break an abstraction boundary by returning a value column
`edn::ValueAndSpan` rather than just an `edn::Value`.  That is, the
transaction processor shouldn't care where the `edn::Value` it is
processing arose -- even we care to track that information we should
bake it into the `Entity` type.  We do this because we need to
dynamically parse the value column to support nested maps, and parsing
requires a full `edn::ValueAndSpan`.  Alternately, we could cheat and
fake the spans when parsing nested maps, but that's potentially
expensive.

* Part 3: Use `value_and_span` apparatus in query-parser/.

* Part 4: Use `value_and_span` apparatus in root crate.

* Review comment: Make Span and SpanPosition Copy.

* Review comment: nits.

* Review comment: Make `or` be `or_exactly`.

I baked the eof checking directly into the parser, rather than using
the skip and eof parsers.  I also took the time to restore some tests
that were mistakenly commented out.

* Review comment: Extract and use def_matches_* macros.

* Review comment: .map() as late as possible.
2017-04-06 10:06:28 -07:00
Richard Newman
a5023c70cb Use Rc for TypedValue, Variable, and query Ident keywords. (#395) r=nalexander
Part 1, core: use Rc for String and Keyword.
Part 2, query: use Rc for Variable.
Part 3, sql: use Rc for args in SQLiteQueryBuilder.
Part 4, query-algebrizer: use Rc.
Part 5, db: use Rc.
Part 6, query-parser: use Rc.
Part 7, query-projector: use Rc.
Part 8, query-translator: use Rc.
Part 9, top level: use Rc.
Part 10: intern Ident and IdentOrKeyword.
2017-04-02 21:38:36 -07:00
Richard Newman
72eeedec74 Part 4: add OrJoin::is_fully_unified.
This allows us to tell if all the variables in a valid `or` join are to
be unified, which is necessary for simple joins.
2017-03-30 19:13:20 -07:00
Richard Newman
460fdac252 Pre: add Variable::from_valid_name, TypedValue::{typed_string,typed_ns_keyword}. 2017-03-30 19:13:19 -07:00
Richard Newman
439f3a2283 Pre: add some 'am I a pattern?' helper predicates to clause types. 2017-03-30 19:06:33 -07:00
Richard Newman
8adb6d97fd Add validation for or-join. r=nalexander 2017-03-27 16:32:45 -07:00
Richard Newman
0d15381e11 Crudely parse or and or-join. (#388) r=nalexander 2017-03-27 16:32:01 -07:00
Richard Newman
97749833d0 Algebrize and translate numeric constraints. (#306) r=nalexander 2017-03-22 10:19:47 -07:00
Richard Newman
d83c8620cd Implement parsing of query predicates. (#380) r=nalexander 2017-03-22 10:19:44 -07:00
Richard Newman
70b112801c Implement projection and querying. (#353) r=nalexander
* Add a failing test for EDN parsing '…'.
* Expose a SQLValueType trait to get value_type_tag values out of a ValueType.
* Add accessors to FindSpec.
* Implement querying.
* Implement rudimentary projection.
* Export mentat_db::new_connection.
* Export symbols from mentat.
* Add rudimentary end-to-end query tests.
2017-03-06 14:40:10 -08:00
Richard Newman
b0120aa446 Pre: fix query/Cargo.toml indenting. 2017-02-23 18:31:57 -08:00
Joe Walker
60b2e6f885 Improve Debug output for ConjoiningClauses; r=rnewman
Fixes #317, and also removes the '::' exploration of rust style
2017-02-23 14:05:56 +00:00
Richard Newman
a10f68fdb7 Mark every project as being part of the workspace. r=nalexander
This allows `cargo test --all` to work.
2017-02-20 11:04:08 -08:00
Richard Newman
5af7082165 Partly flesh out query algebrizer. (#243) r=nalexander 2017-02-15 16:10:59 -08:00
Richard Newman
fc73bfce75 Implement NonIntegerConstant::into_typed_value. r=jsantell 2017-02-09 18:45:46 -08:00
Richard Newman
1aa33423dc Derive PartialOrd and Ord for Variable. r=jsantell 2017-02-09 18:45:36 -08:00
Richard Newman
1122960462 Re-export NamespacedKeyword and PlainSymbol out of query. r=jsantell 2017-02-09 18:45:30 -08:00
Victor Porof
c585715224 Don't depend on num and ordered-float in the db and query crates, r=ncalexan (#223)
Signed-off-by: Victor Porof <victor.porof@gmail.com>
2017-02-08 12:19:16 +01:00
Richard Newman
5b770a54cd Parse basic :find and :where clauses. (#211) r=nalexander
* Make Variable::from_symbol public.
* Implement basic parsing of queries.
* Use pinned dependencies the hard way to fix Travis.
* Bump ordered-float dependency to 0.4.0.
* Error coercions to use ?, and finishing the find interface.
2017-02-02 18:32:00 -08:00