This is a big deck-chair re-arrangement. This puts FindQuery into
query-algebrizer and puts the validation from ParsedFindQuery ->
FindQuery their as well.
Some tests were re-homed for this.
In addition, the little-used maplit crate dependency was replaced with
inline expressions.
* Part 3: Parameterize Entity by value type.
This isn't quite right, because after parsing, we shouldn't care
about` `edn::ValueAndSpan`, we should care only about edn::Value.
However, I think we can drop `ValueAndSpan` entirely if we just use
`rust-peg` (and its simpler error messages) rather than a mix of
`rust-peg` and `combine`.
In any case, this paves the way to transacting `Entity<TypedValue>`,
which is a nice step towards building general entities.
* Part 1: Add AttributePlace.
* Part 2: Name other places EntityPlace and ValuePlace.
Now we're consistent and closer to self-documenting. Both matter more
as we expose `Entity` as the thing to build for programmatic usage.
* Part 4: Allow Ident and TempId in ValuePlace.
The parser will never produce these, since determining whether an
integer/keyword or string is an ident or a tempid, respectively, in
the value place requires the schema.
But a builder that produces `Entity` instances directly will want to
produce these.
This should address #663, by re-inserting type checking in the
transactor stack after the entry point used by the term builder.
Before this commit, we were using an SQLite UNIQUE index to assert
that no `[e a]` pair, with `a` a cardinality one attribute, was
asserted more than once. However, that's not in line with Datomic,
which treats transaction inputs as a set and allows a single datom
like `[e a v]` to appear multiple times. It's both awkward and not
particularly efficient to look for _distinct_ repetitions in SQL, so
we accept some runtime cost in order to check for repetitions in the
transactor. This will allow us to address #532, which is really about
whether we treat inputs as sets. A side benefit is that we can
provide more helpful error messages when the transactor does detect
that the input truly violates the cardinality constraints of the
schema.
This commit builds a trie while error checking and collecting final
terms, which should be fairly efficient. It also allows a simpler
expression of input-provided :db/txInstant datoms, which in turn
uncovered a small issue with the transaction watcher, where-by the
watcher would not see non-input-provided :db/txInstant datoms.
This transition to Datomic-like input-as-set semantics allows us to
address #532. Previously, two tempids that upserted to the same entid
would produce duplicate datoms, and that would have been rejected by
the transactor -- correctly, since we did not allow duplicate datoms
under the input-as-list semantics. With input-as-set semantics,
duplicate datoms are allowed; and that means that we must allow
tempids to be equivalent, i.e., to resolve to the same tempid.
To achieve this, we:
- index the set of tempids
- identify tempid indices that share an upsert
- map tempids to a dense set of contiguous integer labels
We use the well-known union-find algorithm, as implemented by
petgraph, to efficiently manage the set of equivalent tempids.
Along the way, I've fixed and added tests for two small errors in the
transactor. First, don't drop datoms resolved by upsert (#679).
Second, ensure that complex upserts are allocated.
I don't know quite what happened here. The Clojure implementation
correctly kept complex upserts that hadn't resolved as complex
upserts (see
9a9dfb502a/src/common/datomish/transact.cljc (L436))
and then allocated complex upserts if they didn't resolve (see
9a9dfb502a/src/common/datomish/transact.cljc (L509)).
Based on the code comments, I think the Rust implementation must have
incorrectly tried to optimize by handling all complex upserts in at
most a single generation of evolution, and that's just not correct.
We're effectively implementing a topological sort, using very specific
domain knowledge, and its not true that a node in a topological sort
can be considered only once!
Documents the FFI layer for Mentat, and provides transaction functionality via an EDN string. Creates two native libraries for iOS (Swift) and Android (Java) and fully tests the FFI for both platforms.
Closes#619#614#611
* Make properties on NamespacedKeyword/NamespacedSymbol private
* Use only a single String for NamespacedKeyword/NamespacedSymbol
* Review comments.
* Remove unsafe code in namespaced_name.
Benchmarking shows approximately zero change.
* Allow the types of ns and name to differ when constructing a NamespacedName.
* Make symbol namespaces optional.
* Normalize names of keyword/symbol constructors.
This will make the subsequent refactor much less painful.
* Use expect not unwrap.
* Merge Keyword and NamespacedKeyword.
There are few reasons to do this:
- it's difficult to add symbol interning to combine-based parsers like
tx-parser -- literally every type changes to reflect the interner,
and that means every convenience macro we've built needs to chagne.
It's trivial to add interning to rust-peg-based parsers.
- combine has rolled forward to 3.2, and I spent a similar amount of
time investigating how to upgrade tx-parser (to take advantage of
the new parser! macros in combine that I think are necessary for
adapting to changing types) as I did just converting to rust-peg.
- it's easy to improve the error messages in rust-peg, where-as I have
tried twice to improve the nested error messages in combine and am
stumped.
- it's roughly 4x faster to parse strings directly as opposed to
edn::ValueAndSpan, and it'll be even better when we intern directly.
* Refactor AttributeCache populator code for use from pull.
* Pre: add to_value_rc to Cloned.
* Pre: add From<StructuredMap> for Binding.
* Pre: clarify Store::open_empty.
* Pre: StructuredMap cleanup.
* Pre: clean up a doc test.
* Split projector crate. Pass schema to projector.
* CLI support for printing bindings.
* Add and use ConjoiningClauses::derive_types_from_find_spec.
* Define pull types.
* Implement pull on top of the attribute cache layer.
* Add pull support to the projector.
* Parse pull expressions.
* Add simple pull support to connection objects.
* Tests for pull.
* Compile with Rust 1.25.
The only choice involved in this commit is that of replacing the
anonymous lifetime '_ with a named lifetime for the cache; since we're
accepting a Known, which includes the cache in question, I think it's
clear that we expect the function to apply to any given cache
lifetime.
* Review comments.
* Bail on unnamed attribute.
* Make assert_parse_failure_contains safe to use.
* Rework query parser to report better errors for pull.
* Test for mixed wildcard and simple attribute.
This innocuous looking change (upserts_ev -> upserts_e -> resolved in
all situations, rather than upserts_ev -> resolved in some situations)
is a significant change in semantics and assumptions in the
transactor. Witness the large comment being removed about the same
tempid resolving in different generations!
To support this change, we provide more holistic errors for
conflicting upserts, which entails collecting some (relatively
expensive) diagnostic data.
I left in some debug logging, simply since it shouldn't hurt in
general, and will likely be useful for the next bug we see in the
transactor.
* Pre: clean up core/src/lib.rs.
* Pre: use indexmap 1.0 in db and query-projector.
* Change rel results to be a RelResult instance, not a Vec<Vec<TypedValue>>.
This avoids memory fragmentation and improves locality by using a single
heap-allocated vector for all bindings, rather than a separate
heap-allocated vector for each row.
We hide this abstraction behind the `RelResult` type, which tracks the
stride length (width) of each row.
* Don't allocate temporary vectors when projecting RelResults.
Some parts of the query engine and transactor need to know whether an
attribute is a component attribute, and sometimes want to do so in
a generated SQL query. This is one way to do that.
* Tidy up and add txid at beginning of transaction
* Add ffi crate and new_store function
* Add register and unregister observer FFI, Store and Conn functions.
Also add android logging facilities
* Add function for fetching entids for attribute strings
* Add functions for iterating through TxReports
* Add sync to ffi boundary
* Move Extern types from submodule to lib in FFI.
For some reason, if these types are in a submodule, even if they are publically used, the functions inside the FFI are not found in
Android. Works for iOS though. To be investigated later....
* Return to passing TxReports to observer function.
Also, remove some debug
* Expose DateTime and Utc publically
* Use Store in observer tests
Simplify.
This has a watcher collect txid -> AttributeSet mappings each time a
transact occurs. On commit we retrieve those mappings and hand them over
to the observer service, which filters them and packages them up for
dispatch.
Tidy up
* Pre: use debugcli in VSCode.
* Pre: wrap subqueries in parentheses in output SQL.
* Pre: add ExistingColumn.
This lets us make reference to columns by name, rather than only
pointing to qualified aliases.
* Pre: add Into for &str to TypedValue.
* Pre: add Store.transact.
* Pre: cleanup.
* Parse and algebrize simple aggregates. (#312)
* Follow-up: print aggregate columns more neatly in the CLI.
* Useful ValueTypeSet helpers.
* Allow for entity inequalities.
* Add 'differ', which is a ref-specialized not-equals.
* Add 'unpermute', a function for getting unique, distinct pairs from bindings.
* Review comments.
* Add 'the' pseudo-aggregation operator.
This allows for a corresponding value to be returned when a query
includes one 'min' or 'max' aggregate.
* Use the cache to make constant queries super fast.
* Fix translate tests to match: we no longer generate SQL for many of them!
* Accumulate additions and removals into the cache.
* Make attribute cache clone-on-write; store it in Metadata.
* Allow caching of fulltext attributes, interning strings.
This puts caching in mentat_db, adds a reverse lookup capability for
unique attributes, and populates bidirectional caches with a single
SQL cursor walk.
Differentiate between begin_read and begin_uncached_read.
Note that we still allow toggling within InProgress, because there might be
transient local state that makes starting a new transaction impossible.
* Add EntityBuilder.add_kw.
This allows you to skip your own attribute lookups, at the cost of
potentially doing the work more than once.
Also does value type checking.
* Nit: Alphabetical ordering of imports
* Create Cache and provide functions for calling it
* Get tests working. Move to using NamespacedKeyword over KnownEntid in function signature
* Add is_cached check to caching tests
* Move lazy and add/remove boolean flags to enums
* Move function definitions into generic trait and implement trait for AttributeCache
* Remove lazy cache and generalize cache
* Update tests
* Eager cache becomes simple key value store. AttributeMap handles attribute storing specifics
* Update tests to test presence of correct values in cache
* Move EagerCache, AttributeValueProvider and ValueProvider into mentat_db
* Add test for get_for_entid
* Add test for lookup attribute
* Make caches cloneable. Add value_for alongside values_for
* Use cache in attribute lookups
* Split test for values and value and add cardinality
* address review feedback r=rnewman
* Define Store, which is a simple container for a SQLite connection and a Conn.
This is a breaking change.
* Return the FindSpec as part of QueryOutput, not just results.
* Switch to using stderr in appropriate places in CLI.
* Print columns in CLI output.
* Add TypedValue::instant(micros).
* Add From<f64> for TypedValue.
* Add lookup_values_for_attribute to Conn.
* Add q_explain to Queryable.
* Expose an iterator over FindSpec's columns.
* Export edn from mentat crate. Export QueryExecutionResult.
* Implement Display for Variable and Element.
* Introduce a `kw` macro.
This allows you to write:
```rust
kw!(:foo/bar)
```
instead of
```rust
NamespacedKeyword::new("foo", "bar")
```
… and it's more efficient, too.
Add `mentat::open`, eliminate use of `mentat_db` in some places.
This includes two other changes:
* Split transact to expose an interface for TermWithTempIds.
* Return TxReport from each InProgress operation, not from commit.
Improve naming of read-only transactions.
Implement entid_for_type.
Simplify get_attribute.
Name ignored var in algebrizer.
Comment attribute_for_ident.
Make KnownEntid a core concept.
Expose lookup_value_for_attribute.
Implement HasSchema and a new query encapsulation on Conn.
Pre: export Queryable.
