* Pre: remove remnants of 'open_empty'
* Pre: Cleanup 'datoms' table after a timeline move
Since timeline move operations use a transactor, they generate a
"phantom" 'tx' and a 'txInstant' assertion. It is "phantom" in a sense
that it was never present in the 'transactions' table, and is entirely
synthetic as far as our database is concerned.
It's an implementational artifact, and we were not cleaning it up.
It becomes a problem when we start inserting transactions after a move.
Once the transactor clashes with the phantom 'tx', it will retract the
phantom 'txInstant' value, leaving the transactions log in an incorrect state.
This patch adds a test for this scenario and elects the easy way out: simply
remove the offending 'txInstant' datom.
* Part 1: Sync without support for side-effects
A "side-effect" is defined here as a mutation of a remote state as part
of the sync.
If, during a sync we determine that a remote state needs to be changed, bail out.
This generally supports different variations of "baton-passing" syncing, where clients
will succeed syncing if each change is non-conflicting.
* Part 2: Support basic "side-effects" syncing
This patch introduces a concept of a follow-up sync. If a sync generated
a "merge transaction" (a regular transaction that contains assertions
necessary for local and remote transaction logs to converge), then
this transaction needs to be uploaded in a follow-up sync.
Generated SyncReport indicates if a follow-up sync is required.
Follow-up sync itself is just a regular sync. If remote state did not change,
it will result in a simple RemoteFastForward. Otherwise, we'll continue
merging and requesting a follow-up.
Schema alterations are explicitly not supported.
As local transactions are rebased on top of remote, following changes happen:
- entids are changed into tempids, letting transactor upsert :db/unique values
- entids for retractions are changed into lookup-refs if we're confident they'll succeed
-- otherwise, retractions are dropped on the floor
* Post: use a macro for more readable tests
* Tolstoy README
Sync needs to operate over a "mentat transaction", not just a "db transaction".
This shuffle allows internal mentat crates to consume InProgress, which models
the concept of a "mentat transaction".
* Add a top-level "syncable" feature.
Tested with:
cargo test --all
cargo test --all --no-default-features
cargo build --manifest-path tools/cli/Cargo.toml --no-default-features
cargo run --manifest-path tools/cli/Cargo.toml --no-default-features debugcli
Co-authored-by: Nick Alexander <nalexander@mozilla.com>
* Add 'syncable' feature to 'db' crate to conditionally derive serialization for Partition*
This is leading up to syncing with partition support.
Right now, we write code like
```rust
match q_once(q, inputs)?.into_tuple()? {
Some(vs) => match (vs.len(), vs.get(0), vs.get(1)) {
(2, &Some(Binding::Scalar(TypedValue::Long(a))), &Some(Binding::Scalar(TypedValue::Instant(ref b)))) => Some((a, b.clone())),
_ => panic!(),
},
None => None,
}
```
to length-check tuples coming out of the database. It can also lead
to a lot of cloning because references are the easiest thing to hand.
This commit allows to write code like
```rust
match q_once(q, inputs)?.into_tuple()? {
Some((Binding::Scalar(TypedValue::Long(a)), Binding::Scalar(TypedValue::Instant(b)))) => Some((a, b)),
Some(_) => panic!(),
None => None,
}
```
which is generally much easier to reason about.
Perhaps we actually want to subdivide the top-level namespace so that
there is a `mentat::time` module, but I'd prefer to make part of the
process of fixing the public API as we get ready to christen version
1.0.
We're not exposing a uniform API with `mentat::Result` yet, meaning
that early consumers (e.g., the logins work for Mozilla Lockbox) need
to wrap errors from all over the Mentat crate hierarchy.
I elected to keep Tolstoy using `failure::Error`, because Tolstoy
looks rather more like a high-level application (and will continue to
do so for a while) than a production-ready mid- or low-level API.
* Delete the (apparently unused) EntId
* Rename edn's Entid to EntidOrIdent to avoid confusion with the Entid that's actually an i64
* Fix travis beta bustage (This is actually unrelated to entids, but is a trivial fix nonetheless)
Documents the FFI layer for Mentat, and provides transaction functionality via an EDN string. Creates two native libraries for iOS (Swift) and Android (Java) and fully tests the FFI for both platforms.
Closes#619#614#611
* Make properties on NamespacedKeyword/NamespacedSymbol private
* Use only a single String for NamespacedKeyword/NamespacedSymbol
* Review comments.
* Remove unsafe code in namespaced_name.
Benchmarking shows approximately zero change.
* Allow the types of ns and name to differ when constructing a NamespacedName.
* Make symbol namespaces optional.
* Normalize names of keyword/symbol constructors.
This will make the subsequent refactor much less painful.
* Use expect not unwrap.
* Merge Keyword and NamespacedKeyword.
There are few reasons to do this:
- it's difficult to add symbol interning to combine-based parsers like
tx-parser -- literally every type changes to reflect the interner,
and that means every convenience macro we've built needs to chagne.
It's trivial to add interning to rust-peg-based parsers.
- combine has rolled forward to 3.2, and I spent a similar amount of
time investigating how to upgrade tx-parser (to take advantage of
the new parser! macros in combine that I think are necessary for
adapting to changing types) as I did just converting to rust-peg.
- it's easy to improve the error messages in rust-peg, where-as I have
tried twice to improve the nested error messages in combine and am
stumped.
- it's roughly 4x faster to parse strings directly as opposed to
edn::ValueAndSpan, and it'll be even better when we intern directly.
* Refactor AttributeCache populator code for use from pull.
* Pre: add to_value_rc to Cloned.
* Pre: add From<StructuredMap> for Binding.
* Pre: clarify Store::open_empty.
* Pre: StructuredMap cleanup.
* Pre: clean up a doc test.
* Split projector crate. Pass schema to projector.
* CLI support for printing bindings.
* Add and use ConjoiningClauses::derive_types_from_find_spec.
* Define pull types.
* Implement pull on top of the attribute cache layer.
* Add pull support to the projector.
* Parse pull expressions.
* Add simple pull support to connection objects.
* Tests for pull.
* Compile with Rust 1.25.
The only choice involved in this commit is that of replacing the
anonymous lifetime '_ with a named lifetime for the cache; since we're
accepting a Known, which includes the cache in question, I think it's
clear that we expect the function to apply to any given cache
lifetime.
* Review comments.
* Bail on unnamed attribute.
* Make assert_parse_failure_contains safe to use.
* Rework query parser to report better errors for pull.
* Test for mixed wildcard and simple attribute.
* Pre: clean up core/src/lib.rs.
* Pre: use indexmap 1.0 in db and query-projector.
* Change rel results to be a RelResult instance, not a Vec<Vec<TypedValue>>.
This avoids memory fragmentation and improves locality by using a single
heap-allocated vector for all bindings, rather than a separate
heap-allocated vector for each row.
We hide this abstraction behind the `RelResult` type, which tracks the
stride length (width) of each row.
* Don't allocate temporary vectors when projecting RelResults.
* Tidy up and add txid at beginning of transaction
* Add ffi crate and new_store function
* Add register and unregister observer FFI, Store and Conn functions.
Also add android logging facilities
* Add function for fetching entids for attribute strings
* Add functions for iterating through TxReports
* Add sync to ffi boundary
* Move Extern types from submodule to lib in FFI.
For some reason, if these types are in a submodule, even if they are publically used, the functions inside the FFI are not found in
Android. Works for iOS though. To be investigated later....
* Return to passing TxReports to observer function.
Also, remove some debug
* Expose DateTime and Utc publically
* Use Store in observer tests
Simplify.
This has a watcher collect txid -> AttributeSet mappings each time a
transact occurs. On commit we retrieve those mappings and hand them over
to the observer service, which filters them and packages them up for
dispatch.
Tidy up
This puts caching in mentat_db, adds a reverse lookup capability for
unique attributes, and populates bidirectional caches with a single
SQL cursor walk.
Differentiate between begin_read and begin_uncached_read.
Note that we still allow toggling within InProgress, because there might be
transient local state that makes starting a new transaction impossible.
* Nit: Alphabetical ordering of imports
* Create Cache and provide functions for calling it
* Get tests working. Move to using NamespacedKeyword over KnownEntid in function signature
* Add is_cached check to caching tests
* Move lazy and add/remove boolean flags to enums
* Move function definitions into generic trait and implement trait for AttributeCache
* Remove lazy cache and generalize cache
* Update tests
* Eager cache becomes simple key value store. AttributeMap handles attribute storing specifics
* Update tests to test presence of correct values in cache
* Move EagerCache, AttributeValueProvider and ValueProvider into mentat_db
* Add test for get_for_entid
* Add test for lookup attribute
* Make caches cloneable. Add value_for alongside values_for
* Use cache in attribute lookups
* Split test for values and value and add cardinality
* address review feedback r=rnewman
* Define Store, which is a simple container for a SQLite connection and a Conn.
This is a breaking change.
* Return the FindSpec as part of QueryOutput, not just results.
* Switch to using stderr in appropriate places in CLI.
* Print columns in CLI output.
* Add TypedValue::instant(micros).
* Add From<f64> for TypedValue.
* Add lookup_values_for_attribute to Conn.
* Add q_explain to Queryable.
* Expose an iterator over FindSpec's columns.
* Export edn from mentat crate. Export QueryExecutionResult.
* Implement Display for Variable and Element.
* Introduce a `kw` macro.
This allows you to write:
```rust
kw!(:foo/bar)
```
instead of
```rust
NamespacedKeyword::new("foo", "bar")
```
… and it's more efficient, too.
Add `mentat::open`, eliminate use of `mentat_db` in some places.
This includes two other changes:
* Split transact to expose an interface for TermWithTempIds.
* Return TxReport from each InProgress operation, not from commit.
Improve naming of read-only transactions.
Implement entid_for_type.
Simplify get_attribute.
Name ignored var in algebrizer.
Comment attribute_for_ident.
Make KnownEntid a core concept.
Expose lookup_value_for_attribute.
Implement HasSchema and a new query encapsulation on Conn.
Pre: export Queryable.
Pre: export AttributeBuilder from mentat_db.
Pre: fix module-level comment for tx/src/entities.rs.
Pre: rename some `to_` conversions to `into_`.
Pre: make AttributeBuilder::unique less verbose.
Pre: split out a HasSchema trait to abstract over Schema.
Pre: rename SchemaMap/schema_map to AttributeMap/attribute_map.
Pre: TypedValue/NamespacedKeyword conversions.
Pre: turn Unique and ValueType into TypedValue::Keyword.
Pre: export IntoResult.
Pre: export NamespacedKeyword from mentat_core.
Pre: use intern_set in tx.
Pre: add InternSet::len.
Pre: comment gardening.
Pre: remove inaccurate TODO from TxReport comment.
