Commit graph

319 commits

Author SHA1 Message Date
Grisha Kruglov
e9398dd50d Part 1: Move public errors into public-traits 2018-08-09 13:16:05 -07:00
Grisha Kruglov
c00e14f5ff Pre: Remove :: dependency from src/errors.rs 2018-08-09 13:16:05 -07:00
Grisha Kruglov
c8e6a511f4 Pre: Move tolstoy/errors into tolstoy-traits 2018-08-09 13:16:05 -07:00
Grisha Kruglov
9381af4289 Pre: Move core/Attribute* to core-traits 2018-08-09 13:16:05 -07:00
Grisha Kruglov
68d0e17824 Pre: Move sql/errors into sql_traits 2018-08-09 13:16:05 -07:00
Grisha Kruglov
05ef149545 Pre: Fold query-translator into query-projector 2018-08-09 13:16:05 -07:00
Grisha Kruglov
6312e89aba Pre: Move query-projectors/errors and aggregates into query-projector-traits 2018-08-09 13:16:05 -07:00
Grisha Kruglov
ccdd17551a Pre: Move query-algebrizer/error.rs into query-algebrizer-traits 2018-08-09 13:16:05 -07:00
Grisha Kruglov
07beb68c7a Pre: Remove query/ crate 2018-08-09 13:16:05 -07:00
Grisha Kruglov
11aaa193f5 Pre: Move query-pull/errors into query-pull-traits 2018-08-09 13:16:05 -07:00
Grisha Kruglov
cebb85a7fe Pre: Move db/errors.rs into db_traits 2018-08-09 13:16:05 -07:00
Grisha Kruglov
d0214fad7d Pre: Move core/types.rs into core_traits 2018-08-09 13:16:05 -07:00
Grisha Kruglov
a57ba5d79f Pre: Move Entid and KnownEntid into core_traits 2018-08-09 13:16:05 -07:00
Nick Alexander
9e6505a930 [sdks/android] Pre: Fix unused warnings. 2018-07-25 20:38:48 -07:00
Grisha Kruglov
6290cc9db2 Enforce partition integrity when setting its index r=nalexander
Timelines work starts to perform modifications on the partitions
that go beyond simple allocations. This change pre-emptively protects
partition integrity by asserting that index modifications are legal.
2018-07-17 06:20:37 -07:00
Grisha Kruglov
bff24c60b7
Add a top-level "syncable" feature. (#782) r=ncalexan
* Add a top-level "syncable" feature.

Tested with:

cargo test --all
cargo test --all --no-default-features
cargo build --manifest-path tools/cli/Cargo.toml --no-default-features
cargo run --manifest-path tools/cli/Cargo.toml --no-default-features debugcli

Co-authored-by: Nick Alexander <nalexander@mozilla.com>

* Add 'syncable' feature to 'db' crate to conditionally derive serialization for Partition*

This is leading up to syncing with partition support.
2018-07-11 16:26:06 -07:00
Nick Alexander
f65512158b Make kw!(foo.bar/bar.baz) work. 2018-07-06 14:19:50 -07:00
Nick Alexander
2cb7d441dc Part 2: Make it easier to match tuple results.
Right now, we write code like
```rust
match q_once(q, inputs)?.into_tuple()? {
    Some(vs) => match (vs.len(), vs.get(0), vs.get(1)) {
        (2, &Some(Binding::Scalar(TypedValue::Long(a))), &Some(Binding::Scalar(TypedValue::Instant(ref b)))) => Some((a, b.clone())),
        _ => panic!(),
    },
    None => None,
}
```
to length-check tuples coming out of the database.  It can also lead
to a lot of cloning because references are the easiest thing to hand.

This commit allows to write code like
```rust
match q_once(q, inputs)?.into_tuple()? {
    Some((Binding::Scalar(TypedValue::Long(a)), Binding::Scalar(TypedValue::Instant(b)))) => Some((a, b)),
    Some(_) => panic!(),
    None => None,
}
```
which is generally much easier to reason about.
2018-07-05 16:45:42 -07:00
Nick Alexander
2ab481f83e Part 2: Expose time related things at top-level.
Perhaps we actually want to subdivide the top-level namespace so that
there is a `mentat::time` module, but I'd prefer to make part of the
process of fixing the public API as we get ready to christen version
1.0.
2018-07-05 16:45:42 -07:00
Nick Alexander
3744982cd9 Add last_tx_id. 2018-07-05 16:45:42 -07:00
Nick Alexander
d49f702512 Part 1: Expand Binding::val() into Binding::{into_*,as_*}.
This is simply for completeness: we should provide fundamental
conversion patterns even when they are mostly unused in our code base.
2018-07-05 16:45:42 -07:00
Nick Alexander
eb1df31ac4 Part 7: Improve TermBuilder interface; expose lookup refs and tx functions.
These are functions on `TermBuilder` itself to prevent mixing mutable
and immutable references in the most natural style.  That is,
```
builder.add(e, a, builder.lookup_ref(...))
```
fails because `add` borrows `builder` mutably and `lookup_ref` borrows
`builder` immutably.  There's nothing here that requires a specific
builder (since we're not interning lookup refs on the builder, like we
are tempids) so we don't need an instance.
2018-07-05 16:33:51 -07:00
Nick Alexander
06056a8468 Part 6: Lift TxReport to core crate.
The `core` create didn't exist when the `db` was started, but this
type is clearly part of the public interface of Mentat.
2018-07-05 16:33:51 -07:00
Nick Alexander
1cb1847aa6 Part 5: Make existing TermBuilder actually build Entity instances.
There are a few tricky details to call out here.  The first is the
`TransactableValueMarker` trait.  This is strictly a marker (like
`Sized`, for example) to give some control over what types can be used
as value types in `Entity` instances.  This expression is needed due
to the network of `Into` and `From` relations between the parts of
valid `Entity` instances.  This allows to drop the `IntoThing`
work-around trait and use the established patterns.  (Observe that
`KnownEntid` makes this a little harder, due to the cross-crate
consistency restrictions.)

The second is that we can get rid `{add,retract}_kw`, since the
network of relations expresses the coercions directly.

The third is that this commit doesn't change the name `TermBuilder`,
even though it is now building `Entity` instances.  This is because
there's _already_ an `EntityBuilder` which fixes the `EntityPlace`.
It's not clear whether the existing entity building interface should
be removed or whether both should be renamed.  That can be follow-up.
2018-07-05 16:33:51 -07:00
Nick Alexander
87f850a44e Part 1: Move intern_set into edn crate.
It's not great to keep lifting functionality higher and higher up the
crate hierarchy, but we really do want to intern while we parse.
Eventually, I expect that we will split the `edn` crate into `types`
and `parsing`, and the `types` crate can depend on a more efficient
interning dependency.
2018-07-05 11:16:48 -07:00
Nick Alexander
d82c7f8ef2 Cull unused mentat_parser_utils crate.
With the transition toward parsing with `rust-peg` and away from
`combine`, we're not using some of the many helpers we built to
support our unusual `combine` usage.  They can just go!
2018-06-30 16:21:50 -07:00
Nick Alexander
8725bad18c Pre: Fix error printing rusqlite::Error. 2018-06-30 14:58:23 -07:00
Grisha Kruglov
8af5288a60 Use TolstoyError for tolstoy's Results; wrap tolstoy's dependency errors r=nalexander
This is inline with the rest of mentat, and helps with upcoming tolstoy work.
2018-06-29 00:47:19 -04:00
Nick Alexander
ae427849d5 Expose sub-crate *Error types at top-level.
We're not exposing a uniform API with `mentat::Result` yet, meaning
that early consumers (e.g., the logins work for Mozilla Lockbox) need
to wrap errors from all over the Mentat crate hierarchy.
2018-06-27 15:05:43 -07:00
Nick Alexander
d31ec28aa8 Patch it all together: use MentatError at top-level.
I elected to keep Tolstoy using `failure::Error`, because Tolstoy
looks rather more like a high-level application (and will continue to
do so for a while) than a production-ready mid- or low-level API.
2018-06-27 15:05:43 -07:00
Nick Alexander
0e4991fa26 Make db/ use DbErrorKind. 2018-06-27 15:05:43 -07:00
Thom
72a9b302f9
Rename or delete things so that there is only one type named Entid (#768)
* Delete the (apparently unused) EntId

* Rename edn's Entid to EntidOrIdent to avoid confusion with the Entid that's actually an i64

* Fix travis beta bustage (This is actually unrelated to entids, but is a trivial fix nonetheless)
2018-06-26 16:34:18 -07:00
Grisha Kruglov
4e46adeba1 Convert tolstoy/ to failure. 2018-06-20 14:42:36 -07:00
Grisha Kruglov
4e01929334 Convert src/ to failure. 2018-06-20 14:42:18 -07:00
Thom Chiovoloni
99a73ccb03 Avoid using 1.26.0-only features when using sqlcipher, and move the sqlcipher Store support to the correct file 2018-06-13 22:48:28 -07:00
Emily Toop
8e918949fb Separate Store from Conn.
This is a Pre: part extracted from #660.
2018-06-13 15:29:11 -07:00
Thom
6a1a265894
Add support for using sqlcipher (#737). Fixes #118 2018-06-13 08:49:40 -07:00
Nick Alexander
e68cc4016c Part 7: Remove tx entirely.
This was left over from #681.
2018-06-04 15:04:39 -07:00
Nick Alexander
d4166cc67c Part 6: Remove query-parser entirely. 2018-06-04 15:04:39 -07:00
Nick Alexander
47441f56dc Part 5: Push FindQuery into query-algebrizer; structure errors.
This is a big deck-chair re-arrangement.  This puts FindQuery into
query-algebrizer and puts the validation from ParsedFindQuery ->
FindQuery their as well.

Some tests were re-homed for this.

In addition, the little-used maplit crate dependency was replaced with
inline expressions.
2018-06-04 15:04:39 -07:00
Nick Alexander
47a0f40cce Pre: Fix warnings. 2018-06-04 14:52:51 -07:00
Richard Newman
b2e98f44f6
Generalize Entity by value type. (#701) (#691) r=rnewman
* Part 3: Parameterize Entity by value type.

This isn't quite right, because after parsing, we shouldn't care
about` `edn::ValueAndSpan`, we should care only about edn::Value.
However, I think we can drop `ValueAndSpan` entirely if we just use
`rust-peg` (and its simpler error messages) rather than a mix of
`rust-peg` and `combine`.

In any case, this paves the way to transacting `Entity<TypedValue>`,
which is a nice step towards building general entities.

* Part 1: Add AttributePlace.

* Part 2: Name other places EntityPlace and ValuePlace.

Now we're consistent and closer to self-documenting.  Both matter more
as we expose `Entity` as the thing to build for programmatic usage.

* Part 4: Allow Ident and TempId in ValuePlace.

The parser will never produce these, since determining whether an
integer/keyword or string is an ident or a tempid, respectively, in
the value place requires the schema.

But a builder that produces `Entity` instances directly will want to
produce these.
2018-05-15 00:43:07 -07:00
Nick Alexander
46c2a0801f Add type checking and constraint checking to the transactor. (#663, #532, #679)
This should address #663, by re-inserting type checking in the
transactor stack after the entry point used by the term builder.

Before this commit, we were using an SQLite UNIQUE index to assert
that no `[e a]` pair, with `a` a cardinality one attribute, was
asserted more than once.  However, that's not in line with Datomic,
which treats transaction inputs as a set and allows a single datom
like `[e a v]` to appear multiple times.  It's both awkward and not
particularly efficient to look for _distinct_ repetitions in SQL, so
we accept some runtime cost in order to check for repetitions in the
transactor.  This will allow us to address #532, which is really about
whether we treat inputs as sets.  A side benefit is that we can
provide more helpful error messages when the transactor does detect
that the input truly violates the cardinality constraints of the
schema.

This commit builds a trie while error checking and collecting final
terms, which should be fairly efficient.  It also allows a simpler
expression of input-provided :db/txInstant datoms, which in turn
uncovered a small issue with the transaction watcher, where-by the
watcher would not see non-input-provided :db/txInstant datoms.

This transition to Datomic-like input-as-set semantics allows us to
address #532.  Previously, two tempids that upserted to the same entid
would produce duplicate datoms, and that would have been rejected by
the transactor -- correctly, since we did not allow duplicate datoms
under the input-as-list semantics.  With input-as-set semantics,
duplicate datoms are allowed; and that means that we must allow
tempids to be equivalent, i.e., to resolve to the same tempid.

To achieve this, we:
- index the set of tempids
- identify tempid indices that share an upsert
- map tempids to a dense set of contiguous integer labels

We use the well-known union-find algorithm, as implemented by
petgraph, to efficiently manage the set of equivalent tempids.

Along the way, I've fixed and added tests for two small errors in the
transactor.  First, don't drop datoms resolved by upsert (#679).
Second, ensure that complex upserts are allocated.

I don't know quite what happened here.  The Clojure implementation
correctly kept complex upserts that hadn't resolved as complex
upserts (see
9a9dfb502a/src/common/datomish/transact.cljc (L436))
and then allocated complex upserts if they didn't resolve (see
9a9dfb502a/src/common/datomish/transact.cljc (L509)).

Based on the code comments, I think the Rust implementation must have
incorrectly tried to optimize by handling all complex upserts in at
most a single generation of evolution, and that's just not correct.
We're effectively implementing a topological sort, using very specific
domain knowledge, and its not true that a node in a topological sort
can be considered only once!
2018-05-14 15:22:45 -07:00
Emily Toop
013629dec6
iOS and Android (Java) sdk framework (#643)
Documents the FFI layer for Mentat, and provides transaction functionality via an EDN string. Creates two native libraries for iOS (Swift) and Android (Java) and fully tests the FFI for both platforms.

Closes #619 #614 #611
2018-05-14 16:20:36 +01:00
Richard Newman
3dc68bcd38 Combine NamespacedKeyword and Keyword. (#689) r=nalexander
* Make properties on NamespacedKeyword/NamespacedSymbol private

* Use only a single String for NamespacedKeyword/NamespacedSymbol

* Review comments.

* Remove unsafe code in namespaced_name.

Benchmarking shows approximately zero change.

* Allow the types of ns and name to differ when constructing a NamespacedName.

* Make symbol namespaces optional.

* Normalize names of keyword/symbol constructors.

This will make the subsequent refactor much less painful.

* Use expect not unwrap.

* Merge Keyword and NamespacedKeyword.
2018-05-11 09:52:17 -07:00
Nick Alexander
cbffe5e545 Use rust-peg for tx parsing.
There are few reasons to do this:

- it's difficult to add symbol interning to combine-based parsers like
  tx-parser -- literally every type changes to reflect the interner,
  and that means every convenience macro we've built needs to chagne.
  It's trivial to add interning to rust-peg-based parsers.

- combine has rolled forward to 3.2, and I spent a similar amount of
  time investigating how to upgrade tx-parser (to take advantage of
  the new parser! macros in combine that I think are necessary for
  adapting to changing types) as I did just converting to rust-peg.

- it's easy to improve the error messages in rust-peg, where-as I have
  tried twice to improve the nested error messages in combine and am
  stumped.

- it's roughly 4x faster to parse strings directly as opposed to
  edn::ValueAndSpan, and it'll be even better when we intern directly.
2018-05-10 10:24:05 -07:00
Richard Newman
e21156a754
Implement simple pull expressions (#638) r=nalexander
* Refactor AttributeCache populator code for use from pull.

* Pre: add to_value_rc to Cloned.

* Pre: add From<StructuredMap> for Binding.

* Pre: clarify Store::open_empty.

* Pre: StructuredMap cleanup.

* Pre: clean up a doc test.

* Split projector crate. Pass schema to projector.

* CLI support for printing bindings.

* Add and use ConjoiningClauses::derive_types_from_find_spec.

* Define pull types.

* Implement pull on top of the attribute cache layer.

* Add pull support to the projector.

* Parse pull expressions.

* Add simple pull support to connection objects.

* Tests for pull.

* Compile with Rust 1.25.

The only choice involved in this commit is that of replacing the
anonymous lifetime '_ with a named lifetime for the cache; since we're
accepting a Known, which includes the cache in question, I think it's
clear that we expect the function to apply to any given cache
lifetime.

* Review comments.

* Bail on unnamed attribute.

* Make assert_parse_failure_contains safe to use.

* Rework query parser to report better errors for pull.

* Test for mixed wildcard and simple attribute.
2018-05-04 12:56:00 -07:00
Nick Alexander
2b82ffb2e5 [tx] Fail transactions where complex upserts resolve to multiple entids. (#670)
This innocuous looking change (upserts_ev -> upserts_e -> resolved in
all situations, rather than upserts_ev -> resolved in some situations)
is a significant change in semantics and assumptions in the
transactor.  Witness the large comment being removed about the same
tempid resolving in different generations!

To support this change, we provide more holistic errors for
conflicting upserts, which entails collecting some (relatively
expensive) diagnostic data.

I left in some debug logging, simply since it shouldn't hurt in
general, and will likely be useful for the next bug we see in the
transactor.
2018-05-01 15:34:44 -07:00
Richard Newman
a2e13f624c
Add 'Binding', a structured value type to return from queries. (#657) r=nalexander
Bump to 0.7: breaking change.
2018-04-24 15:08:38 -07:00
Richard Newman
a74a2deffc
Introduce RelResult rather than Vec<Vec<TypedValue>>. (#639) r=nalexander
* Pre: clean up core/src/lib.rs.
* Pre: use indexmap 1.0 in db and query-projector.
* Change rel results to be a RelResult instance, not a Vec<Vec<TypedValue>>.

This avoids memory fragmentation and improves locality by using a single
heap-allocated vector for all bindings, rather than a separate
heap-allocated vector for each row.

We hide this abstraction behind the `RelResult` type, which tracks the
stride length (width) of each row.

* Don't allocate temporary vectors when projecting RelResults.
2018-04-24 15:04:00 -07:00