Commit graph

848 commits

Author SHA1 Message Date
Grisha Kruglov
536d40ad84 Part 4: Add support for moving transactions off of main timeline 2018-07-26 17:14:05 -07:00
Grisha Kruglov
4ec780c87a Part 3: Use a view to derive parts table
Being able to derive partition map from partition definitions and current
state of the world (transactions), segmented by timelines, is useful
because it lets us not worry about keeping materialized partition maps
up-to-date - since there's no need for materialized partition maps at that point.

This comes in very handy when we start moving chunks of transactions off of our mainline.
Alternative to this work would look like materializing partition maps per timeline,
growing support for incremental "backwards update" of the materialized maps, etc.

Our core partitions are defined in 'known_parts' table during bootstrap,
and what used to be 'parts' table is a generated view that operates over
transactions to figure out partition index.

'parts' is defined for the main timeline. Querying parts for other timelines
or for particular timeline+tx combinations will look similar.
2018-07-26 17:14:05 -07:00
Grisha Kruglov
3ca5255cde Part 2: Add basic support for timelines to the transactor
This records transactions onto a default timeline (0).
2018-07-26 17:14:05 -07:00
Grisha Kruglov
0974108a52 Part 1: Allow specifying transactor's commit behaviour
Normally we want to both materialize our changes (into 'datoms')
as well as commit source transactions into 'transactions' table.

However, when moving transactions from timeline to timeline
we don't want to persist artifacts (rewind assertions), just their
materializations.

This patch expands the 'db' interface to allow for this split,
and changes transactor's functions to take a crate-private 'action'
which defines desired behaviour.
2018-07-26 17:14:05 -07:00
Grisha Kruglov
5a29efa336 Part 0: Allow retractions of installed attributes
This is necessary for the timelines work ahead. When schema is being
moved off of a main timeline, we need to be able to retract it cleanly.

Retractions are only processed if the whole defining attribute set
is being retracted at once (:db/ident, :db/valueType, :db/cardinality).
2018-07-26 17:14:05 -07:00
Grisha Kruglov
9a47d8905f Pre: 'Into' implementation chaining TermWithoutTempIds -> TermWithTempIds -> TermWithTempIdsAndLookupRefs 2018-07-26 17:14:05 -07:00
Nick Alexander
e6066769ca Pre: Differentiate bad attribute retractions from unrecognized retractions. 2018-07-26 17:14:05 -07:00
Nick Alexander
fba378ee39 [sdks/android] Build Mentat Android SDK in TaskCluster; publish org.mozilla.mentat to nalexander's personal Bintray repo.
I haven't had this reviewed thoroughly, but it mostly works.
2018-07-26 13:12:20 -07:00
Nick Alexander
faef4e9ee8 Bump to version 0.10.0. 2018-07-26 13:09:18 -07:00
Nick Alexander
a8cc9cb70d [sdks/android] Don't strip Mentat library.
Help folks debugging by including symbols in our native libraries.
Yes, this makes the resulting AAR very large.  The Android ecosystem
seems to be in flux around who is in charge of stripping native
binaries, but for now let's provide symbols and see how consumers
react.
2018-07-26 13:06:03 -07:00
Nick Alexander
76d7df5548 [sdks/android] Package JNA using upstream dependency. 2018-07-26 13:05:31 -07:00
Nick Alexander
7e31ca15bc [sdks] Make store_open{_encrypted} return useful errors.
Because this was formerly a constructor, the pattern needed to change
to a factory function, but that's better than what we had.
2018-07-26 13:01:53 -07:00
Nick Alexander
67a14ca756 [sdks/android] Build Mentat Android SDK in TaskCluster; publish org.mozilla.mentat to nalexander's personal Bintray repo.
The automation parts were cribbed directly from
50add3e176.

The automation permissions were added in
https://bugzilla.mozilla.org/show_bug.cgi?id=1477311.

This uses a very rudimentary Gradle plugin, `rust-android-gradle`,
with custom fixes and extensions.  It works pretty well for what it
is!  See https://github.com/ncalexan/rust-android-gradle.
2018-07-25 20:50:44 -07:00
Nick Alexander
0955c784b7 [sdks/android] Pre: Trim unused Android bits.
We don't use UI libraries, don't require UI resources, and don't
require any permissions.
2018-07-25 20:38:56 -07:00
Nick Alexander
9e6505a930 [sdks/android] Pre: Fix unused warnings. 2018-07-25 20:38:48 -07:00
Victor Porof
89d8ac50a8
Run serde_support tests for the EDN module on CI (#792)
Signed-off-by: Victor Porof <victor.porof@gmail.com>
2018-07-19 19:03:19 +02:00
Victor Porof
2540404b00
Generate rust documentation on CI and publish to gh-pages automatically (#793)
Signed-off-by: Victor Porof <victor.porof@gmail.com>
2018-07-19 18:32:54 +02:00
Nick Alexander
22dad5d6ca [build] Include Gradle wrapper JAR in repository.
Presumably this was an error: `.gitignore` ignores all JAR files, but
this one really needs to be in version control.
2018-07-17 15:30:10 -07:00
Grisha Kruglov
69c9f512a0 Move entid allocation logic into Partition r=nalexander 2018-07-17 06:20:37 -07:00
Grisha Kruglov
6290cc9db2 Enforce partition integrity when setting its index r=nalexander
Timelines work starts to perform modifications on the partitions
that go beyond simple allocations. This change pre-emptively protects
partition integrity by asserting that index modifications are legal.
2018-07-17 06:20:37 -07:00
Nick Alexander
38a92229d7 Pre: Replace PartitionMapping trait with newtype. r=grisha
Generally, I think that Mentat is using too many small traits rather
than wrapping types into newtypes.  Wrapping into newtypes is cheap in
Rust, and it makes it easier to reason about the code.
2018-07-17 06:20:37 -07:00
Grisha Kruglov
675a865896
Extract and improve test macros (#787) r=nalexander
* Part 1: Extract low-level test framework into mentat_db::debug for re-use.

* Part 2: Improve assert_matches!.

This corrects an incorrect pattern: a conversion method taking &self
but returning an owned value should be named like `to_FOO(&self) -> FOO`.  (A
reference-to-reference conversion should be named like `as_FOO(&self)
-> &FOO`.  A consuming conversion should be named like `into_FOO(self)
-> FOO`.)

In addition, this pushes the conversion via `to_edn` into the
`assert_matches!` macro, which lets consumers get a real data
structure (say, `Datoms`) and use it directly before or after
`assert_matches!`.  (Currently, consumers get back `edn::Value`
instances, which aren't nearly as pleasant to use as real data
structures.)

Co-authored-by: Grisha Kruglov <gkruglov@mozilla.com>

* Part 3: Use mentat_db::debug framework in Tolstoy crate.

The advantage of this approach is that compiling Tolstoy (or anything
that's not db, really) can be quite a bit faster than compiling db.
2018-07-16 13:58:34 -07:00
Nick Alexander
e9cddd63e4 [tx] Don't treat :db/doc as defining a schema attribute. (#784) r=grisha 2018-07-16 12:20:34 -07:00
Nick Alexander
9291b2a0b0 [tx] Don't treat :db/doc as defining a schema attribute. (#784) 2018-07-13 14:29:29 -07:00
Grisha Kruglov
bff24c60b7
Add a top-level "syncable" feature. (#782) r=ncalexan
* Add a top-level "syncable" feature.

Tested with:

cargo test --all
cargo test --all --no-default-features
cargo build --manifest-path tools/cli/Cargo.toml --no-default-features
cargo run --manifest-path tools/cli/Cargo.toml --no-default-features debugcli

Co-authored-by: Nick Alexander <nalexander@mozilla.com>

* Add 'syncable' feature to 'db' crate to conditionally derive serialization for Partition*

This is leading up to syncing with partition support.
2018-07-11 16:26:06 -07:00
Nick Alexander
61e6b85e6a Make Partition include end of range and allow_excision flag. r=grisha,nalexander 2018-07-06 16:12:28 -07:00
Nick Alexander
82610f17f8 Part 2: Make partition include an allow_excision flag.
This is leading up to the implementation of
https://github.com/mozilla/mentat/issues/21.
2018-07-06 16:11:42 -07:00
Grisha Kruglov
c0ddc2ca70 Part 1: Make Partition include explicit end range bound.
It's helpful to have the full range when syncing.
2018-07-06 15:23:06 -07:00
Nick Alexander
7d2fe8c625 Remove low-hanging dependency fruit. (#773) r=nalexander 2018-07-06 14:58:06 -07:00
Thom Chiovoloni
0549bbd604 Remove needless num dependency from mentat_core. 2018-07-06 14:56:42 -07:00
Thom Chiovoloni
dcc0770ca4 Remove needless num dependency from mentat_db and optimize remove_every.
This implementation of `remove_every` is O(n) and not O(n^2) like it was before.
2018-07-06 14:56:33 -07:00
Thom Chiovoloni
ad2b646700 Remove regex dependency from query_sql. Fixes #771. 2018-07-06 14:56:11 -07:00
Nick Alexander
f65512158b Make kw!(foo.bar/bar.baz) work. 2018-07-06 14:19:50 -07:00
Nick Alexander
07c5d733d6 Bump to version 0.8.0.
We've made many breaking changes, especially to error handling, so
it's time to bump versions.
2018-07-05 16:48:27 -07:00
Nick Alexander
46f7db36c9 Small improvements accumulated while building the logins API on top of Mentat. (#779) r=grisha
These build on #778, and implement a variety of small fixes (related
parts are labelled as such), and one non-trivial part -- matching
tuple results with the `BindingTuple` trait. In practice, this is very
helpful, and greatly streamlined the logins API.
2018-07-05 16:46:02 -07:00
Nick Alexander
2cb7d441dc Part 2: Make it easier to match tuple results.
Right now, we write code like
```rust
match q_once(q, inputs)?.into_tuple()? {
    Some(vs) => match (vs.len(), vs.get(0), vs.get(1)) {
        (2, &Some(Binding::Scalar(TypedValue::Long(a))), &Some(Binding::Scalar(TypedValue::Instant(ref b)))) => Some((a, b.clone())),
        _ => panic!(),
    },
    None => None,
}
```
to length-check tuples coming out of the database.  It can also lead
to a lot of cloning because references are the easiest thing to hand.

This commit allows to write code like
```rust
match q_once(q, inputs)?.into_tuple()? {
    Some((Binding::Scalar(TypedValue::Long(a)), Binding::Scalar(TypedValue::Instant(b)))) => Some((a, b)),
    Some(_) => panic!(),
    None => None,
}
```
which is generally much easier to reason about.
2018-07-05 16:45:42 -07:00
Nick Alexander
e362ca6213 Part 1: Allow to clone useful query structures. 2018-07-05 16:45:42 -07:00
Nick Alexander
2ab481f83e Part 2: Expose time related things at top-level.
Perhaps we actually want to subdivide the top-level namespace so that
there is a `mentat::time` module, but I'd prefer to make part of the
process of fixing the public API as we get ready to christen version
1.0.
2018-07-05 16:45:42 -07:00
Nick Alexander
1c0602fa00 Part 1: Add {From,To}Millis.
I think this is just oversight.  Generally, we should anticipate what
our consumers need to do to interact with Mentat, and producing milli-
and micro-second timestamps is part of that need.
2018-07-05 16:45:42 -07:00
Nick Alexander
3744982cd9 Add last_tx_id. 2018-07-05 16:45:42 -07:00
Nick Alexander
b9f3681728 Part 2: Allow to Deref StructuredMap to the underlying IndexMap.
Again, this is a fundamental Rust pattern for newtypes.  It's awfully
hard to actually use `StructuredMap` without it!
2018-07-05 16:45:42 -07:00
Nick Alexander
d49f702512 Part 1: Expand Binding::val() into Binding::{into_*,as_*}.
This is simply for completeness: we should provide fundamental
conversion patterns even when they are mostly unused in our code base.
2018-07-05 16:45:42 -07:00
Nick Alexander
99deb87b9f Build Entity instances, not Term* instances. Fixes #674. (#778) r=grisha 2018-07-05 16:42:02 -07:00
Nick Alexander
eb1df31ac4 Part 7: Improve TermBuilder interface; expose lookup refs and tx functions.
These are functions on `TermBuilder` itself to prevent mixing mutable
and immutable references in the most natural style.  That is,
```
builder.add(e, a, builder.lookup_ref(...))
```
fails because `add` borrows `builder` mutably and `lookup_ref` borrows
`builder` immutably.  There's nothing here that requires a specific
builder (since we're not interning lookup refs on the builder, like we
are tempids) so we don't need an instance.
2018-07-05 16:33:51 -07:00
Nick Alexander
06056a8468 Part 6: Lift TxReport to core crate.
The `core` create didn't exist when the `db` was started, but this
type is clearly part of the public interface of Mentat.
2018-07-05 16:33:51 -07:00
Nick Alexander
1cb1847aa6 Part 5: Make existing TermBuilder actually build Entity instances.
There are a few tricky details to call out here.  The first is the
`TransactableValueMarker` trait.  This is strictly a marker (like
`Sized`, for example) to give some control over what types can be used
as value types in `Entity` instances.  This expression is needed due
to the network of `Into` and `From` relations between the parts of
valid `Entity` instances.  This allows to drop the `IntoThing`
work-around trait and use the established patterns.  (Observe that
`KnownEntid` makes this a little harder, due to the cross-crate
consistency restrictions.)

The second is that we can get rid `{add,retract}_kw`, since the
network of relations expresses the coercions directly.

The third is that this commit doesn't change the name `TermBuilder`,
even though it is now building `Entity` instances.  This is because
there's _already_ an `EntityBuilder` which fixes the `EntityPlace`.
It's not clear whether the existing entity building interface should
be removed or whether both should be renamed.  That can be follow-up.
2018-07-05 16:33:51 -07:00
Nick Alexander
76507623ac Part 4: Prepare EDN Entity type for interning tempids during parsing.
This is all part of moving the entity builder away from building term
instances and toward building entity instances.  One of the nice
things that the existing term interface does is allow consumers to use
lightweight reference counted tempid handles; I don't want to lose
that, so we'll build it into the entity data structures directly.
2018-07-05 11:17:20 -07:00
Nick Alexander
106d6fae11 Part 3: Implement Deref and DerefMut for InternSet.
This pattern is generally how newtype wrappers (like `struct
Foo(Bar)`) are implemented in Rust.
2018-07-05 11:16:55 -07:00
Nick Alexander
02a163a10f Part 2: Use ValueRc in InternSet.
We haven't observed performance issues using `Arc` instead of `Rc`,
and we want to be able to include things that are interned (including,
soon, `TempId` instances) in errors coming out of the
transactor.  (And `Rc` isn't `Sync`, so it can't be included in errors
directly.)
2018-07-05 11:16:53 -07:00
Nick Alexander
87f850a44e Part 1: Move intern_set into edn crate.
It's not great to keep lifting functionality higher and higher up the
crate hierarchy, but we really do want to intern while we parse.
Eventually, I expect that we will split the `edn` crate into `types`
and `parsing`, and the `types` crate can depend on a more efficient
interning dependency.
2018-07-05 11:16:48 -07:00