Commit graph

133 commits

Author SHA1 Message Date
Grisha Kruglov
a57ba5d79f Pre: Move Entid and KnownEntid into core_traits 2018-08-09 13:16:05 -07:00
Grisha Kruglov
536d40ad84 Part 4: Add support for moving transactions off of main timeline 2018-07-26 17:14:05 -07:00
Grisha Kruglov
4ec780c87a Part 3: Use a view to derive parts table
Being able to derive partition map from partition definitions and current
state of the world (transactions), segmented by timelines, is useful
because it lets us not worry about keeping materialized partition maps
up-to-date - since there's no need for materialized partition maps at that point.

This comes in very handy when we start moving chunks of transactions off of our mainline.
Alternative to this work would look like materializing partition maps per timeline,
growing support for incremental "backwards update" of the materialized maps, etc.

Our core partitions are defined in 'known_parts' table during bootstrap,
and what used to be 'parts' table is a generated view that operates over
transactions to figure out partition index.

'parts' is defined for the main timeline. Querying parts for other timelines
or for particular timeline+tx combinations will look similar.
2018-07-26 17:14:05 -07:00
Grisha Kruglov
3ca5255cde Part 2: Add basic support for timelines to the transactor
This records transactions onto a default timeline (0).
2018-07-26 17:14:05 -07:00
Grisha Kruglov
0974108a52 Part 1: Allow specifying transactor's commit behaviour
Normally we want to both materialize our changes (into 'datoms')
as well as commit source transactions into 'transactions' table.

However, when moving transactions from timeline to timeline
we don't want to persist artifacts (rewind assertions), just their
materializations.

This patch expands the 'db' interface to allow for this split,
and changes transactor's functions to take a crate-private 'action'
which defines desired behaviour.
2018-07-26 17:14:05 -07:00
Grisha Kruglov
5a29efa336 Part 0: Allow retractions of installed attributes
This is necessary for the timelines work ahead. When schema is being
moved off of a main timeline, we need to be able to retract it cleanly.

Retractions are only processed if the whole defining attribute set
is being retracted at once (:db/ident, :db/valueType, :db/cardinality).
2018-07-26 17:14:05 -07:00
Grisha Kruglov
9a47d8905f Pre: 'Into' implementation chaining TermWithoutTempIds -> TermWithTempIds -> TermWithTempIdsAndLookupRefs 2018-07-26 17:14:05 -07:00
Nick Alexander
e6066769ca Pre: Differentiate bad attribute retractions from unrecognized retractions. 2018-07-26 17:14:05 -07:00
Grisha Kruglov
69c9f512a0 Move entid allocation logic into Partition r=nalexander 2018-07-17 06:20:37 -07:00
Grisha Kruglov
6290cc9db2 Enforce partition integrity when setting its index r=nalexander
Timelines work starts to perform modifications on the partitions
that go beyond simple allocations. This change pre-emptively protects
partition integrity by asserting that index modifications are legal.
2018-07-17 06:20:37 -07:00
Nick Alexander
38a92229d7 Pre: Replace PartitionMapping trait with newtype. r=grisha
Generally, I think that Mentat is using too many small traits rather
than wrapping types into newtypes.  Wrapping into newtypes is cheap in
Rust, and it makes it easier to reason about the code.
2018-07-17 06:20:37 -07:00
Grisha Kruglov
675a865896
Extract and improve test macros (#787) r=nalexander
* Part 1: Extract low-level test framework into mentat_db::debug for re-use.

* Part 2: Improve assert_matches!.

This corrects an incorrect pattern: a conversion method taking &self
but returning an owned value should be named like `to_FOO(&self) -> FOO`.  (A
reference-to-reference conversion should be named like `as_FOO(&self)
-> &FOO`.  A consuming conversion should be named like `into_FOO(self)
-> FOO`.)

In addition, this pushes the conversion via `to_edn` into the
`assert_matches!` macro, which lets consumers get a real data
structure (say, `Datoms`) and use it directly before or after
`assert_matches!`.  (Currently, consumers get back `edn::Value`
instances, which aren't nearly as pleasant to use as real data
structures.)

Co-authored-by: Grisha Kruglov <gkruglov@mozilla.com>

* Part 3: Use mentat_db::debug framework in Tolstoy crate.

The advantage of this approach is that compiling Tolstoy (or anything
that's not db, really) can be quite a bit faster than compiling db.
2018-07-16 13:58:34 -07:00
Nick Alexander
9291b2a0b0 [tx] Don't treat :db/doc as defining a schema attribute. (#784) 2018-07-13 14:29:29 -07:00
Grisha Kruglov
bff24c60b7
Add a top-level "syncable" feature. (#782) r=ncalexan
* Add a top-level "syncable" feature.

Tested with:

cargo test --all
cargo test --all --no-default-features
cargo build --manifest-path tools/cli/Cargo.toml --no-default-features
cargo run --manifest-path tools/cli/Cargo.toml --no-default-features debugcli

Co-authored-by: Nick Alexander <nalexander@mozilla.com>

* Add 'syncable' feature to 'db' crate to conditionally derive serialization for Partition*

This is leading up to syncing with partition support.
2018-07-11 16:26:06 -07:00
Nick Alexander
82610f17f8 Part 2: Make partition include an allow_excision flag.
This is leading up to the implementation of
https://github.com/mozilla/mentat/issues/21.
2018-07-06 16:11:42 -07:00
Grisha Kruglov
c0ddc2ca70 Part 1: Make Partition include explicit end range bound.
It's helpful to have the full range when syncing.
2018-07-06 15:23:06 -07:00
Thom Chiovoloni
dcc0770ca4 Remove needless num dependency from mentat_db and optimize remove_every.
This implementation of `remove_every` is O(n) and not O(n^2) like it was before.
2018-07-06 14:56:33 -07:00
Nick Alexander
06056a8468 Part 6: Lift TxReport to core crate.
The `core` create didn't exist when the `db` was started, but this
type is clearly part of the public interface of Mentat.
2018-07-05 16:33:51 -07:00
Nick Alexander
76507623ac Part 4: Prepare EDN Entity type for interning tempids during parsing.
This is all part of moving the entity builder away from building term
instances and toward building entity instances.  One of the nice
things that the existing term interface does is allow consumers to use
lightweight reference counted tempid handles; I don't want to lose
that, so we'll build it into the entity data structures directly.
2018-07-05 11:17:20 -07:00
Nick Alexander
106d6fae11 Part 3: Implement Deref and DerefMut for InternSet.
This pattern is generally how newtype wrappers (like `struct
Foo(Bar)`) are implemented in Rust.
2018-07-05 11:16:55 -07:00
Nick Alexander
02a163a10f Part 2: Use ValueRc in InternSet.
We haven't observed performance issues using `Arc` instead of `Rc`,
and we want to be able to include things that are interned (including,
soon, `TempId` instances) in errors coming out of the
transactor.  (And `Rc` isn't `Sync`, so it can't be included in errors
directly.)
2018-07-05 11:16:53 -07:00
Nick Alexander
87f850a44e Part 1: Move intern_set into edn crate.
It's not great to keep lifting functionality higher and higher up the
crate hierarchy, but we really do want to intern while we parse.
Eventually, I expect that we will split the `edn` crate into `types`
and `parsing`, and the `types` crate can depend on a more efficient
interning dependency.
2018-07-05 11:16:48 -07:00
Nick Alexander
8725bad18c Pre: Fix error printing rusqlite::Error. 2018-06-30 14:58:23 -07:00
Nick Alexander
0e4991fa26 Make db/ use DbErrorKind. 2018-06-27 15:05:43 -07:00
Thom
72a9b302f9
Rename or delete things so that there is only one type named Entid (#768)
* Delete the (apparently unused) EntId

* Rename edn's Entid to EntidOrIdent to avoid confusion with the Entid that's actually an i64

* Fix travis beta bustage (This is actually unrelated to entids, but is a trivial fix nonetheless)
2018-06-26 16:34:18 -07:00
Grisha Kruglov
31de5be64f Convert db/ to failure. 2018-06-20 14:42:34 -07:00
Thom Chiovoloni
99a73ccb03 Avoid using 1.26.0-only features when using sqlcipher, and move the sqlcipher Store support to the correct file 2018-06-13 22:48:28 -07:00
Thom
6a1a265894
Add support for using sqlcipher (#737). Fixes #118 2018-06-13 08:49:40 -07:00
Nick Alexander
e68cc4016c Part 7: Remove tx entirely.
This was left over from #681.
2018-06-04 15:04:39 -07:00
Richard Newman
b2e98f44f6
Generalize Entity by value type. (#701) (#691) r=rnewman
* Part 3: Parameterize Entity by value type.

This isn't quite right, because after parsing, we shouldn't care
about` `edn::ValueAndSpan`, we should care only about edn::Value.
However, I think we can drop `ValueAndSpan` entirely if we just use
`rust-peg` (and its simpler error messages) rather than a mix of
`rust-peg` and `combine`.

In any case, this paves the way to transacting `Entity<TypedValue>`,
which is a nice step towards building general entities.

* Part 1: Add AttributePlace.

* Part 2: Name other places EntityPlace and ValuePlace.

Now we're consistent and closer to self-documenting.  Both matter more
as we expose `Entity` as the thing to build for programmatic usage.

* Part 4: Allow Ident and TempId in ValuePlace.

The parser will never produce these, since determining whether an
integer/keyword or string is an ident or a tempid, respectively, in
the value place requires the schema.

But a builder that produces `Entity` instances directly will want to
produce these.
2018-05-15 00:43:07 -07:00
Nick Alexander
46c2a0801f Add type checking and constraint checking to the transactor. (#663, #532, #679)
This should address #663, by re-inserting type checking in the
transactor stack after the entry point used by the term builder.

Before this commit, we were using an SQLite UNIQUE index to assert
that no `[e a]` pair, with `a` a cardinality one attribute, was
asserted more than once.  However, that's not in line with Datomic,
which treats transaction inputs as a set and allows a single datom
like `[e a v]` to appear multiple times.  It's both awkward and not
particularly efficient to look for _distinct_ repetitions in SQL, so
we accept some runtime cost in order to check for repetitions in the
transactor.  This will allow us to address #532, which is really about
whether we treat inputs as sets.  A side benefit is that we can
provide more helpful error messages when the transactor does detect
that the input truly violates the cardinality constraints of the
schema.

This commit builds a trie while error checking and collecting final
terms, which should be fairly efficient.  It also allows a simpler
expression of input-provided :db/txInstant datoms, which in turn
uncovered a small issue with the transaction watcher, where-by the
watcher would not see non-input-provided :db/txInstant datoms.

This transition to Datomic-like input-as-set semantics allows us to
address #532.  Previously, two tempids that upserted to the same entid
would produce duplicate datoms, and that would have been rejected by
the transactor -- correctly, since we did not allow duplicate datoms
under the input-as-list semantics.  With input-as-set semantics,
duplicate datoms are allowed; and that means that we must allow
tempids to be equivalent, i.e., to resolve to the same tempid.

To achieve this, we:
- index the set of tempids
- identify tempid indices that share an upsert
- map tempids to a dense set of contiguous integer labels

We use the well-known union-find algorithm, as implemented by
petgraph, to efficiently manage the set of equivalent tempids.

Along the way, I've fixed and added tests for two small errors in the
transactor.  First, don't drop datoms resolved by upsert (#679).
Second, ensure that complex upserts are allocated.

I don't know quite what happened here.  The Clojure implementation
correctly kept complex upserts that hadn't resolved as complex
upserts (see
9a9dfb502a/src/common/datomish/transact.cljc (L436))
and then allocated complex upserts if they didn't resolve (see
9a9dfb502a/src/common/datomish/transact.cljc (L509)).

Based on the code comments, I think the Rust implementation must have
incorrectly tried to optimize by handling all complex upserts in at
most a single generation of evolution, and that's just not correct.
We're effectively implementing a topological sort, using very specific
domain knowledge, and its not true that a node in a topological sort
can be considered only once!
2018-05-14 15:22:45 -07:00
Nick Alexander
e5e37178af Pre: Remove ancient Clojure code comments. 2018-05-14 15:22:45 -07:00
Nick Alexander
4fde4fe0a6 Bustage fixes: compile on stable; avoid unused variable warning. 2018-05-11 10:22:57 -07:00
Richard Newman
3dc68bcd38 Combine NamespacedKeyword and Keyword. (#689) r=nalexander
* Make properties on NamespacedKeyword/NamespacedSymbol private

* Use only a single String for NamespacedKeyword/NamespacedSymbol

* Review comments.

* Remove unsafe code in namespaced_name.

Benchmarking shows approximately zero change.

* Allow the types of ns and name to differ when constructing a NamespacedName.

* Make symbol namespaces optional.

* Normalize names of keyword/symbol constructors.

This will make the subsequent refactor much less painful.

* Use expect not unwrap.

* Merge Keyword and NamespacedKeyword.
2018-05-11 09:52:17 -07:00
Nick Alexander
cbffe5e545 Use rust-peg for tx parsing.
There are few reasons to do this:

- it's difficult to add symbol interning to combine-based parsers like
  tx-parser -- literally every type changes to reflect the interner,
  and that means every convenience macro we've built needs to chagne.
  It's trivial to add interning to rust-peg-based parsers.

- combine has rolled forward to 3.2, and I spent a similar amount of
  time investigating how to upgrade tx-parser (to take advantage of
  the new parser! macros in combine that I think are necessary for
  adapting to changing types) as I did just converting to rust-peg.

- it's easy to improve the error messages in rust-peg, where-as I have
  tried twice to improve the nested error messages in combine and am
  stumped.

- it's roughly 4x faster to parse strings directly as opposed to
  edn::ValueAndSpan, and it'll be even better when we intern directly.
2018-05-10 10:24:05 -07:00
Nick Alexander
e437944d94 Pre: Don't use tx-parser for destructuring map notation.
This was always a choice, but we've outgrown it: now we want to accept
value types that don't come from EDN and/or tx-parser.
2018-05-10 10:19:54 -07:00
Nick Alexander
4c4af46315 Add TransactableValue abstracting value places that can be transacted.
This is a stepping stone to transacting entities that are not based on
`edn::ValueAndSpan`.  We need to turn some value places (general) into
entity places (restricted), and those restrictions are captured in
tx-parser right now.  But for `TypedValue` value places, those
restrictions are encoded in the type itself.  This lays the track to
accept other value types in value places, which is good for
programmatic builder interfaces.
2018-05-10 10:19:54 -07:00
Richard Newman
e21156a754
Implement simple pull expressions (#638) r=nalexander
* Refactor AttributeCache populator code for use from pull.

* Pre: add to_value_rc to Cloned.

* Pre: add From<StructuredMap> for Binding.

* Pre: clarify Store::open_empty.

* Pre: StructuredMap cleanup.

* Pre: clean up a doc test.

* Split projector crate. Pass schema to projector.

* CLI support for printing bindings.

* Add and use ConjoiningClauses::derive_types_from_find_spec.

* Define pull types.

* Implement pull on top of the attribute cache layer.

* Add pull support to the projector.

* Parse pull expressions.

* Add simple pull support to connection objects.

* Tests for pull.

* Compile with Rust 1.25.

The only choice involved in this commit is that of replacing the
anonymous lifetime '_ with a named lifetime for the cache; since we're
accepting a Known, which includes the cache in question, I think it's
clear that we expect the function to apply to any given cache
lifetime.

* Review comments.

* Bail on unnamed attribute.

* Make assert_parse_failure_contains safe to use.

* Rework query parser to report better errors for pull.

* Test for mixed wildcard and simple attribute.
2018-05-04 12:56:00 -07:00
Nick Alexander
2b82ffb2e5 [tx] Fail transactions where complex upserts resolve to multiple entids. (#670)
This innocuous looking change (upserts_ev -> upserts_e -> resolved in
all situations, rather than upserts_ev -> resolved in some situations)
is a significant change in semantics and assumptions in the
transactor.  Witness the large comment being removed about the same
tempid resolving in different generations!

To support this change, we provide more holistic errors for
conflicting upserts, which entails collecting some (relatively
expensive) diagnostic data.

I left in some debug logging, simply since it shouldn't hurt in
general, and will likely be useful for the next bug we see in the
transactor.
2018-05-01 15:34:44 -07:00
Nick Alexander
7960b4ccd2 Pre: Get ready to use log in mentat_db.
We don't yet have a logging system for production use, but I'd like to
start experimenting with log, which seems to be (close to) a Rust
standard.  We're already using it in mentat_cli.
2018-05-01 13:46:03 -07:00
Nick Alexander
32ed56685e (tx) Replace :db/tx with (transaction-tx) transaction function and broaden support. (#664)
:db/tx (and Datomic's version, :datomic/tx) suffer from the same
ambiguities that [a v] lookup references do -- determining the type of
the result is context sensitive.  (In this case, is :db/tx a reference
to the current transaction ID, or is it a valid keyword?)  This commit
addresses the ambiguity by introducing a notion of a transaction
functions, and provides a little scaffolding for adding more (should
the need arise).  I left the scaffolding in place rather than handling
just (transaction-tx) because I started trying to
implement (transaction-instant) as well, which is more difficult --
see the comments.

It's worth noting that this approach generalizes more or less directly
to ?input variables, since those can be eagerly bound like the
implemented transaction function (transaction-tx).
2018-04-26 19:32:14 -07:00
Richard Newman
f979044ba1
Refactor value type boxing. (#659) r=nalexander
* Pre: eliminate some occurrences of Rc, largely through the magic of Into.
* Pre: introduce FromRc to convert between refcounted types.
* Introduce ValueRc as an abstraction over Rc/Arc choice.
* Move Cloned to core.
* Move CString-creation methods to TypedValue.
* Finish transition.
2018-04-25 14:23:27 -07:00
Richard Newman
a74a2deffc
Introduce RelResult rather than Vec<Vec<TypedValue>>. (#639) r=nalexander
* Pre: clean up core/src/lib.rs.
* Pre: use indexmap 1.0 in db and query-projector.
* Change rel results to be a RelResult instance, not a Vec<Vec<TypedValue>>.

This avoids memory fragmentation and improves locality by using a single
heap-allocated vector for all bindings, rather than a separate
heap-allocated vector for each row.

We hide this abstraction behind the `RelResult` type, which tracks the
stride length (width) of each row.

* Don't allocate temporary vectors when projecting RelResults.
2018-04-24 15:04:00 -07:00
Richard Newman
27dde378e0 Allow retraction of some schema attributes. (#379) r=nalexander 2018-04-09 09:47:49 -07:00
Richard Newman
4d8e179a59
Expose component_attributes on Schema. (#623) r=nalexander
Some parts of the query engine and transactor need to know whether an
attribute is a component attribute, and sometimes want to do so in
a generated SQL query. This is one way to do that.
2018-04-03 14:25:53 -07:00
Richard Newman
6c54e1d370
Support :db/noHistory for attributes. (#622) r=nalexander
At this point we never discard history, but this completes the API support for doing so.
2018-04-03 14:23:46 -07:00
Richard Newman
9cc5cbf288
Rename the helpful variant, AttributeBuilder::new, to AttributeBuilder::helpful. (#625) r=nalexander 2018-04-03 14:23:20 -07:00
Richard Newman
66b892572c
Don't create a CommandExecutor if there are no observers. (#603) (#604) r=emily
* Don't create a CommandExecutor if there are no observers. (#603)
* Don't log if our executor channel goes away. This is routine.
2018-04-03 09:18:22 -07:00
Emily Toop
ab957948b4 Move to using watcher.
Simplify.

This has a watcher collect txid -> AttributeSet mappings each time a
transact occurs. On commit we retrieve those mappings and hand them over
to the observer service, which filters them and packages them up for
dispatch.

Tidy up
2018-03-20 16:27:35 +00:00
Emily Toop
d4365fa4cd Execute commands in a separate thread
Command Queue Executor to watch for new commands and execute on longer running background thread
2018-03-20 16:27:35 +00:00