2017-02-15 00:50:40 +00:00
|
|
|
// Copyright 2016 Mozilla
|
|
|
|
//
|
|
|
|
// Licensed under the Apache License, Version 2.0 (the "License"); you may not use
|
|
|
|
// this file except in compliance with the License. You may obtain a copy of the
|
|
|
|
// License at http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
// Unless required by applicable law or agreed to in writing, software distributed
|
|
|
|
// under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
|
|
|
|
// CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
|
|
|
// specific language governing permissions and limitations under the License.
|
|
|
|
|
|
|
|
#![allow(dead_code)]
|
|
|
|
|
|
|
|
//! Types used only within the transactor. These should not be exposed outside of this crate.
|
|
|
|
|
Add type checking and constraint checking to the transactor. (#663, #532, #679)
This should address #663, by re-inserting type checking in the
transactor stack after the entry point used by the term builder.
Before this commit, we were using an SQLite UNIQUE index to assert
that no `[e a]` pair, with `a` a cardinality one attribute, was
asserted more than once. However, that's not in line with Datomic,
which treats transaction inputs as a set and allows a single datom
like `[e a v]` to appear multiple times. It's both awkward and not
particularly efficient to look for _distinct_ repetitions in SQL, so
we accept some runtime cost in order to check for repetitions in the
transactor. This will allow us to address #532, which is really about
whether we treat inputs as sets. A side benefit is that we can
provide more helpful error messages when the transactor does detect
that the input truly violates the cardinality constraints of the
schema.
This commit builds a trie while error checking and collecting final
terms, which should be fairly efficient. It also allows a simpler
expression of input-provided :db/txInstant datoms, which in turn
uncovered a small issue with the transaction watcher, where-by the
watcher would not see non-input-provided :db/txInstant datoms.
This transition to Datomic-like input-as-set semantics allows us to
address #532. Previously, two tempids that upserted to the same entid
would produce duplicate datoms, and that would have been rejected by
the transactor -- correctly, since we did not allow duplicate datoms
under the input-as-list semantics. With input-as-set semantics,
duplicate datoms are allowed; and that means that we must allow
tempids to be equivalent, i.e., to resolve to the same tempid.
To achieve this, we:
- index the set of tempids
- identify tempid indices that share an upsert
- map tempids to a dense set of contiguous integer labels
We use the well-known union-find algorithm, as implemented by
petgraph, to efficiently manage the set of equivalent tempids.
Along the way, I've fixed and added tests for two small errors in the
transactor. First, don't drop datoms resolved by upsert (#679).
Second, ensure that complex upserts are allocated.
I don't know quite what happened here. The Clojure implementation
correctly kept complex upserts that hadn't resolved as complex
upserts (see
https://github.com/mozilla/mentat/blob/9a9dfb502acf5e4cdb1059d4aac831d7603063c8/src/common/datomish/transact.cljc#L436)
and then allocated complex upserts if they didn't resolve (see
https://github.com/mozilla/mentat/blob/9a9dfb502acf5e4cdb1059d4aac831d7603063c8/src/common/datomish/transact.cljc#L509).
Based on the code comments, I think the Rust implementation must have
incorrectly tried to optimize by handling all complex upserts in at
most a single generation of evolution, and that's just not correct.
We're effectively implementing a topological sort, using very specific
domain knowledge, and its not true that a node in a topological sort
can be considered only once!
2018-04-30 22:16:05 +00:00
|
|
|
use std::collections::{
|
|
|
|
BTreeMap,
|
|
|
|
BTreeSet,
|
|
|
|
HashMap,
|
|
|
|
};
|
2017-02-15 00:50:40 +00:00
|
|
|
use std::rc::Rc;
|
|
|
|
|
2018-01-23 16:31:27 +00:00
|
|
|
use mentat_core::KnownEntid;
|
|
|
|
|
2017-06-14 21:42:34 +00:00
|
|
|
use mentat_core::util::Either;
|
|
|
|
|
2018-05-07 21:41:59 +00:00
|
|
|
use edn;
|
2018-05-07 20:55:20 +00:00
|
|
|
use edn::{
|
2018-05-07 21:41:59 +00:00
|
|
|
SpannedValue,
|
2018-05-07 20:55:20 +00:00
|
|
|
ValueAndSpan,
|
|
|
|
};
|
|
|
|
|
2017-02-15 00:50:40 +00:00
|
|
|
use errors;
|
2018-05-07 20:55:20 +00:00
|
|
|
use errors::{
|
|
|
|
ErrorKind,
|
|
|
|
Result,
|
|
|
|
};
|
|
|
|
use schema::{
|
|
|
|
SchemaTypeChecking,
|
|
|
|
};
|
2017-02-20 13:13:16 +00:00
|
|
|
use types::{
|
Add type checking and constraint checking to the transactor. (#663, #532, #679)
This should address #663, by re-inserting type checking in the
transactor stack after the entry point used by the term builder.
Before this commit, we were using an SQLite UNIQUE index to assert
that no `[e a]` pair, with `a` a cardinality one attribute, was
asserted more than once. However, that's not in line with Datomic,
which treats transaction inputs as a set and allows a single datom
like `[e a v]` to appear multiple times. It's both awkward and not
particularly efficient to look for _distinct_ repetitions in SQL, so
we accept some runtime cost in order to check for repetitions in the
transactor. This will allow us to address #532, which is really about
whether we treat inputs as sets. A side benefit is that we can
provide more helpful error messages when the transactor does detect
that the input truly violates the cardinality constraints of the
schema.
This commit builds a trie while error checking and collecting final
terms, which should be fairly efficient. It also allows a simpler
expression of input-provided :db/txInstant datoms, which in turn
uncovered a small issue with the transaction watcher, where-by the
watcher would not see non-input-provided :db/txInstant datoms.
This transition to Datomic-like input-as-set semantics allows us to
address #532. Previously, two tempids that upserted to the same entid
would produce duplicate datoms, and that would have been rejected by
the transactor -- correctly, since we did not allow duplicate datoms
under the input-as-list semantics. With input-as-set semantics,
duplicate datoms are allowed; and that means that we must allow
tempids to be equivalent, i.e., to resolve to the same tempid.
To achieve this, we:
- index the set of tempids
- identify tempid indices that share an upsert
- map tempids to a dense set of contiguous integer labels
We use the well-known union-find algorithm, as implemented by
petgraph, to efficiently manage the set of equivalent tempids.
Along the way, I've fixed and added tests for two small errors in the
transactor. First, don't drop datoms resolved by upsert (#679).
Second, ensure that complex upserts are allocated.
I don't know quite what happened here. The Clojure implementation
correctly kept complex upserts that hadn't resolved as complex
upserts (see
https://github.com/mozilla/mentat/blob/9a9dfb502acf5e4cdb1059d4aac831d7603063c8/src/common/datomish/transact.cljc#L436)
and then allocated complex upserts if they didn't resolve (see
https://github.com/mozilla/mentat/blob/9a9dfb502acf5e4cdb1059d4aac831d7603063c8/src/common/datomish/transact.cljc#L509).
Based on the code comments, I think the Rust implementation must have
incorrectly tried to optimize by handling all complex upserts in at
most a single generation of evolution, and that's just not correct.
We're effectively implementing a topological sort, using very specific
domain knowledge, and its not true that a node in a topological sort
can be considered only once!
2018-04-30 22:16:05 +00:00
|
|
|
Attribute,
|
2017-02-20 13:13:16 +00:00
|
|
|
AVMap,
|
|
|
|
AVPair,
|
|
|
|
Entid,
|
2018-05-07 20:55:20 +00:00
|
|
|
Schema,
|
2018-05-15 07:43:07 +00:00
|
|
|
TransactableValue,
|
2017-02-20 13:13:16 +00:00
|
|
|
TypedValue,
|
2018-05-07 20:55:20 +00:00
|
|
|
ValueType,
|
2017-02-20 13:13:16 +00:00
|
|
|
};
|
2018-05-07 21:41:59 +00:00
|
|
|
use mentat_tx::entities;
|
Lookup refs, nested vector values, map notation. Fixes #180, fixes #183, fixes #284. (#382) r=rnewman
* Pre: Fix error in parser macros.
* Pre: Make test unwrapping more verbose.
* Pre: Make lookup refs be (lookup-ref a v) in the entity position.
This has the advantage of being explicit in all situations and
unambiguous at parse-time. This choice agrees with the Clojure
implementation but not with Datomic. Datomic treats [a v] as a lookup
ref, is ambiguous at parse-time, and is disambiguated in ways I do not
understand at transaction time. We mooted making lookup refs [[a v]]
and outlawing nested value vectors in transactions, but after
implementing that approach I decided it was better to handle lookup
refs at parse time and therefore outlawing nested value vectors is not
necessary.
* Handle lookup refs in the entity and value columns. Fixes #183.
* Pre 0a: Use a stack instead of into_iter.
* Pre 0b: Dedent.
* Pre 0c: Handle `e` after `v`.
This allows to use the original `e` while handling `v`.
* Explode value lists for :db.cardinality/many attributes. Fixes #284.
* Parse and accept map notation. Fixes #180.
* Pre: Modernize add() and retract() into one add_or_retract().
* Pre: Add is_collection and is_atom to edn::Value.
* Pre: Differentiate atoms from lookup-refs in value position.
Initially, I expected to accept arbitrary edn::Value instances in the
value position, and to differentiate in the transactor. However, the
implementation quickly became a two-stage parser, since we always
wanted to parse the resulting value position into some other known
thing using the tx-parser. To save calls into the parser and to allow
the parser to move forward with a smaller API surface, I push as much
of this parsing as possible into the initial parse.
* Pre: Modernize entities().
* Pre: Quote edn::Value::Text in Display.
* Review comment: Add and use edn::Value::into_atom.
* Review comment: Use skip(eof()) throughout.
* Review comment: VecDeque instead of Vec.
* Review comment: Part 0: Rename TempId to TempIdHandle.
* Review comment: Part 1: Differentiate internal and external tempids.
This breaks an abstraction boundary by pushing the Internal/External
split up to the Entity level in tx/ and tx-parser/. This just makes
it easier to explode Entity map notation instances into Entity
instances, taking an existing External tempid :db/id or generating a
new Internal tempid as appropriate. To do this without breaking the
abstraction boundary would require adding flexibility to the
transaction processor: we'd need to be able to turn Entity instances
into some internal enum and handle the two cases independently. It
wouldn't be too hard, but this reduces the combinatorial type
explosion.
2017-03-27 23:30:04 +00:00
|
|
|
use mentat_tx::entities::{
|
2018-05-15 07:43:07 +00:00
|
|
|
EntityPlace,
|
Lookup refs, nested vector values, map notation. Fixes #180, fixes #183, fixes #284. (#382) r=rnewman
* Pre: Fix error in parser macros.
* Pre: Make test unwrapping more verbose.
* Pre: Make lookup refs be (lookup-ref a v) in the entity position.
This has the advantage of being explicit in all situations and
unambiguous at parse-time. This choice agrees with the Clojure
implementation but not with Datomic. Datomic treats [a v] as a lookup
ref, is ambiguous at parse-time, and is disambiguated in ways I do not
understand at transaction time. We mooted making lookup refs [[a v]]
and outlawing nested value vectors in transactions, but after
implementing that approach I decided it was better to handle lookup
refs at parse time and therefore outlawing nested value vectors is not
necessary.
* Handle lookup refs in the entity and value columns. Fixes #183.
* Pre 0a: Use a stack instead of into_iter.
* Pre 0b: Dedent.
* Pre 0c: Handle `e` after `v`.
This allows to use the original `e` while handling `v`.
* Explode value lists for :db.cardinality/many attributes. Fixes #284.
* Parse and accept map notation. Fixes #180.
* Pre: Modernize add() and retract() into one add_or_retract().
* Pre: Add is_collection and is_atom to edn::Value.
* Pre: Differentiate atoms from lookup-refs in value position.
Initially, I expected to accept arbitrary edn::Value instances in the
value position, and to differentiate in the transactor. However, the
implementation quickly became a two-stage parser, since we always
wanted to parse the resulting value position into some other known
thing using the tx-parser. To save calls into the parser and to allow
the parser to move forward with a smaller API surface, I push as much
of this parsing as possible into the initial parse.
* Pre: Modernize entities().
* Pre: Quote edn::Value::Text in Display.
* Review comment: Add and use edn::Value::into_atom.
* Review comment: Use skip(eof()) throughout.
* Review comment: VecDeque instead of Vec.
* Review comment: Part 0: Rename TempId to TempIdHandle.
* Review comment: Part 1: Differentiate internal and external tempids.
This breaks an abstraction boundary by pushing the Internal/External
split up to the Entity level in tx/ and tx-parser/. This just makes
it easier to explode Entity map notation instances into Entity
instances, taking an existing External tempid :db/id or generating a
new Internal tempid as appropriate. To do this without breaking the
abstraction boundary would require adding flexibility to the
transaction processor: we'd need to be able to turn Entity instances
into some internal enum and handle the two cases independently. It
wouldn't be too hard, but this reduces the combinatorial type
explosion.
2017-03-27 23:30:04 +00:00
|
|
|
OpType,
|
|
|
|
TempId,
|
2018-05-07 21:41:59 +00:00
|
|
|
TxFunction,
|
Lookup refs, nested vector values, map notation. Fixes #180, fixes #183, fixes #284. (#382) r=rnewman
* Pre: Fix error in parser macros.
* Pre: Make test unwrapping more verbose.
* Pre: Make lookup refs be (lookup-ref a v) in the entity position.
This has the advantage of being explicit in all situations and
unambiguous at parse-time. This choice agrees with the Clojure
implementation but not with Datomic. Datomic treats [a v] as a lookup
ref, is ambiguous at parse-time, and is disambiguated in ways I do not
understand at transaction time. We mooted making lookup refs [[a v]]
and outlawing nested value vectors in transactions, but after
implementing that approach I decided it was better to handle lookup
refs at parse time and therefore outlawing nested value vectors is not
necessary.
* Handle lookup refs in the entity and value columns. Fixes #183.
* Pre 0a: Use a stack instead of into_iter.
* Pre 0b: Dedent.
* Pre 0c: Handle `e` after `v`.
This allows to use the original `e` while handling `v`.
* Explode value lists for :db.cardinality/many attributes. Fixes #284.
* Parse and accept map notation. Fixes #180.
* Pre: Modernize add() and retract() into one add_or_retract().
* Pre: Add is_collection and is_atom to edn::Value.
* Pre: Differentiate atoms from lookup-refs in value position.
Initially, I expected to accept arbitrary edn::Value instances in the
value position, and to differentiate in the transactor. However, the
implementation quickly became a two-stage parser, since we always
wanted to parse the resulting value position into some other known
thing using the tx-parser. To save calls into the parser and to allow
the parser to move forward with a smaller API surface, I push as much
of this parsing as possible into the initial parse.
* Pre: Modernize entities().
* Pre: Quote edn::Value::Text in Display.
* Review comment: Add and use edn::Value::into_atom.
* Review comment: Use skip(eof()) throughout.
* Review comment: VecDeque instead of Vec.
* Review comment: Part 0: Rename TempId to TempIdHandle.
* Review comment: Part 1: Differentiate internal and external tempids.
This breaks an abstraction boundary by pushing the Internal/External
split up to the Entity level in tx/ and tx-parser/. This just makes
it easier to explode Entity map notation instances into Entity
instances, taking an existing External tempid :db/id or generating a
new Internal tempid as appropriate. To do this without breaking the
abstraction boundary would require adding flexibility to the
transaction processor: we'd need to be able to turn Entity instances
into some internal enum and handle the two cases independently. It
wouldn't be too hard, but this reduces the combinatorial type
explosion.
2017-03-27 23:30:04 +00:00
|
|
|
};
|
2018-05-07 20:55:20 +00:00
|
|
|
|
|
|
|
impl TransactableValue for ValueAndSpan {
|
|
|
|
fn into_typed_value(self, schema: &Schema, value_type: ValueType) -> Result<TypedValue> {
|
2018-05-15 07:43:07 +00:00
|
|
|
schema.to_typed_value(&self, value_type)
|
2018-05-07 20:55:20 +00:00
|
|
|
}
|
|
|
|
|
2018-05-15 07:43:07 +00:00
|
|
|
fn into_entity_place(self) -> Result<EntityPlace<Self>> {
|
2018-05-07 21:41:59 +00:00
|
|
|
use self::SpannedValue::*;
|
|
|
|
match self.inner {
|
2018-05-15 07:43:07 +00:00
|
|
|
Integer(v) => Ok(EntityPlace::Entid(entities::Entid::Entid(v))),
|
2018-05-11 16:52:17 +00:00
|
|
|
Keyword(v) => {
|
|
|
|
if v.is_namespaced() {
|
2018-05-15 07:43:07 +00:00
|
|
|
Ok(EntityPlace::Entid(entities::Entid::Ident(v)))
|
2018-05-11 16:52:17 +00:00
|
|
|
} else {
|
|
|
|
// We only allow namespaced idents.
|
|
|
|
bail!(ErrorKind::InputError(errors::InputError::BadEntityPlace))
|
|
|
|
}
|
|
|
|
},
|
2018-05-15 07:43:07 +00:00
|
|
|
Text(v) => Ok(EntityPlace::TempId(TempId::External(v))),
|
2018-05-07 21:41:59 +00:00
|
|
|
List(ls) => {
|
|
|
|
let mut it = ls.iter();
|
|
|
|
match (it.next().map(|x| &x.inner), it.next(), it.next(), it.next()) {
|
|
|
|
// Like "(transaction-id)".
|
|
|
|
(Some(&PlainSymbol(ref op)), None, None, None) => {
|
2018-05-15 07:43:07 +00:00
|
|
|
Ok(EntityPlace::TxFunction(TxFunction { op: op.clone() }))
|
2018-05-07 21:41:59 +00:00
|
|
|
},
|
|
|
|
// Like "(lookup-ref)".
|
|
|
|
(Some(&PlainSymbol(edn::PlainSymbol(ref s))), Some(a), Some(v), None) if s == "lookup-ref" => {
|
|
|
|
match a.clone().into_entity_place()? {
|
2018-05-15 07:43:07 +00:00
|
|
|
EntityPlace::Entid(a) => Ok(EntityPlace::LookupRef(entities::LookupRef { a: entities::AttributePlace::Entid(a), v: v.clone() })),
|
|
|
|
EntityPlace::TempId(_) |
|
|
|
|
EntityPlace::TxFunction(_) |
|
|
|
|
EntityPlace::LookupRef(_) => bail!(ErrorKind::InputError(errors::InputError::BadEntityPlace)),
|
2018-05-07 21:41:59 +00:00
|
|
|
}
|
|
|
|
},
|
2018-05-07 17:32:28 +00:00
|
|
|
_ => bail!(ErrorKind::InputError(errors::InputError::BadEntityPlace)),
|
2018-05-07 21:41:59 +00:00
|
|
|
}
|
|
|
|
},
|
|
|
|
Nil |
|
|
|
|
Boolean(_) |
|
|
|
|
Instant(_) |
|
|
|
|
BigInteger(_) |
|
|
|
|
Float(_) |
|
|
|
|
Uuid(_) |
|
|
|
|
PlainSymbol(_) |
|
|
|
|
NamespacedSymbol(_) |
|
|
|
|
Vector(_) |
|
|
|
|
Set(_) |
|
2018-05-07 17:32:28 +00:00
|
|
|
Map(_) => bail!(ErrorKind::InputError(errors::InputError::BadEntityPlace)),
|
2018-05-07 21:41:59 +00:00
|
|
|
}
|
2018-05-07 20:55:20 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
fn as_tempid(&self) -> Option<TempId> {
|
|
|
|
self.inner.as_text().cloned().map(TempId::External)
|
|
|
|
}
|
|
|
|
}
|
2017-02-15 00:50:40 +00:00
|
|
|
|
2018-05-15 07:43:07 +00:00
|
|
|
impl TransactableValue for TypedValue {
|
|
|
|
fn into_typed_value(self, _schema: &Schema, value_type: ValueType) -> Result<TypedValue> {
|
|
|
|
if self.value_type() != value_type {
|
|
|
|
bail!(ErrorKind::BadValuePair(format!("{:?}", self), value_type));
|
|
|
|
}
|
|
|
|
Ok(self)
|
|
|
|
}
|
|
|
|
|
|
|
|
fn into_entity_place(self) -> Result<EntityPlace<Self>> {
|
|
|
|
match self {
|
|
|
|
TypedValue::Ref(x) => Ok(EntityPlace::Entid(entities::Entid::Entid(x))),
|
|
|
|
TypedValue::Keyword(x) => Ok(EntityPlace::Entid(entities::Entid::Ident((*x).clone()))),
|
|
|
|
TypedValue::String(x) => Ok(EntityPlace::TempId(TempId::External((*x).clone()))),
|
|
|
|
TypedValue::Boolean(_) |
|
|
|
|
TypedValue::Long(_) |
|
|
|
|
TypedValue::Double(_) |
|
|
|
|
TypedValue::Instant(_) |
|
|
|
|
TypedValue::Uuid(_) => bail!(ErrorKind::InputError(errors::InputError::BadEntityPlace)),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
fn as_tempid(&self) -> Option<TempId> {
|
|
|
|
match self {
|
|
|
|
&TypedValue::String(ref s) => Some(TempId::External((**s).clone())),
|
|
|
|
_ => None,
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
Lookup refs, nested vector values, map notation. Fixes #180, fixes #183, fixes #284. (#382) r=rnewman
* Pre: Fix error in parser macros.
* Pre: Make test unwrapping more verbose.
* Pre: Make lookup refs be (lookup-ref a v) in the entity position.
This has the advantage of being explicit in all situations and
unambiguous at parse-time. This choice agrees with the Clojure
implementation but not with Datomic. Datomic treats [a v] as a lookup
ref, is ambiguous at parse-time, and is disambiguated in ways I do not
understand at transaction time. We mooted making lookup refs [[a v]]
and outlawing nested value vectors in transactions, but after
implementing that approach I decided it was better to handle lookup
refs at parse time and therefore outlawing nested value vectors is not
necessary.
* Handle lookup refs in the entity and value columns. Fixes #183.
* Pre 0a: Use a stack instead of into_iter.
* Pre 0b: Dedent.
* Pre 0c: Handle `e` after `v`.
This allows to use the original `e` while handling `v`.
* Explode value lists for :db.cardinality/many attributes. Fixes #284.
* Parse and accept map notation. Fixes #180.
* Pre: Modernize add() and retract() into one add_or_retract().
* Pre: Add is_collection and is_atom to edn::Value.
* Pre: Differentiate atoms from lookup-refs in value position.
Initially, I expected to accept arbitrary edn::Value instances in the
value position, and to differentiate in the transactor. However, the
implementation quickly became a two-stage parser, since we always
wanted to parse the resulting value position into some other known
thing using the tx-parser. To save calls into the parser and to allow
the parser to move forward with a smaller API surface, I push as much
of this parsing as possible into the initial parse.
* Pre: Modernize entities().
* Pre: Quote edn::Value::Text in Display.
* Review comment: Add and use edn::Value::into_atom.
* Review comment: Use skip(eof()) throughout.
* Review comment: VecDeque instead of Vec.
* Review comment: Part 0: Rename TempId to TempIdHandle.
* Review comment: Part 1: Differentiate internal and external tempids.
This breaks an abstraction boundary by pushing the Internal/External
split up to the Entity level in tx/ and tx-parser/. This just makes
it easier to explode Entity map notation instances into Entity
instances, taking an existing External tempid :db/id or generating a
new Internal tempid as appropriate. To do this without breaking the
abstraction boundary would require adding flexibility to the
transaction processor: we'd need to be able to turn Entity instances
into some internal enum and handle the two cases independently. It
wouldn't be too hard, but this reduces the combinatorial type
explosion.
2017-03-27 23:30:04 +00:00
|
|
|
#[derive(Clone, Debug, Eq, Hash, Ord, PartialOrd, PartialEq)]
|
2017-02-15 00:50:40 +00:00
|
|
|
pub enum Term<E, V> {
|
|
|
|
AddOrRetract(OpType, E, Entid, V),
|
|
|
|
}
|
|
|
|
|
Schema alteration. Fixes #294 and #295. (#370) r=rnewman
* Pre: Don't retract :db/ident in test.
Datomic (and eventually Mentat) don't allow to retract :db/ident in
this way, so this runs afoul of future work to support mutating
metadata.
* Pre: s/VALUETYPE/VALUE_TYPE/.
This is consistent with the capitalization (which is "valueType") and
the other identifier.
* Pre: Remove some single quotes from error output.
* Part 1: Make materialized views be uniform [e a v value_type_tag].
This looks ahead to a time when we could support arbitrary
user-defined materialized views. For now, the "idents" materialized
view is those datoms of the form [e :db/ident :namespaced/keyword] and
the "schema" materialized view is those datoms of the form [e a v]
where a is in a particular set of attributes that will become clear in
the following commits.
This change is not backwards compatible, so I'm removing the open
current (really, v2) test. It'll be re-instated when we get to
https://github.com/mozilla/mentat/issues/194.
* Pre: Map TypedValue::Ref to TypedValue::Keyword in debug output.
* Part 3: Separate `schema_to_mutate` from the `schema` used to interpret.
This is just to keep track of the expected changes during
bootstrapping. I want bootstrap metadata mutations to flow through
the same code path as metadata mutations during regular transactions;
by differentiating the schema used for interpretation from the schema
that will be updated I expect to be able to apply bootstrap metadata
mutations to an empty schema and have things like materialized views
created (using the regular code paths).
This commit has been re-ordered for conceptual clarity, but it won't
compile because it references the metadata module. It's possible to
make it compile -- the functionality is there in the schema module --
but it's not worth the rebasing effort until after review (and
possibly not even then, since we'll squash down to a single commit to
land).
* Part 2: Maintain entids separately from idents.
In order to support historical idents, we need to distinguish the
"current" map from entid -> ident from the "complete historical" map
ident -> entid. This is what Datomic does; in Datomic, an ident is
never retracted (although it can be replaced). This approach is an
important part of allowing multiple consumers to share a schema
fragment as it migrates forward.
This fixes a limitation of the Clojure implementation, which did not
handle historical idents across knowledge base close and re-open.
The "entids" materialized view is naturally a slice of the "datoms"
table. The "idents" materialized view is a slice of the
"transactions" table. I hope that representing in this way, and
casting the problem in this light, might generalize to future
materialized views.
* Pre: Add DiffSet.
* Part 4: Collect mutations to a `Schema`.
I haven't taken your review comment about consuming AttributeBuilder
during each fluent function. If you read my response and still want
this, I'm happy to do it in review.
* Part 5: Handle :db/ident and :db.{install,alter}/attribute.
This "loops" the committed datoms out of the SQL store and back
through the metadata (schema, but in future also partition map)
processor. The metadata processor updates the schema and produces a
report of what changed; that report is then used to update the SQL
store. That update includes:
- the materialized views ("entids", "idents", and "schema");
- if needed, a subset of the datoms themselves (as flags change).
I've left a TODO for handling attribute retraction in the cases that
it makes sense. I expect that to be straight-forward.
* Review comment: Rename DiffSet to AddRetractAlterSet.
Also adds a little more commentary and a simple test.
* Review comment: Use ToIdent trait.
* Review comment: partially revert "Part 2: Maintain entids separately from idents."
This reverts commit 23a91df9c35e14398f2ddbd1ba25315821e67401.
Following our discussion, this removes the "entids" materialized
view. The next commit will remove historical idents from the "idents"
materialized view.
* Post: Use custom Either rather than std::result::Result.
This is not necessary, but it was suggested that we might be paying an
overhead creating Err instances while using error_chain. That seems
not to be the case, but this change shows that we don't actually use
any of the Result helper methods, so there's no reason to overload
Result. This change might avoid some future confusion, so I'm going
to land it anyway.
Signed-off-by: Nick Alexander <nalexander@mozilla.com>
* Review comment: Don't preserve historical idents.
* Review comment: More prepared statements when updating materialized views.
* Post: Test altering :db/cardinality and :db/unique.
These tests fail due to a Datomic limitation, namely that the marker
flag :db.alter/attribute can only be asserted once for an attribute!
That is, [:db.part/db :db.alter/attribute :attribute] will only be
transacted at most once. Since older versions of Datomic required the
:db.alter/attribute flag, I can only imagine they either never wrote
:db.alter/attribute to the store, or they handled it specially. I'll
need to remove the marker flag system from Mentat in order to address
this fundamental limitation.
* Post: Remove some more single quotes from error output.
* Post: Add assert_transact! macro to unwrap safely.
I was finding it very difficult to track unwrapping errors while
making changes, due to an underlying Mac OS X symbolication issue that
makes running tests with RUST_BACKTRACE=1 so slow that they all time
out.
* Post: Don't expect or recognize :db.{install,alter}/attribute.
I had this all working... except we will never see a repeated
`[:db.part/db :db.alter/attribute :attribute]` assertion in the store!
That means my approach would let you alter an attribute at most one
time. It's not worth hacking around this; it's better to just stop
expecting (and recognizing) the marker flags. (We have all the data
to distinguish the various cases that we need without the marker
flags.)
This brings Mentat in line with the thrust of newer Datomic versions,
but isn't compatible with Datomic, because (if I understand correctly)
Datomic automatically adds :db.{install,alter}/attribute assertions to
transactions.
I haven't purged the corresponding :db/ident and schema fragments just
yet:
- we might want them back
- we might want them in order to upgrade v1 and v2 databases to the
new on-disk layout we're fleshing out (v3?).
* Post: Don't make :db/unique :db.unique/* imply :db/index true.
This patch avoids a potential bug with the "schema" materialized view.
If :db/unique :db.unique/value implies :db/index true, then what
happens when you _retract_ :db.unique/value? I think Datomic defines
this in some way, but I really want the "schema" materialized view to
be a slice of "datoms" and not have these sort of ambiguities and
persistent effects. Therefore, to ensure that we don't retract a
schema characteristic and accidentally change more than we intended
to, this patch stops having any schema characteristic imply any other
schema characteristic(s). To achieve that, I added an
Option<Unique::{Value,Identity}> type to Attribute; this helps with
this patch, and also looks ahead to when we allow to retract
:db/unique attributes.
* Post: Allow to retract :db/ident.
* Post: Include more details about invalid schema changes.
The tests use strings, so they hide the chained errors which do in
fact provide more detail.
* Review comment: Fix outdated comment.
* Review comment: s/_SET/_SQL_LIST/.
* Review comment: Use a sub-select for checking cardinality.
This might be faster in practice.
* Review comment: Put `attribute::Unique` into its own namespace.
2017-03-20 20:18:59 +00:00
|
|
|
use self::Either::*;
|
|
|
|
|
2017-05-08 03:00:04 +00:00
|
|
|
pub type KnownEntidOr<T> = Either<KnownEntid, T>;
|
Schema alteration. Fixes #294 and #295. (#370) r=rnewman
* Pre: Don't retract :db/ident in test.
Datomic (and eventually Mentat) don't allow to retract :db/ident in
this way, so this runs afoul of future work to support mutating
metadata.
* Pre: s/VALUETYPE/VALUE_TYPE/.
This is consistent with the capitalization (which is "valueType") and
the other identifier.
* Pre: Remove some single quotes from error output.
* Part 1: Make materialized views be uniform [e a v value_type_tag].
This looks ahead to a time when we could support arbitrary
user-defined materialized views. For now, the "idents" materialized
view is those datoms of the form [e :db/ident :namespaced/keyword] and
the "schema" materialized view is those datoms of the form [e a v]
where a is in a particular set of attributes that will become clear in
the following commits.
This change is not backwards compatible, so I'm removing the open
current (really, v2) test. It'll be re-instated when we get to
https://github.com/mozilla/mentat/issues/194.
* Pre: Map TypedValue::Ref to TypedValue::Keyword in debug output.
* Part 3: Separate `schema_to_mutate` from the `schema` used to interpret.
This is just to keep track of the expected changes during
bootstrapping. I want bootstrap metadata mutations to flow through
the same code path as metadata mutations during regular transactions;
by differentiating the schema used for interpretation from the schema
that will be updated I expect to be able to apply bootstrap metadata
mutations to an empty schema and have things like materialized views
created (using the regular code paths).
This commit has been re-ordered for conceptual clarity, but it won't
compile because it references the metadata module. It's possible to
make it compile -- the functionality is there in the schema module --
but it's not worth the rebasing effort until after review (and
possibly not even then, since we'll squash down to a single commit to
land).
* Part 2: Maintain entids separately from idents.
In order to support historical idents, we need to distinguish the
"current" map from entid -> ident from the "complete historical" map
ident -> entid. This is what Datomic does; in Datomic, an ident is
never retracted (although it can be replaced). This approach is an
important part of allowing multiple consumers to share a schema
fragment as it migrates forward.
This fixes a limitation of the Clojure implementation, which did not
handle historical idents across knowledge base close and re-open.
The "entids" materialized view is naturally a slice of the "datoms"
table. The "idents" materialized view is a slice of the
"transactions" table. I hope that representing in this way, and
casting the problem in this light, might generalize to future
materialized views.
* Pre: Add DiffSet.
* Part 4: Collect mutations to a `Schema`.
I haven't taken your review comment about consuming AttributeBuilder
during each fluent function. If you read my response and still want
this, I'm happy to do it in review.
* Part 5: Handle :db/ident and :db.{install,alter}/attribute.
This "loops" the committed datoms out of the SQL store and back
through the metadata (schema, but in future also partition map)
processor. The metadata processor updates the schema and produces a
report of what changed; that report is then used to update the SQL
store. That update includes:
- the materialized views ("entids", "idents", and "schema");
- if needed, a subset of the datoms themselves (as flags change).
I've left a TODO for handling attribute retraction in the cases that
it makes sense. I expect that to be straight-forward.
* Review comment: Rename DiffSet to AddRetractAlterSet.
Also adds a little more commentary and a simple test.
* Review comment: Use ToIdent trait.
* Review comment: partially revert "Part 2: Maintain entids separately from idents."
This reverts commit 23a91df9c35e14398f2ddbd1ba25315821e67401.
Following our discussion, this removes the "entids" materialized
view. The next commit will remove historical idents from the "idents"
materialized view.
* Post: Use custom Either rather than std::result::Result.
This is not necessary, but it was suggested that we might be paying an
overhead creating Err instances while using error_chain. That seems
not to be the case, but this change shows that we don't actually use
any of the Result helper methods, so there's no reason to overload
Result. This change might avoid some future confusion, so I'm going
to land it anyway.
Signed-off-by: Nick Alexander <nalexander@mozilla.com>
* Review comment: Don't preserve historical idents.
* Review comment: More prepared statements when updating materialized views.
* Post: Test altering :db/cardinality and :db/unique.
These tests fail due to a Datomic limitation, namely that the marker
flag :db.alter/attribute can only be asserted once for an attribute!
That is, [:db.part/db :db.alter/attribute :attribute] will only be
transacted at most once. Since older versions of Datomic required the
:db.alter/attribute flag, I can only imagine they either never wrote
:db.alter/attribute to the store, or they handled it specially. I'll
need to remove the marker flag system from Mentat in order to address
this fundamental limitation.
* Post: Remove some more single quotes from error output.
* Post: Add assert_transact! macro to unwrap safely.
I was finding it very difficult to track unwrapping errors while
making changes, due to an underlying Mac OS X symbolication issue that
makes running tests with RUST_BACKTRACE=1 so slow that they all time
out.
* Post: Don't expect or recognize :db.{install,alter}/attribute.
I had this all working... except we will never see a repeated
`[:db.part/db :db.alter/attribute :attribute]` assertion in the store!
That means my approach would let you alter an attribute at most one
time. It's not worth hacking around this; it's better to just stop
expecting (and recognizing) the marker flags. (We have all the data
to distinguish the various cases that we need without the marker
flags.)
This brings Mentat in line with the thrust of newer Datomic versions,
but isn't compatible with Datomic, because (if I understand correctly)
Datomic automatically adds :db.{install,alter}/attribute assertions to
transactions.
I haven't purged the corresponding :db/ident and schema fragments just
yet:
- we might want them back
- we might want them in order to upgrade v1 and v2 databases to the
new on-disk layout we're fleshing out (v3?).
* Post: Don't make :db/unique :db.unique/* imply :db/index true.
This patch avoids a potential bug with the "schema" materialized view.
If :db/unique :db.unique/value implies :db/index true, then what
happens when you _retract_ :db.unique/value? I think Datomic defines
this in some way, but I really want the "schema" materialized view to
be a slice of "datoms" and not have these sort of ambiguities and
persistent effects. Therefore, to ensure that we don't retract a
schema characteristic and accidentally change more than we intended
to, this patch stops having any schema characteristic imply any other
schema characteristic(s). To achieve that, I added an
Option<Unique::{Value,Identity}> type to Attribute; this helps with
this patch, and also looks ahead to when we allow to retract
:db/unique attributes.
* Post: Allow to retract :db/ident.
* Post: Include more details about invalid schema changes.
The tests use strings, so they hide the chained errors which do in
fact provide more detail.
* Review comment: Fix outdated comment.
* Review comment: s/_SET/_SQL_LIST/.
* Review comment: Use a sub-select for checking cardinality.
This might be faster in practice.
* Review comment: Put `attribute::Unique` into its own namespace.
2017-03-20 20:18:59 +00:00
|
|
|
pub type TypedValueOr<T> = Either<TypedValue, T>;
|
2017-02-15 00:50:40 +00:00
|
|
|
|
Lookup refs, nested vector values, map notation. Fixes #180, fixes #183, fixes #284. (#382) r=rnewman
* Pre: Fix error in parser macros.
* Pre: Make test unwrapping more verbose.
* Pre: Make lookup refs be (lookup-ref a v) in the entity position.
This has the advantage of being explicit in all situations and
unambiguous at parse-time. This choice agrees with the Clojure
implementation but not with Datomic. Datomic treats [a v] as a lookup
ref, is ambiguous at parse-time, and is disambiguated in ways I do not
understand at transaction time. We mooted making lookup refs [[a v]]
and outlawing nested value vectors in transactions, but after
implementing that approach I decided it was better to handle lookup
refs at parse time and therefore outlawing nested value vectors is not
necessary.
* Handle lookup refs in the entity and value columns. Fixes #183.
* Pre 0a: Use a stack instead of into_iter.
* Pre 0b: Dedent.
* Pre 0c: Handle `e` after `v`.
This allows to use the original `e` while handling `v`.
* Explode value lists for :db.cardinality/many attributes. Fixes #284.
* Parse and accept map notation. Fixes #180.
* Pre: Modernize add() and retract() into one add_or_retract().
* Pre: Add is_collection and is_atom to edn::Value.
* Pre: Differentiate atoms from lookup-refs in value position.
Initially, I expected to accept arbitrary edn::Value instances in the
value position, and to differentiate in the transactor. However, the
implementation quickly became a two-stage parser, since we always
wanted to parse the resulting value position into some other known
thing using the tx-parser. To save calls into the parser and to allow
the parser to move forward with a smaller API surface, I push as much
of this parsing as possible into the initial parse.
* Pre: Modernize entities().
* Pre: Quote edn::Value::Text in Display.
* Review comment: Add and use edn::Value::into_atom.
* Review comment: Use skip(eof()) throughout.
* Review comment: VecDeque instead of Vec.
* Review comment: Part 0: Rename TempId to TempIdHandle.
* Review comment: Part 1: Differentiate internal and external tempids.
This breaks an abstraction boundary by pushing the Internal/External
split up to the Entity level in tx/ and tx-parser/. This just makes
it easier to explode Entity map notation instances into Entity
instances, taking an existing External tempid :db/id or generating a
new Internal tempid as appropriate. To do this without breaking the
abstraction boundary would require adding flexibility to the
transaction processor: we'd need to be able to turn Entity instances
into some internal enum and handle the two cases independently. It
wouldn't be too hard, but this reduces the combinatorial type
explosion.
2017-03-27 23:30:04 +00:00
|
|
|
pub type TempIdHandle = Rc<TempId>;
|
2017-05-08 03:00:04 +00:00
|
|
|
pub type TempIdMap = HashMap<TempIdHandle, KnownEntid>;
|
2017-02-15 00:50:40 +00:00
|
|
|
|
|
|
|
pub type LookupRef = Rc<AVPair>;
|
|
|
|
|
|
|
|
/// Internal representation of an entid on its way to resolution. We either have the simple case (a
|
|
|
|
/// numeric entid), a lookup-ref that still needs to be resolved (an atomized [a v] pair), or a temp
|
|
|
|
/// ID that needs to be upserted or allocated (an atomized tempid).
|
|
|
|
#[derive(Clone,Debug,Eq,Hash,Ord,PartialOrd,PartialEq)]
|
|
|
|
pub enum LookupRefOrTempId {
|
|
|
|
LookupRef(LookupRef),
|
Lookup refs, nested vector values, map notation. Fixes #180, fixes #183, fixes #284. (#382) r=rnewman
* Pre: Fix error in parser macros.
* Pre: Make test unwrapping more verbose.
* Pre: Make lookup refs be (lookup-ref a v) in the entity position.
This has the advantage of being explicit in all situations and
unambiguous at parse-time. This choice agrees with the Clojure
implementation but not with Datomic. Datomic treats [a v] as a lookup
ref, is ambiguous at parse-time, and is disambiguated in ways I do not
understand at transaction time. We mooted making lookup refs [[a v]]
and outlawing nested value vectors in transactions, but after
implementing that approach I decided it was better to handle lookup
refs at parse time and therefore outlawing nested value vectors is not
necessary.
* Handle lookup refs in the entity and value columns. Fixes #183.
* Pre 0a: Use a stack instead of into_iter.
* Pre 0b: Dedent.
* Pre 0c: Handle `e` after `v`.
This allows to use the original `e` while handling `v`.
* Explode value lists for :db.cardinality/many attributes. Fixes #284.
* Parse and accept map notation. Fixes #180.
* Pre: Modernize add() and retract() into one add_or_retract().
* Pre: Add is_collection and is_atom to edn::Value.
* Pre: Differentiate atoms from lookup-refs in value position.
Initially, I expected to accept arbitrary edn::Value instances in the
value position, and to differentiate in the transactor. However, the
implementation quickly became a two-stage parser, since we always
wanted to parse the resulting value position into some other known
thing using the tx-parser. To save calls into the parser and to allow
the parser to move forward with a smaller API surface, I push as much
of this parsing as possible into the initial parse.
* Pre: Modernize entities().
* Pre: Quote edn::Value::Text in Display.
* Review comment: Add and use edn::Value::into_atom.
* Review comment: Use skip(eof()) throughout.
* Review comment: VecDeque instead of Vec.
* Review comment: Part 0: Rename TempId to TempIdHandle.
* Review comment: Part 1: Differentiate internal and external tempids.
This breaks an abstraction boundary by pushing the Internal/External
split up to the Entity level in tx/ and tx-parser/. This just makes
it easier to explode Entity map notation instances into Entity
instances, taking an existing External tempid :db/id or generating a
new Internal tempid as appropriate. To do this without breaking the
abstraction boundary would require adding flexibility to the
transaction processor: we'd need to be able to turn Entity instances
into some internal enum and handle the two cases independently. It
wouldn't be too hard, but this reduces the combinatorial type
explosion.
2017-03-27 23:30:04 +00:00
|
|
|
TempId(TempIdHandle)
|
2017-02-15 00:50:40 +00:00
|
|
|
}
|
|
|
|
|
2017-05-08 03:00:04 +00:00
|
|
|
pub type TermWithTempIdsAndLookupRefs = Term<KnownEntidOr<LookupRefOrTempId>, TypedValueOr<LookupRefOrTempId>>;
|
|
|
|
pub type TermWithTempIds = Term<KnownEntidOr<TempIdHandle>, TypedValueOr<TempIdHandle>>;
|
|
|
|
pub type TermWithoutTempIds = Term<KnownEntid, TypedValue>;
|
2017-02-15 00:50:40 +00:00
|
|
|
pub type Population = Vec<TermWithTempIds>;
|
|
|
|
|
|
|
|
impl TermWithTempIds {
|
|
|
|
// These have no tempids by definition, and just need to be unwrapped. This operation might
|
|
|
|
// also be called "lowering" or "level lowering", but the concept of "unwrapping" is common in
|
|
|
|
// Rust and seems appropriate here.
|
2018-03-06 17:03:00 +00:00
|
|
|
pub(crate) fn unwrap(self) -> TermWithoutTempIds {
|
2017-02-15 00:50:40 +00:00
|
|
|
match self {
|
Schema alteration. Fixes #294 and #295. (#370) r=rnewman
* Pre: Don't retract :db/ident in test.
Datomic (and eventually Mentat) don't allow to retract :db/ident in
this way, so this runs afoul of future work to support mutating
metadata.
* Pre: s/VALUETYPE/VALUE_TYPE/.
This is consistent with the capitalization (which is "valueType") and
the other identifier.
* Pre: Remove some single quotes from error output.
* Part 1: Make materialized views be uniform [e a v value_type_tag].
This looks ahead to a time when we could support arbitrary
user-defined materialized views. For now, the "idents" materialized
view is those datoms of the form [e :db/ident :namespaced/keyword] and
the "schema" materialized view is those datoms of the form [e a v]
where a is in a particular set of attributes that will become clear in
the following commits.
This change is not backwards compatible, so I'm removing the open
current (really, v2) test. It'll be re-instated when we get to
https://github.com/mozilla/mentat/issues/194.
* Pre: Map TypedValue::Ref to TypedValue::Keyword in debug output.
* Part 3: Separate `schema_to_mutate` from the `schema` used to interpret.
This is just to keep track of the expected changes during
bootstrapping. I want bootstrap metadata mutations to flow through
the same code path as metadata mutations during regular transactions;
by differentiating the schema used for interpretation from the schema
that will be updated I expect to be able to apply bootstrap metadata
mutations to an empty schema and have things like materialized views
created (using the regular code paths).
This commit has been re-ordered for conceptual clarity, but it won't
compile because it references the metadata module. It's possible to
make it compile -- the functionality is there in the schema module --
but it's not worth the rebasing effort until after review (and
possibly not even then, since we'll squash down to a single commit to
land).
* Part 2: Maintain entids separately from idents.
In order to support historical idents, we need to distinguish the
"current" map from entid -> ident from the "complete historical" map
ident -> entid. This is what Datomic does; in Datomic, an ident is
never retracted (although it can be replaced). This approach is an
important part of allowing multiple consumers to share a schema
fragment as it migrates forward.
This fixes a limitation of the Clojure implementation, which did not
handle historical idents across knowledge base close and re-open.
The "entids" materialized view is naturally a slice of the "datoms"
table. The "idents" materialized view is a slice of the
"transactions" table. I hope that representing in this way, and
casting the problem in this light, might generalize to future
materialized views.
* Pre: Add DiffSet.
* Part 4: Collect mutations to a `Schema`.
I haven't taken your review comment about consuming AttributeBuilder
during each fluent function. If you read my response and still want
this, I'm happy to do it in review.
* Part 5: Handle :db/ident and :db.{install,alter}/attribute.
This "loops" the committed datoms out of the SQL store and back
through the metadata (schema, but in future also partition map)
processor. The metadata processor updates the schema and produces a
report of what changed; that report is then used to update the SQL
store. That update includes:
- the materialized views ("entids", "idents", and "schema");
- if needed, a subset of the datoms themselves (as flags change).
I've left a TODO for handling attribute retraction in the cases that
it makes sense. I expect that to be straight-forward.
* Review comment: Rename DiffSet to AddRetractAlterSet.
Also adds a little more commentary and a simple test.
* Review comment: Use ToIdent trait.
* Review comment: partially revert "Part 2: Maintain entids separately from idents."
This reverts commit 23a91df9c35e14398f2ddbd1ba25315821e67401.
Following our discussion, this removes the "entids" materialized
view. The next commit will remove historical idents from the "idents"
materialized view.
* Post: Use custom Either rather than std::result::Result.
This is not necessary, but it was suggested that we might be paying an
overhead creating Err instances while using error_chain. That seems
not to be the case, but this change shows that we don't actually use
any of the Result helper methods, so there's no reason to overload
Result. This change might avoid some future confusion, so I'm going
to land it anyway.
Signed-off-by: Nick Alexander <nalexander@mozilla.com>
* Review comment: Don't preserve historical idents.
* Review comment: More prepared statements when updating materialized views.
* Post: Test altering :db/cardinality and :db/unique.
These tests fail due to a Datomic limitation, namely that the marker
flag :db.alter/attribute can only be asserted once for an attribute!
That is, [:db.part/db :db.alter/attribute :attribute] will only be
transacted at most once. Since older versions of Datomic required the
:db.alter/attribute flag, I can only imagine they either never wrote
:db.alter/attribute to the store, or they handled it specially. I'll
need to remove the marker flag system from Mentat in order to address
this fundamental limitation.
* Post: Remove some more single quotes from error output.
* Post: Add assert_transact! macro to unwrap safely.
I was finding it very difficult to track unwrapping errors while
making changes, due to an underlying Mac OS X symbolication issue that
makes running tests with RUST_BACKTRACE=1 so slow that they all time
out.
* Post: Don't expect or recognize :db.{install,alter}/attribute.
I had this all working... except we will never see a repeated
`[:db.part/db :db.alter/attribute :attribute]` assertion in the store!
That means my approach would let you alter an attribute at most one
time. It's not worth hacking around this; it's better to just stop
expecting (and recognizing) the marker flags. (We have all the data
to distinguish the various cases that we need without the marker
flags.)
This brings Mentat in line with the thrust of newer Datomic versions,
but isn't compatible with Datomic, because (if I understand correctly)
Datomic automatically adds :db.{install,alter}/attribute assertions to
transactions.
I haven't purged the corresponding :db/ident and schema fragments just
yet:
- we might want them back
- we might want them in order to upgrade v1 and v2 databases to the
new on-disk layout we're fleshing out (v3?).
* Post: Don't make :db/unique :db.unique/* imply :db/index true.
This patch avoids a potential bug with the "schema" materialized view.
If :db/unique :db.unique/value implies :db/index true, then what
happens when you _retract_ :db.unique/value? I think Datomic defines
this in some way, but I really want the "schema" materialized view to
be a slice of "datoms" and not have these sort of ambiguities and
persistent effects. Therefore, to ensure that we don't retract a
schema characteristic and accidentally change more than we intended
to, this patch stops having any schema characteristic imply any other
schema characteristic(s). To achieve that, I added an
Option<Unique::{Value,Identity}> type to Attribute; this helps with
this patch, and also looks ahead to when we allow to retract
:db/unique attributes.
* Post: Allow to retract :db/ident.
* Post: Include more details about invalid schema changes.
The tests use strings, so they hide the chained errors which do in
fact provide more detail.
* Review comment: Fix outdated comment.
* Review comment: s/_SET/_SQL_LIST/.
* Review comment: Use a sub-select for checking cardinality.
This might be faster in practice.
* Review comment: Put `attribute::Unique` into its own namespace.
2017-03-20 20:18:59 +00:00
|
|
|
Term::AddOrRetract(op, Left(n), a, Left(v)) => Term::AddOrRetract(op, n, a, v),
|
2017-02-15 00:50:40 +00:00
|
|
|
_ => unreachable!(),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2017-05-08 03:00:04 +00:00
|
|
|
/// Given a `KnownEntidOr` or a `TypedValueOr`, replace any internal `LookupRef` with the entid from
|
2017-02-15 00:50:40 +00:00
|
|
|
/// the given map. Fail if any `LookupRef` cannot be replaced.
|
|
|
|
///
|
|
|
|
/// `lift` allows to specify how the entid found is mapped into the output type. (This could
|
|
|
|
/// also be an `Into` or `From` requirement.)
|
|
|
|
///
|
|
|
|
/// The reason for this awkward expression is that we're parameterizing over the _type constructor_
|
|
|
|
/// (`EntidOr` or `TypedValueOr`), which is not trivial to express in Rust. This only works because
|
|
|
|
/// they're both the same `Result<...>` type with different parameterizations.
|
Lookup refs, nested vector values, map notation. Fixes #180, fixes #183, fixes #284. (#382) r=rnewman
* Pre: Fix error in parser macros.
* Pre: Make test unwrapping more verbose.
* Pre: Make lookup refs be (lookup-ref a v) in the entity position.
This has the advantage of being explicit in all situations and
unambiguous at parse-time. This choice agrees with the Clojure
implementation but not with Datomic. Datomic treats [a v] as a lookup
ref, is ambiguous at parse-time, and is disambiguated in ways I do not
understand at transaction time. We mooted making lookup refs [[a v]]
and outlawing nested value vectors in transactions, but after
implementing that approach I decided it was better to handle lookup
refs at parse time and therefore outlawing nested value vectors is not
necessary.
* Handle lookup refs in the entity and value columns. Fixes #183.
* Pre 0a: Use a stack instead of into_iter.
* Pre 0b: Dedent.
* Pre 0c: Handle `e` after `v`.
This allows to use the original `e` while handling `v`.
* Explode value lists for :db.cardinality/many attributes. Fixes #284.
* Parse and accept map notation. Fixes #180.
* Pre: Modernize add() and retract() into one add_or_retract().
* Pre: Add is_collection and is_atom to edn::Value.
* Pre: Differentiate atoms from lookup-refs in value position.
Initially, I expected to accept arbitrary edn::Value instances in the
value position, and to differentiate in the transactor. However, the
implementation quickly became a two-stage parser, since we always
wanted to parse the resulting value position into some other known
thing using the tx-parser. To save calls into the parser and to allow
the parser to move forward with a smaller API surface, I push as much
of this parsing as possible into the initial parse.
* Pre: Modernize entities().
* Pre: Quote edn::Value::Text in Display.
* Review comment: Add and use edn::Value::into_atom.
* Review comment: Use skip(eof()) throughout.
* Review comment: VecDeque instead of Vec.
* Review comment: Part 0: Rename TempId to TempIdHandle.
* Review comment: Part 1: Differentiate internal and external tempids.
This breaks an abstraction boundary by pushing the Internal/External
split up to the Entity level in tx/ and tx-parser/. This just makes
it easier to explode Entity map notation instances into Entity
instances, taking an existing External tempid :db/id or generating a
new Internal tempid as appropriate. To do this without breaking the
abstraction boundary would require adding flexibility to the
transaction processor: we'd need to be able to turn Entity instances
into some internal enum and handle the two cases independently. It
wouldn't be too hard, but this reduces the combinatorial type
explosion.
2017-03-27 23:30:04 +00:00
|
|
|
pub fn replace_lookup_ref<T, U>(lookup_map: &AVMap, desired_or: Either<T, LookupRefOrTempId>, lift: U) -> errors::Result<Either<T, TempIdHandle>> where U: FnOnce(Entid) -> T {
|
2017-02-15 00:50:40 +00:00
|
|
|
match desired_or {
|
Schema alteration. Fixes #294 and #295. (#370) r=rnewman
* Pre: Don't retract :db/ident in test.
Datomic (and eventually Mentat) don't allow to retract :db/ident in
this way, so this runs afoul of future work to support mutating
metadata.
* Pre: s/VALUETYPE/VALUE_TYPE/.
This is consistent with the capitalization (which is "valueType") and
the other identifier.
* Pre: Remove some single quotes from error output.
* Part 1: Make materialized views be uniform [e a v value_type_tag].
This looks ahead to a time when we could support arbitrary
user-defined materialized views. For now, the "idents" materialized
view is those datoms of the form [e :db/ident :namespaced/keyword] and
the "schema" materialized view is those datoms of the form [e a v]
where a is in a particular set of attributes that will become clear in
the following commits.
This change is not backwards compatible, so I'm removing the open
current (really, v2) test. It'll be re-instated when we get to
https://github.com/mozilla/mentat/issues/194.
* Pre: Map TypedValue::Ref to TypedValue::Keyword in debug output.
* Part 3: Separate `schema_to_mutate` from the `schema` used to interpret.
This is just to keep track of the expected changes during
bootstrapping. I want bootstrap metadata mutations to flow through
the same code path as metadata mutations during regular transactions;
by differentiating the schema used for interpretation from the schema
that will be updated I expect to be able to apply bootstrap metadata
mutations to an empty schema and have things like materialized views
created (using the regular code paths).
This commit has been re-ordered for conceptual clarity, but it won't
compile because it references the metadata module. It's possible to
make it compile -- the functionality is there in the schema module --
but it's not worth the rebasing effort until after review (and
possibly not even then, since we'll squash down to a single commit to
land).
* Part 2: Maintain entids separately from idents.
In order to support historical idents, we need to distinguish the
"current" map from entid -> ident from the "complete historical" map
ident -> entid. This is what Datomic does; in Datomic, an ident is
never retracted (although it can be replaced). This approach is an
important part of allowing multiple consumers to share a schema
fragment as it migrates forward.
This fixes a limitation of the Clojure implementation, which did not
handle historical idents across knowledge base close and re-open.
The "entids" materialized view is naturally a slice of the "datoms"
table. The "idents" materialized view is a slice of the
"transactions" table. I hope that representing in this way, and
casting the problem in this light, might generalize to future
materialized views.
* Pre: Add DiffSet.
* Part 4: Collect mutations to a `Schema`.
I haven't taken your review comment about consuming AttributeBuilder
during each fluent function. If you read my response and still want
this, I'm happy to do it in review.
* Part 5: Handle :db/ident and :db.{install,alter}/attribute.
This "loops" the committed datoms out of the SQL store and back
through the metadata (schema, but in future also partition map)
processor. The metadata processor updates the schema and produces a
report of what changed; that report is then used to update the SQL
store. That update includes:
- the materialized views ("entids", "idents", and "schema");
- if needed, a subset of the datoms themselves (as flags change).
I've left a TODO for handling attribute retraction in the cases that
it makes sense. I expect that to be straight-forward.
* Review comment: Rename DiffSet to AddRetractAlterSet.
Also adds a little more commentary and a simple test.
* Review comment: Use ToIdent trait.
* Review comment: partially revert "Part 2: Maintain entids separately from idents."
This reverts commit 23a91df9c35e14398f2ddbd1ba25315821e67401.
Following our discussion, this removes the "entids" materialized
view. The next commit will remove historical idents from the "idents"
materialized view.
* Post: Use custom Either rather than std::result::Result.
This is not necessary, but it was suggested that we might be paying an
overhead creating Err instances while using error_chain. That seems
not to be the case, but this change shows that we don't actually use
any of the Result helper methods, so there's no reason to overload
Result. This change might avoid some future confusion, so I'm going
to land it anyway.
Signed-off-by: Nick Alexander <nalexander@mozilla.com>
* Review comment: Don't preserve historical idents.
* Review comment: More prepared statements when updating materialized views.
* Post: Test altering :db/cardinality and :db/unique.
These tests fail due to a Datomic limitation, namely that the marker
flag :db.alter/attribute can only be asserted once for an attribute!
That is, [:db.part/db :db.alter/attribute :attribute] will only be
transacted at most once. Since older versions of Datomic required the
:db.alter/attribute flag, I can only imagine they either never wrote
:db.alter/attribute to the store, or they handled it specially. I'll
need to remove the marker flag system from Mentat in order to address
this fundamental limitation.
* Post: Remove some more single quotes from error output.
* Post: Add assert_transact! macro to unwrap safely.
I was finding it very difficult to track unwrapping errors while
making changes, due to an underlying Mac OS X symbolication issue that
makes running tests with RUST_BACKTRACE=1 so slow that they all time
out.
* Post: Don't expect or recognize :db.{install,alter}/attribute.
I had this all working... except we will never see a repeated
`[:db.part/db :db.alter/attribute :attribute]` assertion in the store!
That means my approach would let you alter an attribute at most one
time. It's not worth hacking around this; it's better to just stop
expecting (and recognizing) the marker flags. (We have all the data
to distinguish the various cases that we need without the marker
flags.)
This brings Mentat in line with the thrust of newer Datomic versions,
but isn't compatible with Datomic, because (if I understand correctly)
Datomic automatically adds :db.{install,alter}/attribute assertions to
transactions.
I haven't purged the corresponding :db/ident and schema fragments just
yet:
- we might want them back
- we might want them in order to upgrade v1 and v2 databases to the
new on-disk layout we're fleshing out (v3?).
* Post: Don't make :db/unique :db.unique/* imply :db/index true.
This patch avoids a potential bug with the "schema" materialized view.
If :db/unique :db.unique/value implies :db/index true, then what
happens when you _retract_ :db.unique/value? I think Datomic defines
this in some way, but I really want the "schema" materialized view to
be a slice of "datoms" and not have these sort of ambiguities and
persistent effects. Therefore, to ensure that we don't retract a
schema characteristic and accidentally change more than we intended
to, this patch stops having any schema characteristic imply any other
schema characteristic(s). To achieve that, I added an
Option<Unique::{Value,Identity}> type to Attribute; this helps with
this patch, and also looks ahead to when we allow to retract
:db/unique attributes.
* Post: Allow to retract :db/ident.
* Post: Include more details about invalid schema changes.
The tests use strings, so they hide the chained errors which do in
fact provide more detail.
* Review comment: Fix outdated comment.
* Review comment: s/_SET/_SQL_LIST/.
* Review comment: Use a sub-select for checking cardinality.
This might be faster in practice.
* Review comment: Put `attribute::Unique` into its own namespace.
2017-03-20 20:18:59 +00:00
|
|
|
Left(desired) => Ok(Left(desired)), // N.b., must unwrap here -- the ::Left types are different!
|
|
|
|
Right(other) => {
|
2017-02-15 00:50:40 +00:00
|
|
|
match other {
|
Schema alteration. Fixes #294 and #295. (#370) r=rnewman
* Pre: Don't retract :db/ident in test.
Datomic (and eventually Mentat) don't allow to retract :db/ident in
this way, so this runs afoul of future work to support mutating
metadata.
* Pre: s/VALUETYPE/VALUE_TYPE/.
This is consistent with the capitalization (which is "valueType") and
the other identifier.
* Pre: Remove some single quotes from error output.
* Part 1: Make materialized views be uniform [e a v value_type_tag].
This looks ahead to a time when we could support arbitrary
user-defined materialized views. For now, the "idents" materialized
view is those datoms of the form [e :db/ident :namespaced/keyword] and
the "schema" materialized view is those datoms of the form [e a v]
where a is in a particular set of attributes that will become clear in
the following commits.
This change is not backwards compatible, so I'm removing the open
current (really, v2) test. It'll be re-instated when we get to
https://github.com/mozilla/mentat/issues/194.
* Pre: Map TypedValue::Ref to TypedValue::Keyword in debug output.
* Part 3: Separate `schema_to_mutate` from the `schema` used to interpret.
This is just to keep track of the expected changes during
bootstrapping. I want bootstrap metadata mutations to flow through
the same code path as metadata mutations during regular transactions;
by differentiating the schema used for interpretation from the schema
that will be updated I expect to be able to apply bootstrap metadata
mutations to an empty schema and have things like materialized views
created (using the regular code paths).
This commit has been re-ordered for conceptual clarity, but it won't
compile because it references the metadata module. It's possible to
make it compile -- the functionality is there in the schema module --
but it's not worth the rebasing effort until after review (and
possibly not even then, since we'll squash down to a single commit to
land).
* Part 2: Maintain entids separately from idents.
In order to support historical idents, we need to distinguish the
"current" map from entid -> ident from the "complete historical" map
ident -> entid. This is what Datomic does; in Datomic, an ident is
never retracted (although it can be replaced). This approach is an
important part of allowing multiple consumers to share a schema
fragment as it migrates forward.
This fixes a limitation of the Clojure implementation, which did not
handle historical idents across knowledge base close and re-open.
The "entids" materialized view is naturally a slice of the "datoms"
table. The "idents" materialized view is a slice of the
"transactions" table. I hope that representing in this way, and
casting the problem in this light, might generalize to future
materialized views.
* Pre: Add DiffSet.
* Part 4: Collect mutations to a `Schema`.
I haven't taken your review comment about consuming AttributeBuilder
during each fluent function. If you read my response and still want
this, I'm happy to do it in review.
* Part 5: Handle :db/ident and :db.{install,alter}/attribute.
This "loops" the committed datoms out of the SQL store and back
through the metadata (schema, but in future also partition map)
processor. The metadata processor updates the schema and produces a
report of what changed; that report is then used to update the SQL
store. That update includes:
- the materialized views ("entids", "idents", and "schema");
- if needed, a subset of the datoms themselves (as flags change).
I've left a TODO for handling attribute retraction in the cases that
it makes sense. I expect that to be straight-forward.
* Review comment: Rename DiffSet to AddRetractAlterSet.
Also adds a little more commentary and a simple test.
* Review comment: Use ToIdent trait.
* Review comment: partially revert "Part 2: Maintain entids separately from idents."
This reverts commit 23a91df9c35e14398f2ddbd1ba25315821e67401.
Following our discussion, this removes the "entids" materialized
view. The next commit will remove historical idents from the "idents"
materialized view.
* Post: Use custom Either rather than std::result::Result.
This is not necessary, but it was suggested that we might be paying an
overhead creating Err instances while using error_chain. That seems
not to be the case, but this change shows that we don't actually use
any of the Result helper methods, so there's no reason to overload
Result. This change might avoid some future confusion, so I'm going
to land it anyway.
Signed-off-by: Nick Alexander <nalexander@mozilla.com>
* Review comment: Don't preserve historical idents.
* Review comment: More prepared statements when updating materialized views.
* Post: Test altering :db/cardinality and :db/unique.
These tests fail due to a Datomic limitation, namely that the marker
flag :db.alter/attribute can only be asserted once for an attribute!
That is, [:db.part/db :db.alter/attribute :attribute] will only be
transacted at most once. Since older versions of Datomic required the
:db.alter/attribute flag, I can only imagine they either never wrote
:db.alter/attribute to the store, or they handled it specially. I'll
need to remove the marker flag system from Mentat in order to address
this fundamental limitation.
* Post: Remove some more single quotes from error output.
* Post: Add assert_transact! macro to unwrap safely.
I was finding it very difficult to track unwrapping errors while
making changes, due to an underlying Mac OS X symbolication issue that
makes running tests with RUST_BACKTRACE=1 so slow that they all time
out.
* Post: Don't expect or recognize :db.{install,alter}/attribute.
I had this all working... except we will never see a repeated
`[:db.part/db :db.alter/attribute :attribute]` assertion in the store!
That means my approach would let you alter an attribute at most one
time. It's not worth hacking around this; it's better to just stop
expecting (and recognizing) the marker flags. (We have all the data
to distinguish the various cases that we need without the marker
flags.)
This brings Mentat in line with the thrust of newer Datomic versions,
but isn't compatible with Datomic, because (if I understand correctly)
Datomic automatically adds :db.{install,alter}/attribute assertions to
transactions.
I haven't purged the corresponding :db/ident and schema fragments just
yet:
- we might want them back
- we might want them in order to upgrade v1 and v2 databases to the
new on-disk layout we're fleshing out (v3?).
* Post: Don't make :db/unique :db.unique/* imply :db/index true.
This patch avoids a potential bug with the "schema" materialized view.
If :db/unique :db.unique/value implies :db/index true, then what
happens when you _retract_ :db.unique/value? I think Datomic defines
this in some way, but I really want the "schema" materialized view to
be a slice of "datoms" and not have these sort of ambiguities and
persistent effects. Therefore, to ensure that we don't retract a
schema characteristic and accidentally change more than we intended
to, this patch stops having any schema characteristic imply any other
schema characteristic(s). To achieve that, I added an
Option<Unique::{Value,Identity}> type to Attribute; this helps with
this patch, and also looks ahead to when we allow to retract
:db/unique attributes.
* Post: Allow to retract :db/ident.
* Post: Include more details about invalid schema changes.
The tests use strings, so they hide the chained errors which do in
fact provide more detail.
* Review comment: Fix outdated comment.
* Review comment: s/_SET/_SQL_LIST/.
* Review comment: Use a sub-select for checking cardinality.
This might be faster in practice.
* Review comment: Put `attribute::Unique` into its own namespace.
2017-03-20 20:18:59 +00:00
|
|
|
LookupRefOrTempId::TempId(t) => Ok(Right(t)),
|
2017-02-15 00:50:40 +00:00
|
|
|
LookupRefOrTempId::LookupRef(av) => lookup_map.get(&*av)
|
Schema alteration. Fixes #294 and #295. (#370) r=rnewman
* Pre: Don't retract :db/ident in test.
Datomic (and eventually Mentat) don't allow to retract :db/ident in
this way, so this runs afoul of future work to support mutating
metadata.
* Pre: s/VALUETYPE/VALUE_TYPE/.
This is consistent with the capitalization (which is "valueType") and
the other identifier.
* Pre: Remove some single quotes from error output.
* Part 1: Make materialized views be uniform [e a v value_type_tag].
This looks ahead to a time when we could support arbitrary
user-defined materialized views. For now, the "idents" materialized
view is those datoms of the form [e :db/ident :namespaced/keyword] and
the "schema" materialized view is those datoms of the form [e a v]
where a is in a particular set of attributes that will become clear in
the following commits.
This change is not backwards compatible, so I'm removing the open
current (really, v2) test. It'll be re-instated when we get to
https://github.com/mozilla/mentat/issues/194.
* Pre: Map TypedValue::Ref to TypedValue::Keyword in debug output.
* Part 3: Separate `schema_to_mutate` from the `schema` used to interpret.
This is just to keep track of the expected changes during
bootstrapping. I want bootstrap metadata mutations to flow through
the same code path as metadata mutations during regular transactions;
by differentiating the schema used for interpretation from the schema
that will be updated I expect to be able to apply bootstrap metadata
mutations to an empty schema and have things like materialized views
created (using the regular code paths).
This commit has been re-ordered for conceptual clarity, but it won't
compile because it references the metadata module. It's possible to
make it compile -- the functionality is there in the schema module --
but it's not worth the rebasing effort until after review (and
possibly not even then, since we'll squash down to a single commit to
land).
* Part 2: Maintain entids separately from idents.
In order to support historical idents, we need to distinguish the
"current" map from entid -> ident from the "complete historical" map
ident -> entid. This is what Datomic does; in Datomic, an ident is
never retracted (although it can be replaced). This approach is an
important part of allowing multiple consumers to share a schema
fragment as it migrates forward.
This fixes a limitation of the Clojure implementation, which did not
handle historical idents across knowledge base close and re-open.
The "entids" materialized view is naturally a slice of the "datoms"
table. The "idents" materialized view is a slice of the
"transactions" table. I hope that representing in this way, and
casting the problem in this light, might generalize to future
materialized views.
* Pre: Add DiffSet.
* Part 4: Collect mutations to a `Schema`.
I haven't taken your review comment about consuming AttributeBuilder
during each fluent function. If you read my response and still want
this, I'm happy to do it in review.
* Part 5: Handle :db/ident and :db.{install,alter}/attribute.
This "loops" the committed datoms out of the SQL store and back
through the metadata (schema, but in future also partition map)
processor. The metadata processor updates the schema and produces a
report of what changed; that report is then used to update the SQL
store. That update includes:
- the materialized views ("entids", "idents", and "schema");
- if needed, a subset of the datoms themselves (as flags change).
I've left a TODO for handling attribute retraction in the cases that
it makes sense. I expect that to be straight-forward.
* Review comment: Rename DiffSet to AddRetractAlterSet.
Also adds a little more commentary and a simple test.
* Review comment: Use ToIdent trait.
* Review comment: partially revert "Part 2: Maintain entids separately from idents."
This reverts commit 23a91df9c35e14398f2ddbd1ba25315821e67401.
Following our discussion, this removes the "entids" materialized
view. The next commit will remove historical idents from the "idents"
materialized view.
* Post: Use custom Either rather than std::result::Result.
This is not necessary, but it was suggested that we might be paying an
overhead creating Err instances while using error_chain. That seems
not to be the case, but this change shows that we don't actually use
any of the Result helper methods, so there's no reason to overload
Result. This change might avoid some future confusion, so I'm going
to land it anyway.
Signed-off-by: Nick Alexander <nalexander@mozilla.com>
* Review comment: Don't preserve historical idents.
* Review comment: More prepared statements when updating materialized views.
* Post: Test altering :db/cardinality and :db/unique.
These tests fail due to a Datomic limitation, namely that the marker
flag :db.alter/attribute can only be asserted once for an attribute!
That is, [:db.part/db :db.alter/attribute :attribute] will only be
transacted at most once. Since older versions of Datomic required the
:db.alter/attribute flag, I can only imagine they either never wrote
:db.alter/attribute to the store, or they handled it specially. I'll
need to remove the marker flag system from Mentat in order to address
this fundamental limitation.
* Post: Remove some more single quotes from error output.
* Post: Add assert_transact! macro to unwrap safely.
I was finding it very difficult to track unwrapping errors while
making changes, due to an underlying Mac OS X symbolication issue that
makes running tests with RUST_BACKTRACE=1 so slow that they all time
out.
* Post: Don't expect or recognize :db.{install,alter}/attribute.
I had this all working... except we will never see a repeated
`[:db.part/db :db.alter/attribute :attribute]` assertion in the store!
That means my approach would let you alter an attribute at most one
time. It's not worth hacking around this; it's better to just stop
expecting (and recognizing) the marker flags. (We have all the data
to distinguish the various cases that we need without the marker
flags.)
This brings Mentat in line with the thrust of newer Datomic versions,
but isn't compatible with Datomic, because (if I understand correctly)
Datomic automatically adds :db.{install,alter}/attribute assertions to
transactions.
I haven't purged the corresponding :db/ident and schema fragments just
yet:
- we might want them back
- we might want them in order to upgrade v1 and v2 databases to the
new on-disk layout we're fleshing out (v3?).
* Post: Don't make :db/unique :db.unique/* imply :db/index true.
This patch avoids a potential bug with the "schema" materialized view.
If :db/unique :db.unique/value implies :db/index true, then what
happens when you _retract_ :db.unique/value? I think Datomic defines
this in some way, but I really want the "schema" materialized view to
be a slice of "datoms" and not have these sort of ambiguities and
persistent effects. Therefore, to ensure that we don't retract a
schema characteristic and accidentally change more than we intended
to, this patch stops having any schema characteristic imply any other
schema characteristic(s). To achieve that, I added an
Option<Unique::{Value,Identity}> type to Attribute; this helps with
this patch, and also looks ahead to when we allow to retract
:db/unique attributes.
* Post: Allow to retract :db/ident.
* Post: Include more details about invalid schema changes.
The tests use strings, so they hide the chained errors which do in
fact provide more detail.
* Review comment: Fix outdated comment.
* Review comment: s/_SET/_SQL_LIST/.
* Review comment: Use a sub-select for checking cardinality.
This might be faster in practice.
* Review comment: Put `attribute::Unique` into its own namespace.
2017-03-20 20:18:59 +00:00
|
|
|
.map(|x| lift(*x)).map(Left)
|
2017-02-15 00:50:40 +00:00
|
|
|
// XXX TODO: fix this error kind!
|
|
|
|
.ok_or_else(|| ErrorKind::UnrecognizedIdent(format!("couldn't lookup [a v]: {:?}", (*av).clone())).into()),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
Add type checking and constraint checking to the transactor. (#663, #532, #679)
This should address #663, by re-inserting type checking in the
transactor stack after the entry point used by the term builder.
Before this commit, we were using an SQLite UNIQUE index to assert
that no `[e a]` pair, with `a` a cardinality one attribute, was
asserted more than once. However, that's not in line with Datomic,
which treats transaction inputs as a set and allows a single datom
like `[e a v]` to appear multiple times. It's both awkward and not
particularly efficient to look for _distinct_ repetitions in SQL, so
we accept some runtime cost in order to check for repetitions in the
transactor. This will allow us to address #532, which is really about
whether we treat inputs as sets. A side benefit is that we can
provide more helpful error messages when the transactor does detect
that the input truly violates the cardinality constraints of the
schema.
This commit builds a trie while error checking and collecting final
terms, which should be fairly efficient. It also allows a simpler
expression of input-provided :db/txInstant datoms, which in turn
uncovered a small issue with the transaction watcher, where-by the
watcher would not see non-input-provided :db/txInstant datoms.
This transition to Datomic-like input-as-set semantics allows us to
address #532. Previously, two tempids that upserted to the same entid
would produce duplicate datoms, and that would have been rejected by
the transactor -- correctly, since we did not allow duplicate datoms
under the input-as-list semantics. With input-as-set semantics,
duplicate datoms are allowed; and that means that we must allow
tempids to be equivalent, i.e., to resolve to the same tempid.
To achieve this, we:
- index the set of tempids
- identify tempid indices that share an upsert
- map tempids to a dense set of contiguous integer labels
We use the well-known union-find algorithm, as implemented by
petgraph, to efficiently manage the set of equivalent tempids.
Along the way, I've fixed and added tests for two small errors in the
transactor. First, don't drop datoms resolved by upsert (#679).
Second, ensure that complex upserts are allocated.
I don't know quite what happened here. The Clojure implementation
correctly kept complex upserts that hadn't resolved as complex
upserts (see
https://github.com/mozilla/mentat/blob/9a9dfb502acf5e4cdb1059d4aac831d7603063c8/src/common/datomish/transact.cljc#L436)
and then allocated complex upserts if they didn't resolve (see
https://github.com/mozilla/mentat/blob/9a9dfb502acf5e4cdb1059d4aac831d7603063c8/src/common/datomish/transact.cljc#L509).
Based on the code comments, I think the Rust implementation must have
incorrectly tried to optimize by handling all complex upserts in at
most a single generation of evolution, and that's just not correct.
We're effectively implementing a topological sort, using very specific
domain knowledge, and its not true that a node in a topological sort
can be considered only once!
2018-04-30 22:16:05 +00:00
|
|
|
|
|
|
|
#[derive(Clone, Debug, Default)]
|
|
|
|
pub(crate) struct AddAndRetract {
|
|
|
|
pub(crate) add: BTreeSet<TypedValue>,
|
|
|
|
pub(crate) retract: BTreeSet<TypedValue>,
|
|
|
|
}
|
|
|
|
|
|
|
|
// A trie-like structure mapping a -> e -> v that prefix compresses and makes uniqueness constraint
|
|
|
|
// checking more efficient. BTree* for deterministic errors.
|
|
|
|
pub(crate) type AEVTrie<'schema> = BTreeMap<(Entid, &'schema Attribute), BTreeMap<Entid, AddAndRetract>>;
|