* Pre: remove remnants of 'open_empty'
* Pre: Cleanup 'datoms' table after a timeline move
Since timeline move operations use a transactor, they generate a
"phantom" 'tx' and a 'txInstant' assertion. It is "phantom" in a sense
that it was never present in the 'transactions' table, and is entirely
synthetic as far as our database is concerned.
It's an implementational artifact, and we were not cleaning it up.
It becomes a problem when we start inserting transactions after a move.
Once the transactor clashes with the phantom 'tx', it will retract the
phantom 'txInstant' value, leaving the transactions log in an incorrect state.
This patch adds a test for this scenario and elects the easy way out: simply
remove the offending 'txInstant' datom.
* Part 1: Sync without support for side-effects
A "side-effect" is defined here as a mutation of a remote state as part
of the sync.
If, during a sync we determine that a remote state needs to be changed, bail out.
This generally supports different variations of "baton-passing" syncing, where clients
will succeed syncing if each change is non-conflicting.
* Part 2: Support basic "side-effects" syncing
This patch introduces a concept of a follow-up sync. If a sync generated
a "merge transaction" (a regular transaction that contains assertions
necessary for local and remote transaction logs to converge), then
this transaction needs to be uploaded in a follow-up sync.
Generated SyncReport indicates if a follow-up sync is required.
Follow-up sync itself is just a regular sync. If remote state did not change,
it will result in a simple RemoteFastForward. Otherwise, we'll continue
merging and requesting a follow-up.
Schema alterations are explicitly not supported.
As local transactions are rebased on top of remote, following changes happen:
- entids are changed into tempids, letting transactor upsert :db/unique values
- entids for retractions are changed into lookup-refs if we're confident they'll succeed
-- otherwise, retractions are dropped on the floor
* Post: use a macro for more readable tests
* Tolstoy README
Sync needs to operate over a "mentat transaction", not just a "db transaction".
This shuffle allows internal mentat crates to consume InProgress, which models
the concept of a "mentat transaction".
Being able to derive partition map from partition definitions and current
state of the world (transactions), segmented by timelines, is useful
because it lets us not worry about keeping materialized partition maps
up-to-date - since there's no need for materialized partition maps at that point.
This comes in very handy when we start moving chunks of transactions off of our mainline.
Alternative to this work would look like materializing partition maps per timeline,
growing support for incremental "backwards update" of the materialized maps, etc.
Our core partitions are defined in 'known_parts' table during bootstrap,
and what used to be 'parts' table is a generated view that operates over
transactions to figure out partition index.
'parts' is defined for the main timeline. Querying parts for other timelines
or for particular timeline+tx combinations will look similar.
Normally we want to both materialize our changes (into 'datoms')
as well as commit source transactions into 'transactions' table.
However, when moving transactions from timeline to timeline
we don't want to persist artifacts (rewind assertions), just their
materializations.
This patch expands the 'db' interface to allow for this split,
and changes transactor's functions to take a crate-private 'action'
which defines desired behaviour.
This is necessary for the timelines work ahead. When schema is being
moved off of a main timeline, we need to be able to retract it cleanly.
Retractions are only processed if the whole defining attribute set
is being retracted at once (:db/ident, :db/valueType, :db/cardinality).
Timelines work starts to perform modifications on the partitions
that go beyond simple allocations. This change pre-emptively protects
partition integrity by asserting that index modifications are legal.
Generally, I think that Mentat is using too many small traits rather
than wrapping types into newtypes. Wrapping into newtypes is cheap in
Rust, and it makes it easier to reason about the code.
* Part 1: Extract low-level test framework into mentat_db::debug for re-use.
* Part 2: Improve assert_matches!.
This corrects an incorrect pattern: a conversion method taking &self
but returning an owned value should be named like `to_FOO(&self) -> FOO`. (A
reference-to-reference conversion should be named like `as_FOO(&self)
-> &FOO`. A consuming conversion should be named like `into_FOO(self)
-> FOO`.)
In addition, this pushes the conversion via `to_edn` into the
`assert_matches!` macro, which lets consumers get a real data
structure (say, `Datoms`) and use it directly before or after
`assert_matches!`. (Currently, consumers get back `edn::Value`
instances, which aren't nearly as pleasant to use as real data
structures.)
Co-authored-by: Grisha Kruglov <gkruglov@mozilla.com>
* Part 3: Use mentat_db::debug framework in Tolstoy crate.
The advantage of this approach is that compiling Tolstoy (or anything
that's not db, really) can be quite a bit faster than compiling db.
* Add a top-level "syncable" feature.
Tested with:
cargo test --all
cargo test --all --no-default-features
cargo build --manifest-path tools/cli/Cargo.toml --no-default-features
cargo run --manifest-path tools/cli/Cargo.toml --no-default-features debugcli
Co-authored-by: Nick Alexander <nalexander@mozilla.com>
* Add 'syncable' feature to 'db' crate to conditionally derive serialization for Partition*
This is leading up to syncing with partition support.
This is all part of moving the entity builder away from building term
instances and toward building entity instances. One of the nice
things that the existing term interface does is allow consumers to use
lightweight reference counted tempid handles; I don't want to lose
that, so we'll build it into the entity data structures directly.
We haven't observed performance issues using `Arc` instead of `Rc`,
and we want to be able to include things that are interned (including,
soon, `TempId` instances) in errors coming out of the
transactor. (And `Rc` isn't `Sync`, so it can't be included in errors
directly.)
It's not great to keep lifting functionality higher and higher up the
crate hierarchy, but we really do want to intern while we parse.
Eventually, I expect that we will split the `edn` crate into `types`
and `parsing`, and the `types` crate can depend on a more efficient
interning dependency.
* Delete the (apparently unused) EntId
* Rename edn's Entid to EntidOrIdent to avoid confusion with the Entid that's actually an i64
* Fix travis beta bustage (This is actually unrelated to entids, but is a trivial fix nonetheless)
* Part 3: Parameterize Entity by value type.
This isn't quite right, because after parsing, we shouldn't care
about` `edn::ValueAndSpan`, we should care only about edn::Value.
However, I think we can drop `ValueAndSpan` entirely if we just use
`rust-peg` (and its simpler error messages) rather than a mix of
`rust-peg` and `combine`.
In any case, this paves the way to transacting `Entity<TypedValue>`,
which is a nice step towards building general entities.
* Part 1: Add AttributePlace.
* Part 2: Name other places EntityPlace and ValuePlace.
Now we're consistent and closer to self-documenting. Both matter more
as we expose `Entity` as the thing to build for programmatic usage.
* Part 4: Allow Ident and TempId in ValuePlace.
The parser will never produce these, since determining whether an
integer/keyword or string is an ident or a tempid, respectively, in
the value place requires the schema.
But a builder that produces `Entity` instances directly will want to
produce these.
