[meta] Sync prototyping #176

New issue

Open

opened 2020-08-06 16:56:12 +00:00 by gburd · 0 comments

gburd commented

2020-08-06 16:56:12 +00:00

(Migrated from github.com)

We've identified a number of small changes and additions to Mentat in order for it to support sync prototyping. These are:

Switching txes to use a transaction UUID for identification rather than an entid in a specified part. UUIDs are stable and globally unique, which makes them suitable for identifying non-smushable entities. This also opens the part space to be entirely free for user entities. A transaction identifier can now be a distinct type alongside ref and uuid. Naturally some work is needed to allow transactions in the e position. This is a departure from Datomic compatibility. We'll need a transaction sequence number, too, to provide ordering of transactions.
Implementing renumbering of entities and transaction sequence numbers: the shifting of allocated identifiers to avoid inadvertent collision with identifiers allocated on other devices, and the repositioning of transacted data to implement rebasing.
Exposing an operation for raw assertion: asserting datoms with a defined tx and sequence. This is how we'll copy remote transactions into the local store.
Implementing deliberate collision or smushing. Given an identifying attribute :foo/id, a local (renumbered) datom [1234 :foo/id "bar"], and a remote datom [1212 :foo/id "bar"], we want to replace 1234 with 1212 throughout a specified range of transactions. 1234 and 1212 will typically have been allocated since the last sync. This is similar to upserting except we want to take the remote entid, and we'll have less work to do because we already have a fully formed graph.
Compound atomic operations: #489.

With these primitives we'll be able to prototype some kinds of syncing: straightforward replication, rebases, and lossy merges.

There are additional things we might need as we continue to prototype, depending on how we implement syncing:

Representation of parallel timelines and merges
Mechanisms for indicating to consumers that renumbering has occurred
Porting the schema management code

but we'll get to those when we come to them.

We've identified a number of small changes and additions to Mentat in order for it to support sync prototyping. These are: - [ ] Switching txes to use a **transaction UUID** for identification rather than an entid in a specified part. UUIDs are stable and globally unique, which makes them suitable for identifying non-smushable entities. This also opens the part space to be entirely free for user entities. A transaction identifier can now be a distinct type alongside `ref` and `uuid`. Naturally some work is needed to allow transactions in the `e` position. This is a departure from Datomic compatibility. We'll need a transaction sequence number, too, to provide ordering of transactions. - [ ] Implementing **renumbering** of entities and transaction sequence numbers: the shifting of allocated identifiers to avoid inadvertent collision with identifiers allocated on other devices, and the repositioning of transacted data to implement rebasing. - [ ] Exposing an operation for **raw assertion**: asserting datoms with a defined `tx` and sequence. This is how we'll copy remote transactions into the local store. - [ ] Implementing deliberate **collision** or **smushing**. Given an identifying attribute `:foo/id`, a local (renumbered) datom `[1234 :foo/id "bar"]`, and a remote datom `[1212 :foo/id "bar"]`, we want to replace `1234` with `1212` throughout a specified range of transactions. `1234` and `1212` will typically have been allocated since the last sync. This is similar to upserting except we want to take the remote entid, and we'll have less work to do because we already have a fully formed graph. - [ ] Compound atomic operations: #489. With these primitives we'll be able to prototype some kinds of syncing: straightforward replication, rebases, and lossy merges. There are additional things we might need as we continue to prototype, depending on how we implement syncing: - Representation of parallel timelines and merges - Mechanisms for indicating to consumers that renumbering has occurred - Porting the schema management code but we'll get to those when we come to them.