This doesn't yet introduce a working Cargo.toml for 'mentatweb', but it
does allow RLS to build correctly without errors, and it reduces the
core library's dependency space, which is more important in the short
term.
* Pre: put query parts in alphabetical order.
* Pre: rename 'input' to 'query' in translate tests.
* Part 1: parse :limit.
* Part 2: validate and escape variable parameters in SQL.
* Part 3: algebrize and translate limits.
* Pre: Expose more in edn.
* Pre: Make it easier to work with ValueAndSpan.
with_spans() is a temporary hack, needed only because I don't care to
parse the bootstrap assertions from text right now.
* Part 1a: Add `value_and_span` for parsing nested `edn::ValueAndSpan` instances.
I wasn't able to abstract over `edn::Value` and `edn::ValueAndSpan`;
there are multiple obstacles. I chose to roll with
`edn::ValueAndSpan` since it exposes the additional span information
that we will want to form good error messages in the future.
* Part 1b: Add keyword_map() parsing an `edn::Value::Vector` into an `edn::Value::map`.
* Part 1c: Add `Log`/`.log(...)` for logging parser progress.
This is a terrible hack, but it sure helps to debug complicated nested
parsers. I don't even know what a principled approach would look
like; since our parser combinators are so frequently expressed in
code, it's hard to imagine a data-driven interpreter that can help
debug things.
* Part 2: Use `value_and_span` apparatus in tx-parser/.
I break an abstraction boundary by returning a value column
`edn::ValueAndSpan` rather than just an `edn::Value`. That is, the
transaction processor shouldn't care where the `edn::Value` it is
processing arose -- even we care to track that information we should
bake it into the `Entity` type. We do this because we need to
dynamically parse the value column to support nested maps, and parsing
requires a full `edn::ValueAndSpan`. Alternately, we could cheat and
fake the spans when parsing nested maps, but that's potentially
expensive.
* Part 3: Use `value_and_span` apparatus in query-parser/.
* Part 4: Use `value_and_span` apparatus in root crate.
* Review comment: Make Span and SpanPosition Copy.
* Review comment: nits.
* Review comment: Make `or` be `or_exactly`.
I baked the eof checking directly into the parser, rather than using
the skip and eof parsers. I also took the time to restore some tests
that were mistakenly commented out.
* Review comment: Extract and use def_matches_* macros.
* Review comment: .map() as late as possible.
Part 1, core: use Rc for String and Keyword.
Part 2, query: use Rc for Variable.
Part 3, sql: use Rc for args in SQLiteQueryBuilder.
Part 4, query-algebrizer: use Rc.
Part 5, db: use Rc.
Part 6, query-parser: use Rc.
Part 7, query-projector: use Rc.
Part 8, query-translator: use Rc.
Part 9, top level: use Rc.
Part 10: intern Ident and IdentOrKeyword.
* Pre: Don't retract :db/ident in test.
Datomic (and eventually Mentat) don't allow to retract :db/ident in
this way, so this runs afoul of future work to support mutating
metadata.
* Pre: s/VALUETYPE/VALUE_TYPE/.
This is consistent with the capitalization (which is "valueType") and
the other identifier.
* Pre: Remove some single quotes from error output.
* Part 1: Make materialized views be uniform [e a v value_type_tag].
This looks ahead to a time when we could support arbitrary
user-defined materialized views. For now, the "idents" materialized
view is those datoms of the form [e :db/ident :namespaced/keyword] and
the "schema" materialized view is those datoms of the form [e a v]
where a is in a particular set of attributes that will become clear in
the following commits.
This change is not backwards compatible, so I'm removing the open
current (really, v2) test. It'll be re-instated when we get to
https://github.com/mozilla/mentat/issues/194.
* Pre: Map TypedValue::Ref to TypedValue::Keyword in debug output.
* Part 3: Separate `schema_to_mutate` from the `schema` used to interpret.
This is just to keep track of the expected changes during
bootstrapping. I want bootstrap metadata mutations to flow through
the same code path as metadata mutations during regular transactions;
by differentiating the schema used for interpretation from the schema
that will be updated I expect to be able to apply bootstrap metadata
mutations to an empty schema and have things like materialized views
created (using the regular code paths).
This commit has been re-ordered for conceptual clarity, but it won't
compile because it references the metadata module. It's possible to
make it compile -- the functionality is there in the schema module --
but it's not worth the rebasing effort until after review (and
possibly not even then, since we'll squash down to a single commit to
land).
* Part 2: Maintain entids separately from idents.
In order to support historical idents, we need to distinguish the
"current" map from entid -> ident from the "complete historical" map
ident -> entid. This is what Datomic does; in Datomic, an ident is
never retracted (although it can be replaced). This approach is an
important part of allowing multiple consumers to share a schema
fragment as it migrates forward.
This fixes a limitation of the Clojure implementation, which did not
handle historical idents across knowledge base close and re-open.
The "entids" materialized view is naturally a slice of the "datoms"
table. The "idents" materialized view is a slice of the
"transactions" table. I hope that representing in this way, and
casting the problem in this light, might generalize to future
materialized views.
* Pre: Add DiffSet.
* Part 4: Collect mutations to a `Schema`.
I haven't taken your review comment about consuming AttributeBuilder
during each fluent function. If you read my response and still want
this, I'm happy to do it in review.
* Part 5: Handle :db/ident and :db.{install,alter}/attribute.
This "loops" the committed datoms out of the SQL store and back
through the metadata (schema, but in future also partition map)
processor. The metadata processor updates the schema and produces a
report of what changed; that report is then used to update the SQL
store. That update includes:
- the materialized views ("entids", "idents", and "schema");
- if needed, a subset of the datoms themselves (as flags change).
I've left a TODO for handling attribute retraction in the cases that
it makes sense. I expect that to be straight-forward.
* Review comment: Rename DiffSet to AddRetractAlterSet.
Also adds a little more commentary and a simple test.
* Review comment: Use ToIdent trait.
* Review comment: partially revert "Part 2: Maintain entids separately from idents."
This reverts commit 23a91df9c35e14398f2ddbd1ba25315821e67401.
Following our discussion, this removes the "entids" materialized
view. The next commit will remove historical idents from the "idents"
materialized view.
* Post: Use custom Either rather than std::result::Result.
This is not necessary, but it was suggested that we might be paying an
overhead creating Err instances while using error_chain. That seems
not to be the case, but this change shows that we don't actually use
any of the Result helper methods, so there's no reason to overload
Result. This change might avoid some future confusion, so I'm going
to land it anyway.
Signed-off-by: Nick Alexander <nalexander@mozilla.com>
* Review comment: Don't preserve historical idents.
* Review comment: More prepared statements when updating materialized views.
* Post: Test altering :db/cardinality and :db/unique.
These tests fail due to a Datomic limitation, namely that the marker
flag :db.alter/attribute can only be asserted once for an attribute!
That is, [:db.part/db :db.alter/attribute :attribute] will only be
transacted at most once. Since older versions of Datomic required the
:db.alter/attribute flag, I can only imagine they either never wrote
:db.alter/attribute to the store, or they handled it specially. I'll
need to remove the marker flag system from Mentat in order to address
this fundamental limitation.
* Post: Remove some more single quotes from error output.
* Post: Add assert_transact! macro to unwrap safely.
I was finding it very difficult to track unwrapping errors while
making changes, due to an underlying Mac OS X symbolication issue that
makes running tests with RUST_BACKTRACE=1 so slow that they all time
out.
* Post: Don't expect or recognize :db.{install,alter}/attribute.
I had this all working... except we will never see a repeated
`[:db.part/db :db.alter/attribute :attribute]` assertion in the store!
That means my approach would let you alter an attribute at most one
time. It's not worth hacking around this; it's better to just stop
expecting (and recognizing) the marker flags. (We have all the data
to distinguish the various cases that we need without the marker
flags.)
This brings Mentat in line with the thrust of newer Datomic versions,
but isn't compatible with Datomic, because (if I understand correctly)
Datomic automatically adds :db.{install,alter}/attribute assertions to
transactions.
I haven't purged the corresponding :db/ident and schema fragments just
yet:
- we might want them back
- we might want them in order to upgrade v1 and v2 databases to the
new on-disk layout we're fleshing out (v3?).
* Post: Don't make :db/unique :db.unique/* imply :db/index true.
This patch avoids a potential bug with the "schema" materialized view.
If :db/unique :db.unique/value implies :db/index true, then what
happens when you _retract_ :db.unique/value? I think Datomic defines
this in some way, but I really want the "schema" materialized view to
be a slice of "datoms" and not have these sort of ambiguities and
persistent effects. Therefore, to ensure that we don't retract a
schema characteristic and accidentally change more than we intended
to, this patch stops having any schema characteristic imply any other
schema characteristic(s). To achieve that, I added an
Option<Unique::{Value,Identity}> type to Attribute; this helps with
this patch, and also looks ahead to when we allow to retract
:db/unique attributes.
* Post: Allow to retract :db/ident.
* Post: Include more details about invalid schema changes.
The tests use strings, so they hide the chained errors which do in
fact provide more detail.
* Review comment: Fix outdated comment.
* Review comment: s/_SET/_SQL_LIST/.
* Review comment: Use a sub-select for checking cardinality.
This might be faster in practice.
* Review comment: Put `attribute::Unique` into its own namespace.
* Add a failing test for EDN parsing '…'.
* Expose a SQLValueType trait to get value_type_tag values out of a ValueType.
* Add accessors to FindSpec.
* Implement querying.
* Implement rudimentary projection.
* Export mentat_db::new_connection.
* Export symbols from mentat.
* Add rudimentary end-to-end query tests.
* Add top-level `Conn`. Fixes#296.
This is a little different than the API rnewman and I originally
discussed in https://public.etherpad-mozilla.org/p/db-conn-thoughts.
A few notes:
- I was led to make a `Schema` instance the thing that is shared,
rather than a `db::DB`. It's possible that queries will want to
know the current transaction at some point (to prevent races, or to
query historical data), but that can be a future consideration.
- The generation number just allows for a cheap comparison. I don't
care to handle races to transact just yet; the long term plan might
be to make embedding applications responsible for avoiding races, or
we might handle queuing transactions and yielding report futures in
Mentat itself.
- The sharing of the partition maps is a little more subtle than
expected. Partition maps are volatile: a successful Mentat
transaction always advances the :db.part/tx partition, so it's not
worth passing references around. This means that consumers must
clone in order to maintain just a single clone per transaction.
Clean some cruft.
* Review comments.
* Pre: Drop unneeded tx0 from search results.
* Pre: Don't require a schema in some of the DB code.
The idea is to separate the transaction applying code, which is
schema-aware, from the concrete storage code, which is just concerned
with getting bits onto disk.
* Pre: Only reference Schema, not DB, in debug module.
This is part of a larger separation of the volatile PartitionMap,
which is modified every transaction, from the stable Schema, which is
infrequently modified.
* Pre: Fix indentation.
* Extract part of DB to new SchemaTypeChecking trait.
* Extract part of DB to new PartitionMapping trait.
* Pre: Don't expect :db.part/tx partition to advance when tx fails.
This fails right now, because we allocate tx IDs even when we shouldn't.
* Sketch a db interface without DB.
* Add ValueParseError; use error-chain in tx-parser.
This can be simplified when
https://github.com/Marwes/combine/issues/86 makes it to a published
release, but this unblocks us for now. This converts the `combine`
error type `ParseError<&'a [edn::Value]>` to a type with owned
`Vec<edn::Value>` collections, re-using `edn::Value::Vector` for
making them `Display`.
* Pre: Accept Borrow<Schema> instead of just &Schema in debug module.
This makes it easy to use Rc<Schema> or Arc<Schema> without inserting
&* sigils throughout the code.
* Use error-chain in query-parser.
There are a few things to point out here:
- the fine grained error types have been flattened into one crate-wide
error type; it's pretty easy to regain the granularity as needed.
- edn::ParseError is automatically lifted to
mentat_query_parser::errors::Error;
- we use mentat_parser_utils::ValueParser to maintain parsing error
information from `combine`.
* Patch up top-level.
* Review comment: Only `borrow()` once.
* Leave a pointer to issue 288.
* Re-export mentat_db::types::DB from mentat_db.
* Parse EDN strings in the query parser.
* Export 'public' API from mentat_query_parser's top level.
* Stub out mentat::q_once.
* Test the mentat_query directory on Travis.
* Export common types from edn.
This allows you to write
use edn::{PlainSymbol,Keyword};
instead of
use edn:🔣:{PlainSymbol,Keyword};
* Add an edn::Value::is_keyword predicate.
* Clean up query, preparing for query-parser.
* Make EDN keywords and symbols take Into<String> arguments.
* Implement parsing of simple :find lists.
* Rustfmt query-parser. Split find and query.
* Review comment: values_to_variables now returns a NotAVariableError on failure.
* Review comment: rename gimme to to_parsed_value.
* Review comment: add comments.
Starting to work out the project layout for sub-crates. The crate inside query-parser/ is "datomish-query-parser" and the core code in src/ depends on it.
This allows for code to run before and after a schema fragment is
added for the first time.
The anticipated use for this is twofold:
1. To do initial setup, e.g., defining global entities.
2. To 'adopt' unmanaged attributes already defined in the store.
This 'pre' would manually alter or retract attributes so that the
transact of the new schema datoms can complete.
For example, if properties :foo/bar and :foo/baz will be unchanged,
but :noo/zob needs to change from a string to an integer, the :none
pre-function can alter the ident, and the :none post-function can
migrate and clean up.
This generalizes the transactor loop to allow callers to run
an arbitrary function within an `in-transaction!` body.
Combined with exposing `<report-transact-tx-data!`, this allows
an admittedly sophisticated consumer to conditionally query and
transact in a consistent way -- for example, cleaning up inconsistent
data then transacting a new schema version.
Altering uniqueness and cardinality attributes works, with the exception
of enabling uniqueness from nothing.
:db/noHistory and :db/isComponent changes are implemented but untested,
and aren't really supported by Datomish anyway.
The metaphor we use is that of "evolution", where each "evolutionary
step" contains a number of different "generations". Entities in the
process of being resolved are increasingly "evolved" into simpler
generations, until no further evolution is possible.
The test would fail because we would have an [a v] pair with a string
value, but we were looking for the fulltext rowid in <avs. Using
all_datoms correctly looks up the string value, at the cost of crippling
the speed of <avs.
This sorts fulltext values inserted in a single transaction, not across
transactions. This makes the rowids assigned in the fulltext_values
table internally consistent, even as the order of entities and datoms
changes (as the transaction applying algorithm evolves over time). The
test changes simply make the fulltext values sort easily.
In theory, these fulltext values could be very large, and sorting might
be very expensive. In practice, we expect values to differ in their
first few characters, so that this is efficient (i.e., proportional to
the number of fulltext values inserted and not their size).
This uses a common table expression and multiple SQL calls rather than a
temporary table, since transactions with huge numbers of distinct
lookup-refs are likely to be very rare.
We mark lookup-refs with `lookup-ref`, which is a little awkward because
binding `(let [[a v] lookup-ref] ...)` doesn't directly work, but avoids
some ambiguity present in Datomic and DataScript around interpreting
lookup-refs as multiple value lists. (Which bit the tests in an earlier
version of this patch!)
There's no distinction made for fulltext attributes, since the values
found by the retractAttributes SELECT are already rowids into the
fulltext_values table and therefore need no additional mapping.
These temp files will almost certainly live in memory only, speeding our
test suite evaluation significantly. Before this patch, in a warmed
REPL environment I get:
Testing datomish.db-test
Ran 19 tests containing 97 assertions.
0 failures, 0 errors.
"Elapsed time: 1408.720681 msecs"
"Elapsed time: 1343.986464 msecs"
"Elapsed time: 1338.660762 msecs"
After this patch, in a warmed REPL environment I get:
Testing datomish.db-test
Ran 19 tests containing 97 assertions.
0 failures, 0 errors.
"Elapsed time: 587.605168 msecs"
"Elapsed time: 569.522333 msecs"
"Elapsed time: 589.080282 msecs"
We'd like this to be part of the query syntax itself, but doing so
requires extending DataScript's parser.
Instead we generalize our `args` to `options`, and take `:limit`
and `:order-by-vars`. The former must be an integer or nil, and the
latter is an array of `[var direction]` pairs.
This commit includes descriptive error messages and tests for success
and failure.
This caches a partition map per DB, which is helpful because it exposes
what the point in time DB partition state is, but is unhelpful because
the partition state can advance underneath the DB cache. This is
generally true of the approach -- this can happen to the ident/entid
maps, and the datoms themselves -- so we'll roll with it for now.
This reduces the number of SQL UPDATE operations from linear in the
number of id-literals used to constant in the number of known
partitions.
* Alter how clauses are concatenated. They now preserve order more accurately.
* Track mappings between vars and extracted type columns.
* Generate type code constraints.
* Push known types down into :not.
* Push known types down into :or.
* Tests and test fixes.
Note that `go` (and `go-pair`) don't descend into `for` comprehensions
and other situations in which a fn is created. This commit rewrites to
use nested `loop`s, and also improves use of `<av`.
* Batch up datoms into a smaller number of queries, improving transact speed by about 50%.
* Restore transacting FTS attributes.
* Implement retraction of freetext datoms.
This is almost complete; it passes the test suite save for retracting
fulltext datoms correctly.
There's a lot to say about this approach, but I don't have time to give
too many details. The broad outline is as follows. We collect datoms
to add and retract in a tx_lookup table. Depending on flags ("search
value" sv and "search value type tag" svalue_type_tag) we "complete" the
tx_lookup table by joining matching datoms. This allows us to find
datoms that are present (and should not be added as part of the
transaction, or should be retracted as part of the transaction, or
should be replaced as part of the transaction. We complete the
tx_lookup (in place!) in two separate INSERTs to avoid a quadratic
two-table walk (explain the queries to observe that both INSERTs walk
the lookup table once and then use the datoms indexes to complete the
matching values).
We could simplify the code by using multiple lookup tables, both for the
two cases of search parameters (eav vs. ea) and for the incomplete and
completed rows. Right now we differentiate the former with NULL checks,
and the latter by incrementing the added0 column. It performs well
enough, so I haven't tried to understand the performance of separating
these things.
After the tx_lookup table is completed, we build the transaction from
it; and update the datoms materialized view table as well. Observe the
careful handling of the "search value" sv parameters to handle replacing
:db.cardinality/one datoms.
Finally, we read the processed transaction back to produce to the API.
This is strictly to match the Datomic API; we might make allow to skip
this, since many consumers will not want to stream this over the wire.
Rough timings show the transactor processing a single >50k datom
transaction in about 3.5s, of which less than 0.5s is spent in the
expensive joins. Further, repeating the processing of the same
transaction is only about 3.5s again! That's the worst possible for the
joins, since every single inserted datom will already be present in the
database, making the most expensive join match every row.
