This is a stepping stone to transacting entities that are not based on
`edn::ValueAndSpan`. We need to turn some value places (general) into
entity places (restricted), and those restrictions are captured in
tx-parser right now. But for `TypedValue` value places, those
restrictions are encoded in the type itself. This lays the track to
accept other value types in value places, which is good for
programmatic builder interfaces.
* Refactor AttributeCache populator code for use from pull.
* Pre: add to_value_rc to Cloned.
* Pre: add From<StructuredMap> for Binding.
* Pre: clarify Store::open_empty.
* Pre: StructuredMap cleanup.
* Pre: clean up a doc test.
* Split projector crate. Pass schema to projector.
* CLI support for printing bindings.
* Add and use ConjoiningClauses::derive_types_from_find_spec.
* Define pull types.
* Implement pull on top of the attribute cache layer.
* Add pull support to the projector.
* Parse pull expressions.
* Add simple pull support to connection objects.
* Tests for pull.
* Compile with Rust 1.25.
The only choice involved in this commit is that of replacing the
anonymous lifetime '_ with a named lifetime for the cache; since we're
accepting a Known, which includes the cache in question, I think it's
clear that we expect the function to apply to any given cache
lifetime.
* Review comments.
* Bail on unnamed attribute.
* Make assert_parse_failure_contains safe to use.
* Rework query parser to report better errors for pull.
* Test for mixed wildcard and simple attribute.
This innocuous looking change (upserts_ev -> upserts_e -> resolved in
all situations, rather than upserts_ev -> resolved in some situations)
is a significant change in semantics and assumptions in the
transactor. Witness the large comment being removed about the same
tempid resolving in different generations!
To support this change, we provide more holistic errors for
conflicting upserts, which entails collecting some (relatively
expensive) diagnostic data.
I left in some debug logging, simply since it shouldn't hurt in
general, and will likely be useful for the next bug we see in the
transactor.
We don't yet have a logging system for production use, but I'd like to
start experimenting with log, which seems to be (close to) a Rust
standard. We're already using it in mentat_cli.
:db/tx (and Datomic's version, :datomic/tx) suffer from the same
ambiguities that [a v] lookup references do -- determining the type of
the result is context sensitive. (In this case, is :db/tx a reference
to the current transaction ID, or is it a valid keyword?) This commit
addresses the ambiguity by introducing a notion of a transaction
functions, and provides a little scaffolding for adding more (should
the need arise). I left the scaffolding in place rather than handling
just (transaction-tx) because I started trying to
implement (transaction-instant) as well, which is more difficult --
see the comments.
It's worth noting that this approach generalizes more or less directly
to ?input variables, since those can be eagerly bound like the
implemented transaction function (transaction-tx).
* Pre: eliminate some occurrences of Rc, largely through the magic of Into.
* Pre: introduce FromRc to convert between refcounted types.
* Introduce ValueRc as an abstraction over Rc/Arc choice.
* Move Cloned to core.
* Move CString-creation methods to TypedValue.
* Finish transition.
* Pre: clean up core/src/lib.rs.
* Pre: use indexmap 1.0 in db and query-projector.
* Change rel results to be a RelResult instance, not a Vec<Vec<TypedValue>>.
This avoids memory fragmentation and improves locality by using a single
heap-allocated vector for all bindings, rather than a separate
heap-allocated vector for each row.
We hide this abstraction behind the `RelResult` type, which tracks the
stride length (width) of each row.
* Don't allocate temporary vectors when projecting RelResults.
Some parts of the query engine and transactor need to know whether an
attribute is a component attribute, and sometimes want to do so in
a generated SQL query. This is one way to do that.
Simplify.
This has a watcher collect txid -> AttributeSet mappings each time a
transact occurs. On commit we retrieve those mappings and hand them over
to the observer service, which filters them and packages them up for
dispatch.
Tidy up
* Use fixed-size arrays for bootstrap datoms, not vecs.
* Wide-ranging cleanup.
This commit:
- Deletes some dead code.
- Marks some functions only used by tests as cfg(test).
- Adds pub(crate) to a bunch of functions.
- Cleans up a few other nits.
* Use the cache to make constant queries super fast.
* Fix translate tests to match: we no longer generate SQL for many of them!
* Accumulate additions and removals into the cache.
* Make attribute cache clone-on-write; store it in Metadata.
* Allow caching of fulltext attributes, interning strings.
This puts caching in mentat_db, adds a reverse lookup capability for
unique attributes, and populates bidirectional caches with a single
SQL cursor walk.
Differentiate between begin_read and begin_uncached_read.
Note that we still allow toggling within InProgress, because there might be
transient local state that makes starting a new transaction impossible.
You can use this in conjunction with setting SQLITE3_LIB_DIR to control which SQLite is used.
See https://github.com/jgallagher/rusqlite for more.
Also add recent contributors to the authors array.
* Nit: Alphabetical ordering of imports
* Create Cache and provide functions for calling it
* Get tests working. Move to using NamespacedKeyword over KnownEntid in function signature
* Add is_cached check to caching tests
* Move lazy and add/remove boolean flags to enums
* Move function definitions into generic trait and implement trait for AttributeCache
* Remove lazy cache and generalize cache
* Update tests
* Eager cache becomes simple key value store. AttributeMap handles attribute storing specifics
* Update tests to test presence of correct values in cache
* Move EagerCache, AttributeValueProvider and ValueProvider into mentat_db
* Add test for get_for_entid
* Add test for lookup attribute
* Make caches cloneable. Add value_for alongside values_for
* Use cache in attribute lookups
* Split test for values and value and add cardinality
* address review feedback r=rnewman
Also move `now` into core, implement microsecond truncation.
This is so we don't return a more granular -- and thus subtly different --
timestamp in a `TxReport` than we put into the store.
This includes two other changes:
* Split transact to expose an interface for TermWithTempIds.
* Return TxReport from each InProgress operation, not from commit.
Improve naming of read-only transactions.
Implement entid_for_type.
Simplify get_attribute.
Name ignored var in algebrizer.
Comment attribute_for_ident.
Make KnownEntid a core concept.
Expose lookup_value_for_attribute.
Implement HasSchema and a new query encapsulation on Conn.
Pre: export Queryable.
Pre: export AttributeBuilder from mentat_db.
Pre: fix module-level comment for tx/src/entities.rs.
Pre: rename some `to_` conversions to `into_`.
Pre: make AttributeBuilder::unique less verbose.
Pre: split out a HasSchema trait to abstract over Schema.
Pre: rename SchemaMap/schema_map to AttributeMap/attribute_map.
Pre: TypedValue/NamespacedKeyword conversions.
Pre: turn Unique and ValueType into TypedValue::Keyword.
Pre: export IntoResult.
Pre: export NamespacedKeyword from mentat_core.
Pre: use intern_set in tx.
Pre: add InternSet::len.
Pre: comment gardening.
Pre: remove inaccurate TODO from TxReport comment.
This was done using the following shell script:
```
find . -type f -not -path "*target*" \
'(' -name '*.rs' -o -name '*.md' -o -name '*.toml' ')' -print0 | \
xargs -0 sed -i '' -E 's/[[:space:]]*$//'
```
Which is admittedly imperfect, but manages to hit everything that was a problem in this repo.
* Pre: rename begin_transaction to begin_tx_application.
* Take an EXCLUSIVE transaction when bootstrapping, and an IMMEDIATE transaction when writing.
This avoids the remote possibility of another write sneaking in the door
while we're preparing to write, avoids us needing to upgrade locks, etc.
After a BEGIN IMMEDIATE, no other database connection will be able to write
to the database or do a BEGIN IMMEDIATE or BEGIN EXCLUSIVE. Other processes
can continue to read from the database, however.
An exclusive transaction causes EXCLUSIVE locks to be acquired on all
databases. After a BEGIN EXCLUSIVE, no other database connection except for
read_uncommitted connections will be able to read the database and no other
connection without exception will be able to write the database until the
transaction is complete.
* Hacky implementation of atomic multi-tx.
* Hold the last report, returning the InProgress from each operation.
* Rewrite transact in terms of InProgress.
* Test rollback.
* Remove unused imports.
* Don't use Rc for transaction reports.
* Pre: break out USER0 as a part boundary constant.
* Export TX0 and USER0 from mentat_db. This is for testing.
* Review comments: commenting.
* Test tempid allocation and rollback.
* Create mentat command line.
* Create tools directory containing new crate for mentat_cli.
* Add simple cli with mentat prompt.
* Remove rustc-serialize dependency
* Open DB inside CLI (#452) (#463)
* Open named database OR default to in memory database if no name provided
Rearrange workspace to allow import of mentat crate in cli crate
Create store object inside repl when started for connecting to mentat
Use provided DB name to open connection in store
Accept DB name as command line arg.
Open on CLI start
Implement '.open' command to open desired DB from inside CLI
* Implement Close command to close current DB.
* Closes existing open db and opens new in memory db
* Review comment: Use `combine` to parse arguments.
Move over to using Result rather than enums with err
* Accept and parse EDN Query and Transact commands (#453) (#465)
* Parse query and transact commands
* Implement is_complete for transactions and queries
* Improve query parser. Am still not happy with it though.
There must be some way that I can retain the eof() after the `then` that means I don't have to move the skip on spaces and eof
Make in process command storing clearer.
Add comments around in process commands.
Add alternative commands for transact/t and query/q
* Address review comments r=nalexander.
* Bump rust version number.
* Use `bail` when throwing errors.
* Improve edn parser.
* Remove references to unused `more` flag.
* Improve naming of query and transact commands.
* Send queries and transactions to mentat and output the results (#466)
* Send queries and transactions to mentat and output the results
move outputting query and transaction results out of store and into repl
* Add query and transact commands to help
* Execute queries and transacts passed in at startup
* Address review comments =nalexander.
* Bump rust version number.
* Use `bail` when throwing errors.
* Improve edn parser.
* Remove references to unused `more` flag.
* Improve naming of query and transact commands.
* Execute command line args in order
* Addressing rebase issues
* Exit CLI (#457) (#484) r-rnewman
* Implement exit command for cli tool
* Address review comments r=rnewman
* Include exit commands in help
* Show schema of current DB (#487)
* Fixing rebase issues
* addressing nit
* Match updated dependencies on CLI crate and remove unused import
* Update some dependencies.
* Update rusqlite to 0.12.
* Update error-chain to a forked version that implements Sync.
* Fix some compiler warnings.
* Remove unused imports in tests.
* Parse errors no longer naturally print with the expected symbol.
This commit adds a check to the partition map that a provided entity ID
has been mentioned (i.e., is present in the start:index range of one of
our partitions).
We introduce a newtype for known entity IDs, using this internally in
the tx expander to track user-provided entids that have passed the above
check (and IDs that we allocate as part of tempid processing). This
newtype is stripped prior to tx assertion.
In order that DB tests can continue to write
[:db/add 111 :foo/bar 222]
we add an additional fake partition to our test connections, ranging
from 100 to 1000.
This is an optimization that trades rejecting inputs earlier at the
cost of expressive error messages. It should be possible to recover
the error messages, however.
This will reject input like `[:db/{add,retract} v :attribute/_reversed NOT-AN-ENTITY]`.
There are two broad approaches:
1) Handle reverse attribute notation dynamically, in the style that
Datomic does. This is the most flexible, but it's not a good fit
given that we produce strongly typed output from the parser.
Strongly typed input to the transactor has had many benefits, so I
don't want to roll it back for a relatively unimportant feature
like reverse notation -- especially not since Mentat does not
require :db.install/_attribute to modify schema attributes.
2) Handle reverse attribute in the parser itself, so that we can
produce strongly typed parser output while restricting the input.
I implemented this first and discovered that it's very difficult to
give sensible error messages in common cases.
In any case, the bulk of the code is the same between the two
approaches, and I wrote the tests for the dynamic version (with error
output), so that's what I'm rolling with.
This patch preserves the existing indentation, to highlight the
differences. The next patch will indent.
