Commit graph

178 commits

Author SHA1 Message Date
Richard Newman
5e50d2a9b4 Update to rusqlite 0.13. 2018-02-22 11:41:57 -08:00
Richard Newman
54bd883c65 Follow-up: remove logging and such elsewhere in the codebase. 2018-02-21 11:51:45 -08:00
Richard Newman
e33fe71c47 Rework caching and use it inside the query engine. (#553) r=emily
This puts caching in mentat_db, adds a reverse lookup capability for
unique attributes, and populates bidirectional caches with a single
SQL cursor walk.

Differentiate between begin_read and begin_uncached_read.

Note that we still allow toggling within InProgress, because there might be
transient local state that makes starting a new transaction impossible.
2018-02-21 11:51:45 -08:00
Richard Newman
df3cdb5db6 Allow two datoms in the same transaction to have the same fulltext string. (#565) r=emily 2018-02-21 11:51:45 -08:00
Richard Newman
ae91603bd0 Provide an API for creating truly empty stores (#561) r=grisha
* Part 1: split create_current_version.

* Part 2: add Store::create_empty and Conn::empty.

* Part 3 - Expose 'open_empty' command via CLI
2018-02-16 02:01:00 -08:00
Richard Newman
2ac7a1b1de Add a feature flag to control the use of rusqlite's bundled SQLite. r=emily
You can use this in conjunction with setting SQLITE3_LIB_DIR to control which SQLite is used.

See https://github.com/jgallagher/rusqlite for more.

Also add recent contributors to the authors array.
2018-02-13 08:25:58 -08:00
Richard Newman
d11810dca7 Fix warning in tx.rs. 2018-02-09 08:51:13 -08:00
Emily Toop
715d434945
Create generalized in-memory cache for attributes (#525)
* Nit: Alphabetical ordering of imports

* Create Cache and provide functions for calling it

* Get tests working. Move to using NamespacedKeyword over KnownEntid in function signature

* Add is_cached check to caching tests

* Move lazy and add/remove boolean flags to enums

* Move function definitions into generic trait and implement trait for AttributeCache

* Remove lazy cache and generalize cache

* Update tests

* Eager cache becomes simple key value store. AttributeMap handles attribute storing specifics

* Update tests to test presence of correct values in cache

* Move EagerCache, AttributeValueProvider and ValueProvider into mentat_db

* Add test for get_for_entid

* Add test for lookup attribute

* Make caches cloneable. Add value_for alongside values_for

* Use cache in attribute lookups

* Split test for values and value and add cardinality

* address review feedback r=rnewman
2018-02-07 10:56:12 -08:00
Grisha Kruglov
d848d954cf Issue 508 - Iterating transcation processor r=rnewman
Review comments
2018-02-06 12:24:12 -08:00
Richard Newman
e9ed103723 Revert "Make debug structs and functions non-public. r=grisha"
This reverts commit e817f67470.
2018-02-06 11:06:06 -08:00
Richard Newman
e817f67470 Make debug structs and functions non-public. r=grisha 2018-02-06 10:25:42 -08:00
Richard Newman
37a7c9ea48 Validate attributes installed after open. (#538) r=emily
Make AttributeBuilder optionally helpful, fix tests.
2018-02-01 09:29:04 -08:00
Edouard Oger
3bf7459315 Allow customers to assert facts about the current transaction. (#225) r=rnewman
Also move `now` into core, implement microsecond truncation.

This is so we don't return a more granular -- and thus subtly different --
timestamp in a `TxReport` than we put into the store.
2018-01-29 16:46:04 -08:00
Richard Newman
ef9f2d9c51 Don't allow violation of cardinality-one restrictions within a single tx. (#531) r=nalexander 2018-01-23 15:11:38 -08:00
Richard Newman
812f10b3e4 Add an EntityBuilder abstraction. r=nalexander,emily
This includes two other changes:

* Split transact to expose an interface for TermWithTempIds.
* Return TxReport from each InProgress operation, not from commit.
2018-01-23 08:52:09 -08:00
Richard Newman
3d28949add Describe the default core schema, v1 (:db.schema/core). r=nalexander 2018-01-23 08:51:58 -08:00
Richard Newman
4acc6d0658 InProgressRead, KnownEntid. r=nalexander,emily
Improve naming of read-only transactions.
    Implement entid_for_type.
    Simplify get_attribute.
    Name ignored var in algebrizer.
    Comment attribute_for_ident.
    Make KnownEntid a core concept.
    Expose lookup_value_for_attribute.
    Implement HasSchema and a new query encapsulation on Conn.
    Pre: export Queryable.
2018-01-23 08:40:18 -08:00
Richard Newman
6797a606b5 Preliminary work for vocabulary management. r=emily,nalexander
Pre: export AttributeBuilder from mentat_db.
Pre: fix module-level comment for tx/src/entities.rs.
Pre: rename some `to_` conversions to `into_`.
Pre: make AttributeBuilder::unique less verbose.
Pre: split out a HasSchema trait to abstract over Schema.
Pre: rename SchemaMap/schema_map to AttributeMap/attribute_map.
Pre: TypedValue/NamespacedKeyword conversions.
Pre: turn Unique and ValueType into TypedValue::Keyword.
Pre: export IntoResult.
Pre: export NamespacedKeyword from mentat_core.
Pre: use intern_set in tx.
Pre: add InternSet::len.
Pre: comment gardening.
Pre: remove inaccurate TODO from TxReport comment.
2018-01-23 08:25:32 -08:00
Thom
9740cafdbd Automatically remove trailing whitespace from text files. (#527) r=rnewman
This was done using the following shell script:

```
find . -type f -not -path "*target*" \
       '(' -name '*.rs' -o -name '*.md' -o -name '*.toml' ')' -print0 | \
    xargs -0 sed -i '' -E 's/[[:space:]]*$//'
```

Which is admittedly imperfect, but manages to hit everything that was a problem in this repo.
2018-01-19 21:21:04 -06:00
Richard Newman
95e95d735e
Correct an assert relating Datalog projection and SQL column counts. (#519) r=tcsc
* Correct an assert relating Datalog projection and SQL column counts. (#517)
* Fix a comment that shouldn't be a doc comment.
2018-01-17 16:36:15 -06:00
Grisha Kruglov
1589104841
Force SQLite temp files to be stored in memory (#505) r=rnewman (#506)
* Force SQLite temp files to be stored in memory (#505) r=rnewman
2017-12-20 01:52:09 -05:00
Richard Newman
95b9c7f7f5
Atomic multi-tx (#489). r=emily,nalexander
* Pre: rename begin_transaction to begin_tx_application.

* Take an EXCLUSIVE transaction when bootstrapping, and an IMMEDIATE transaction when writing.

This avoids the remote possibility of another write sneaking in the door
while we're preparing to write, avoids us needing to upgrade locks, etc.

  After a BEGIN IMMEDIATE, no other database connection will be able to write
  to the database or do a BEGIN IMMEDIATE or BEGIN EXCLUSIVE. Other processes
  can continue to read from the database, however.

  An exclusive transaction causes EXCLUSIVE locks to be acquired on all
  databases. After a BEGIN EXCLUSIVE, no other database connection except for
  read_uncommitted connections will be able to read the database and no other
  connection without exception will be able to write the database until the
  transaction is complete.

* Hacky implementation of atomic multi-tx.

* Hold the last report, returning the InProgress from each operation.

* Rewrite transact in terms of InProgress.

* Test rollback.

* Remove unused imports.

* Don't use Rc for transaction reports.

* Pre: break out USER0 as a part boundary constant.

* Export TX0 and USER0 from mentat_db. This is for testing.

* Review comments: commenting.

* Test tempid allocation and rollback.
2017-12-05 07:58:24 -08:00
Emily Toop
55588209c2
CLI (#493)
* Create mentat command line.
* Create tools directory containing new crate for mentat_cli.
* Add simple cli with mentat prompt.

* Remove rustc-serialize dependency

* Open DB inside CLI (#452) (#463)

* Open named database OR default to in memory database if no name provided

Rearrange workspace to allow import of mentat crate in cli crate

Create store object inside repl when started for connecting to mentat

Use provided DB name to open connection in store

Accept DB name as command line arg.

Open on CLI start

Implement '.open' command to open desired DB from inside CLI

* Implement Close command to close current DB.
* Closes existing open db and opens new in memory db

* Review comment: Use `combine` to parse arguments.

Move over to using Result rather than enums with err

* Accept and parse EDN Query and Transact commands (#453) (#465)

* Parse query and transact commands

* Implement is_complete for transactions and queries

* Improve query parser. Am still not happy with it though.

There must be some way that I can retain the eof() after the `then` that means I don't have to move the skip on spaces and eof

Make in process command storing clearer.

Add comments around in process commands.
Add alternative commands for transact/t and query/q

* Address review comments r=nalexander.

* Bump rust version number.
* Use `bail` when throwing errors.
* Improve edn parser.
* Remove references to unused `more` flag.
* Improve naming of query and transact commands.

* Send queries and transactions to mentat and output the results (#466)

* Send queries and transactions to mentat and output the results

move outputting query and transaction results out of store and into repl

* Add query and transact commands to help

* Execute queries and transacts passed in at startup

* Address review comments =nalexander.

* Bump rust version number.
* Use `bail` when throwing errors.
* Improve edn parser.
* Remove references to unused `more` flag.
* Improve naming of query and transact commands.

* Execute command line args in order

* Addressing rebase issues

* Exit CLI (#457) (#484) r-rnewman

* Implement exit command for cli tool

* Address review comments r=rnewman

* Include exit commands in help

* Show schema of current DB (#487)

* Fixing rebase issues

* addressing nit

* Match updated dependencies on CLI crate and remove unused import
2017-11-21 16:56:16 +00:00
Richard Newman
c600152d78
Update some dependencies. (#492) r=etoop
* Update some dependencies.

* Update rusqlite to 0.12.

* Update error-chain to a forked version that implements Sync.

* Fix some compiler warnings.

* Remove unused imports in tests.

* Parse errors no longer naturally print with the expected symbol.
2017-11-21 16:24:08 +00:00
Richard Newman
ea0e9d4c7b Allow instants to pass through schema validation. (#481) r=fluffyemily
* Allow instants to pass through schema validation.
* Expand cases in SchemaTypeChecking to catch enum bugs.
2017-06-16 09:15:29 -07:00
Richard Newman
c2ec1a6bdf Pre: move Either to mentat_core::util. 2017-06-15 10:28:02 -07:00
Richard Newman
9a12ced317 Don't allow callers to specify arbitrary new entity IDs. (#447) r=nalexander
This commit adds a check to the partition map that a provided entity ID
has been mentioned (i.e., is present in the start:index range of one of
our partitions).

We introduce a newtype for known entity IDs, using this internally in
the tx expander to track user-provided entids that have passed the above
check (and IDs that we allocate as part of tempid processing). This
newtype is stripped prior to tx assertion.

In order that DB tests can continue to write

  [:db/add 111 :foo/bar 222]

we add an additional fake partition to our test connections, ranging
from 100 to 1000.
2017-06-09 15:45:26 -07:00
Nick Alexander
c165972684 Post: Reject at parse-time reversed attributes in direct notation with bad values.
This is an optimization that trades rejecting inputs earlier at the
cost of expressive error messages.  It should be possible to recover
the error messages, however.

This will reject input like `[:db/{add,retract} v :attribute/_reversed NOT-AN-ENTITY]`.
2017-06-08 10:30:31 -07:00
Nick Alexander
59a710f80f Review comments: another test, add unreversed(). 2017-06-08 10:30:31 -07:00
Nick Alexander
eb220528bf Post: Indent. 2017-06-08 10:30:31 -07:00
Nick Alexander
d88823e7c4 Handle :attribute/_reverse in transactor. Fixes #187
There are two broad approaches:

1) Handle reverse attribute notation dynamically, in the style that
   Datomic does.  This is the most flexible, but it's not a good fit
   given that we produce strongly typed output from the parser.
   Strongly typed input to the transactor has had many benefits, so I
   don't want to roll it back for a relatively unimportant feature
   like reverse notation -- especially not since Mentat does not
   require :db.install/_attribute to modify schema attributes.

2) Handle reverse attribute in the parser itself, so that we can
   produce strongly typed parser output while restricting the input.
   I implemented this first and discovered that it's very difficult to
   give sensible error messages in common cases.

In any case, the bulk of the code is the same between the two
approaches, and I wrote the tests for the dynamic version (with error
output), so that's what I'm rolling with.

This patch preserves the existing indentation, to highlight the
differences.  The next patch will indent.
2017-06-08 10:30:31 -07:00
Nick Alexander
0be78cf956 Pre: Extract entity_*_into_term_* helpers. 2017-06-08 10:30:31 -07:00
Nick Alexander
4b0881a957 Pre: Push bookkeeping into an InProcess struct. 2017-06-08 10:30:31 -07:00
Nick Alexander
05129cefbb Pre: Use ValueType rather than Attribute to convert edn::Value to TypedValue.
This is expedient now, but might require work in the future to achieve
better error messages.
2017-06-08 10:30:31 -07:00
Nick Alexander
2650fe163d Pre: Intern lookup_ref by reference. 2017-06-08 10:30:31 -07:00
Nick Alexander
a4fc04ea86 Pre: Crib map_{left,right} for Either. 2017-06-08 10:30:31 -07:00
Nick Alexander
d1ac752de6 Parse without copying; parse keyword maps using macros.
This is a big commit, but it breaks into two conceptual pieces.  The
first is to "parse without copying".  We replace a stream of an owned
collection of edn::ValueAndSpan and instead have a stream of a
borrowed collection of &edn::ValueAndSpan references.  (Generally,
this is represented as an iterator over a slice, but it can be over
other things too.)  Cloning such iterators is constant time, which
improves on cloning an owned collection of edn::ValueAndSpan, which is
linear time in the length of the collection and additional time
depending on the complexity of the EDN values.

The second conceptual piece is to parse keyword maps using a special
parser and a macro to build the parser implementations.  Before, we
created a new edn::ValueAndSpan::Map to represent a keyword map in
vector form; since we're working with &edn::ValueAndSpan references
now, we can't create an &edn::ValueAndSpan reference with an
appropriate lifetime.  Therefore we generalize the concept of
iteration slightly and turn keyword maps in map form into linear
iterators by flattening the value maps.  This is a potentially
obscuring transformation, so we have to take care to protect against
some failure cases.  (See the comments and the tests in the code.)

After these changes, parsing using `combine` is linear time (and
reasonably fast).
2017-05-18 10:17:13 -07:00
Richard Newman
3d4615fb8c Allow opening a DB. (#462) r=fluffyemily 2017-05-09 09:42:35 -07:00
Richard Newman
523d5ea5f1 Bump dependency versions. r=bgrins. (#441) 2017-05-03 12:53:16 -07:00
Richard Newman
daca8def57 UUIDs and instants. Fixes #44, #45, #426, #427. (#438) r=nalexander
* Pre: unused import in translate.rs.

* Part 2: take a dependency on rusqlite for query arguments.

* Part 1: flatten V2 schema into V1. Add UUID and URI.

Bump expected ident and bootstrap datom count in tests.

* Part 5: parse edn::Value::Uuid.

* Part 3: extend ValueType and TypedValue to include Uuid.

* Part 4: add Uuid to query arguments.

* Part 6: extend db to support Uuid.

* Part 8: add a tx-parser test for #f NaN and #uuid.

* Part 7: parse and algebrize UUIDs in queries.

* Part 1: parse #inst in EDN and throughout query engine.

* Part 3: handle instants in db.

* Part 2: instants never matches integers in queries.

* Part 4: use DateTime for tx_instants.

* Add a test for adding and querying UUIDs and instants.

* Review comments.
2017-04-28 20:11:55 -07:00
Nick Alexander
ff0147e89c Review comments: downgrade to error-chain 0.8.1 for Send + Sync bound; use combine::primitive::Error. 2017-04-18 13:19:50 -07:00
Nick Alexander
5369f03464 Improve parsing of nested edn::ValueAndSpan streams. r=rnewman (#393)
* Pre: Expose more in edn.

* Pre: Make it easier to work with ValueAndSpan.

with_spans() is a temporary hack, needed only because I don't care to
parse the bootstrap assertions from text right now.

* Part 1a: Add `value_and_span` for parsing nested `edn::ValueAndSpan` instances.

I wasn't able to abstract over `edn::Value` and `edn::ValueAndSpan`;
there are multiple obstacles.  I chose to roll with
`edn::ValueAndSpan` since it exposes the additional span information
that we will want to form good error messages in the future.

* Part 1b: Add keyword_map() parsing an `edn::Value::Vector` into an `edn::Value::map`.

* Part 1c: Add `Log`/`.log(...)` for logging parser progress.

This is a terrible hack, but it sure helps to debug complicated nested
parsers.  I don't even know what a principled approach would look
like; since our parser combinators are so frequently expressed in
code, it's hard to imagine a data-driven interpreter that can help
debug things.

* Part 2: Use `value_and_span` apparatus in tx-parser/.

I break an abstraction boundary by returning a value column
`edn::ValueAndSpan` rather than just an `edn::Value`.  That is, the
transaction processor shouldn't care where the `edn::Value` it is
processing arose -- even we care to track that information we should
bake it into the `Entity` type.  We do this because we need to
dynamically parse the value column to support nested maps, and parsing
requires a full `edn::ValueAndSpan`.  Alternately, we could cheat and
fake the spans when parsing nested maps, but that's potentially
expensive.

* Part 3: Use `value_and_span` apparatus in query-parser/.

* Part 4: Use `value_and_span` apparatus in root crate.

* Review comment: Make Span and SpanPosition Copy.

* Review comment: nits.

* Review comment: Make `or` be `or_exactly`.

I baked the eof checking directly into the parser, rather than using
the skip and eof parsers.  I also took the time to restore some tests
that were mistakenly commented out.

* Review comment: Extract and use def_matches_* macros.

* Review comment: .map() as late as possible.
2017-04-06 10:06:28 -07:00
Richard Newman
a5023c70cb Use Rc for TypedValue, Variable, and query Ident keywords. (#395) r=nalexander
Part 1, core: use Rc for String and Keyword.
Part 2, query: use Rc for Variable.
Part 3, sql: use Rc for args in SQLiteQueryBuilder.
Part 4, query-algebrizer: use Rc.
Part 5, db: use Rc.
Part 6, query-parser: use Rc.
Part 7, query-projector: use Rc.
Part 8, query-translator: use Rc.
Part 9, top level: use Rc.
Part 10: intern Ident and IdentOrKeyword.
2017-04-02 21:38:36 -07:00
Emily Toop
8e6f37e709 #260 Convert Schema into edn::Value (#384) r=nalexander, r=rnewman
* Part 1 - Create as_edn_value function.

* Do not include defaults inside output.
* Pretty-printed by default. Do we want to make that a flag?
* Includes simple test just to make sure it works.

* Part 2 - only include ident if available.

* Part 3 - Remove spacing and newlines as unnecessary.

* Update function to build edn::Value directly rather than parsing from string

* Update test to actually test the functionality.

* Address review comments ncalexan.

 * Rename `as_edn_value` to `to_edn_value`.
 * Move `db/src/values.rs` to `core/src/values.rs` so we can reference inside `core/src/ib.rs`.
 * Add `lazy-static` crate to core `Cargo.toml`
 * Expose `values` as a public module from `core`.
 * Update references to values in `db/src/bootstrap.rs` & `db/src/lib.rs`.
 * Add new static vars for `DB_FULLTEXT`, `DB_INDEX` & `DB_IS_COMPONENT`.
 * Use static vars exposed in `values` inside `to_edn_value`.
 * Remove `db/id` as key in attribute output and use `entid` as `db/ident` if no `ident` is found for that `entid`.
 * Update test to match new expected output.

* Add doc comment for function

* Address review comments ncalexan.

* Update function docstring to give clearer description of function.
* Do not all entid at all to output.
* Clean up code fetching ident (make it rustier).

* Address review comments rnewman.

* Extract out to new `to_edn_value` functions code for creating `edn::Value`\'s for `ValueType` and `Attribute`.
* Use `map()` to create schema edn value rather than a loop.

* Address review comments rnewman.

* pass cloned instance of ident to `Attribute::get_edn_value`.
* update `use` import for `edn`.
* remove unnecessary  call when using ident as key on `associate_ident`.

* Fixed bug whereby we didn't differentiate between `db.index/value` and `db.index/identity` when generating `edn::Value`

* Add extra assert at the end to ensure we get the same output when we convert the same schema to edn multiple times

* Move check for type of uniqueness to `match` statement.

* Also use `iter` instead of `into_iter` when iterating schema map.
2017-03-30 11:08:36 +01:00
Emily Toop
b24db01744 Add tests for validate_schema_map (#391) r=rnewman
* Add tests for `validate_schema_map`

* Update test to ensure we get the right error out
2017-03-30 11:07:49 +01:00
Nick Alexander
4b874deae1 Lookup refs, nested vector values, map notation. Fixes #180, fixes #183, fixes #284. (#382) r=rnewman
* Pre: Fix error in parser macros.

* Pre: Make test unwrapping more verbose.

* Pre: Make lookup refs be (lookup-ref a v) in the entity position.

This has the advantage of being explicit in all situations and
unambiguous at parse-time.  This choice agrees with the Clojure
implementation but not with Datomic.  Datomic treats [a v] as a lookup
ref, is ambiguous at parse-time, and is disambiguated in ways I do not
understand at transaction time.  We mooted making lookup refs [[a v]]
and outlawing nested value vectors in transactions, but after
implementing that approach I decided it was better to handle lookup
refs at parse time and therefore outlawing nested value vectors is not
necessary.

* Handle lookup refs in the entity and value columns. Fixes #183.

* Pre 0a: Use a stack instead of into_iter.

* Pre 0b: Dedent.

* Pre 0c: Handle `e` after `v`.

This allows to use the original `e` while handling `v`.

* Explode value lists for :db.cardinality/many attributes. Fixes #284.

* Parse and accept map notation. Fixes #180.

* Pre: Modernize add() and retract() into one add_or_retract().

* Pre: Add is_collection and is_atom to edn::Value.

* Pre: Differentiate atoms from lookup-refs in value position.

Initially, I expected to accept arbitrary edn::Value instances in the
value position, and to differentiate in the transactor.  However, the
implementation quickly became a two-stage parser, since we always
wanted to parse the resulting value position into some other known
thing using the tx-parser.  To save calls into the parser and to allow
the parser to move forward with a smaller API surface, I push as much
of this parsing as possible into the initial parse.

* Pre: Modernize entities().

* Pre: Quote edn::Value::Text in Display.

* Review comment: Add and use edn::Value::into_atom.

* Review comment: Use skip(eof()) throughout.

* Review comment: VecDeque instead of Vec.

* Review comment: Part 0: Rename TempId to TempIdHandle.

* Review comment: Part 1: Differentiate internal and external tempids.

This breaks an abstraction boundary by pushing the Internal/External
split up to the Entity level in tx/ and tx-parser/.  This just makes
it easier to explode Entity map notation instances into Entity
instances, taking an existing External tempid :db/id or generating a
new Internal tempid as appropriate.  To do this without breaking the
abstraction boundary would require adding flexibility to the
transaction processor: we'd need to be able to turn Entity instances
into some internal enum and handle the two cases independently.  It
wouldn't be too hard, but this reduces the combinatorial type
explosion.
2017-03-27 16:30:04 -07:00
Nick Alexander
2129514e86 Support transacting :db/fulltext true attributes. Fixes #189. (#375) r=rnewman
These tests are direct translations of the Clojure tests.
2017-03-21 13:12:10 -07:00
Emily Toop
55291b4d30 Check sqlite version. Fixes #366. (#376) r=rnewman
Checks whether current SQLite version is at least the minimum required version and panics if not.
2017-03-21 16:50:31 +00:00
Nick Alexander
15b4195a6e Schema alteration. Fixes #294 and #295. (#370) r=rnewman
* Pre: Don't retract :db/ident in test.

Datomic (and eventually Mentat) don't allow to retract :db/ident in
this way, so this runs afoul of future work to support mutating
metadata.

* Pre: s/VALUETYPE/VALUE_TYPE/.

This is consistent with the capitalization (which is "valueType") and
the other identifier.

* Pre: Remove some single quotes from error output.

* Part 1: Make materialized views be uniform [e a v value_type_tag].

This looks ahead to a time when we could support arbitrary
user-defined materialized views.  For now, the "idents" materialized
view is those datoms of the form [e :db/ident :namespaced/keyword] and
the "schema" materialized view is those datoms of the form [e a v]
where a is in a particular set of attributes that will become clear in
the following commits.

This change is not backwards compatible, so I'm removing the open
current (really, v2) test.  It'll be re-instated when we get to
https://github.com/mozilla/mentat/issues/194.

* Pre: Map TypedValue::Ref to TypedValue::Keyword in debug output.

* Part 3: Separate `schema_to_mutate` from the `schema` used to interpret.

This is just to keep track of the expected changes during
bootstrapping.  I want bootstrap metadata mutations to flow through
the same code path as metadata mutations during regular transactions;
by differentiating the schema used for interpretation from the schema
that will be updated I expect to be able to apply bootstrap metadata
mutations to an empty schema and have things like materialized views
created (using the regular code paths).

This commit has been re-ordered for conceptual clarity, but it won't
compile because it references the metadata module.  It's possible to
make it compile -- the functionality is there in the schema module --
but it's not worth the rebasing effort until after review (and
possibly not even then, since we'll squash down to a single commit to
land).

* Part 2: Maintain entids separately from idents.

In order to support historical idents, we need to distinguish the
"current" map from entid -> ident from the "complete historical" map
ident -> entid.  This is what Datomic does; in Datomic, an ident is
never retracted (although it can be replaced).  This approach is an
important part of allowing multiple consumers to share a schema
fragment as it migrates forward.

This fixes a limitation of the Clojure implementation, which did not
handle historical idents across knowledge base close and re-open.

The "entids" materialized view is naturally a slice of the "datoms"
table.  The "idents" materialized view is a slice of the
"transactions" table.  I hope that representing in this way, and
casting the problem in this light, might generalize to future
materialized views.

* Pre: Add DiffSet.

* Part 4: Collect mutations to a `Schema`.

I haven't taken your review comment about consuming AttributeBuilder
during each fluent function.  If you read my response and still want
this, I'm happy to do it in review.

* Part 5: Handle :db/ident and :db.{install,alter}/attribute.

This "loops" the committed datoms out of the SQL store and back
through the metadata (schema, but in future also partition map)
processor.  The metadata processor updates the schema and produces a
report of what changed; that report is then used to update the SQL
store.  That update includes:
- the materialized views ("entids", "idents", and "schema");
- if needed, a subset of the datoms themselves (as flags change).

I've left a TODO for handling attribute retraction in the cases that
it makes sense.  I expect that to be straight-forward.

* Review comment: Rename DiffSet to AddRetractAlterSet.

Also adds a little more commentary and a simple test.

* Review comment: Use ToIdent trait.

* Review comment: partially revert "Part 2: Maintain entids separately from idents."

This reverts commit 23a91df9c35e14398f2ddbd1ba25315821e67401.

Following our discussion, this removes the "entids" materialized
view.  The next commit will remove historical idents from the "idents"
materialized view.

* Post: Use custom Either rather than std::result::Result.

This is not necessary, but it was suggested that we might be paying an
overhead creating Err instances while using error_chain.  That seems
not to be the case, but this change shows that we don't actually use
any of the Result helper methods, so there's no reason to overload
Result.  This change might avoid some future confusion, so I'm going
to land it anyway.

Signed-off-by: Nick Alexander <nalexander@mozilla.com>

* Review comment: Don't preserve historical idents.

* Review comment: More prepared statements when updating materialized views.

* Post: Test altering :db/cardinality and :db/unique.

These tests fail due to a Datomic limitation, namely that the marker
flag :db.alter/attribute can only be asserted once for an attribute!
That is, [:db.part/db :db.alter/attribute :attribute] will only be
transacted at most once.  Since older versions of Datomic required the
:db.alter/attribute flag, I can only imagine they either never wrote
:db.alter/attribute to the store, or they handled it specially.  I'll
need to remove the marker flag system from Mentat in order to address
this fundamental limitation.

* Post: Remove some more single quotes from error output.

* Post: Add assert_transact! macro to unwrap safely.

I was finding it very difficult to track unwrapping errors while
making changes, due to an underlying Mac OS X symbolication issue that
makes running tests with RUST_BACKTRACE=1 so slow that they all time
out.

* Post: Don't expect or recognize :db.{install,alter}/attribute.

I had this all working... except we will never see a repeated
`[:db.part/db :db.alter/attribute :attribute]` assertion in the store!
That means my approach would let you alter an attribute at most one
time.  It's not worth hacking around this; it's better to just stop
expecting (and recognizing) the marker flags.  (We have all the data
to distinguish the various cases that we need without the marker
flags.)

This brings Mentat in line with the thrust of newer Datomic versions,
but isn't compatible with Datomic, because (if I understand correctly)
Datomic automatically adds :db.{install,alter}/attribute assertions to
transactions.

I haven't purged the corresponding :db/ident and schema fragments just
yet:
- we might want them back
- we might want them in order to upgrade v1 and v2 databases to the
  new on-disk layout we're fleshing out (v3?).

* Post: Don't make :db/unique :db.unique/* imply :db/index true.

This patch avoids a potential bug with the "schema" materialized view.
If :db/unique :db.unique/value implies :db/index true, then what
happens when you _retract_ :db.unique/value?  I think Datomic defines
this in some way, but I really want the "schema" materialized view to
be a slice of "datoms" and not have these sort of ambiguities and
persistent effects.  Therefore, to ensure that we don't retract a
schema characteristic and accidentally change more than we intended
to, this patch stops having any schema characteristic imply any other
schema characteristic(s).  To achieve that, I added an
Option<Unique::{Value,Identity}> type to Attribute; this helps with
this patch, and also looks ahead to when we allow to retract
:db/unique attributes.

* Post: Allow to retract :db/ident.

* Post: Include more details about invalid schema changes.

The tests use strings, so they hide the chained errors which do in
fact provide more detail.

* Review comment: Fix outdated comment.

* Review comment: s/_SET/_SQL_LIST/.

* Review comment: Use a sub-select for checking cardinality.

This might be faster in practice.

* Review comment: Put `attribute::Unique` into its own namespace.
2017-03-20 13:18:59 -07:00
Nick Alexander
8beea55e39 Collect tempids after upsert resolution. Fixes #299. (#365) r=rnewman
* Test collecting tempids after upsert resolution. Fixes #299.

I just didn't finish and expose the tempid collection when I
implemented upsert resolution.  Here it is!

* Review comment: Take ownership of temp_id_map; avoid contains_key().
2017-03-20 11:34:38 -07:00
Nick Alexander
1801db2a77 Convert EDN transaction tests to Rust code. Fixes #271. (#364) r=rnewman
* Pre: Order datoms deterministically in debug output.

This makes comparison much easier, and avoids a whole class of
difficult problems when introducing pattern matching with placeholder
values.

* Pre: Don't rewrite ?txN and ?msN in debug module into_edn() methods.

* Convert EDN transaction tests to Rust code. Fixes #271.

This implements
https://github.com/mozilla/mentat/issues/271#issuecomment-283125963.
I'm using the EDN pattern matching functionality
internally (extensively!), but specifically working around the tricky
edges we encountered.  This should let us implement tests quickly (and
hopefully legibly) while not requiring us to encode as much behaviour
into non-standard EDN notations.
2017-03-20 11:29:17 -07:00
Emily Toop
fddc57d548 Use sqlite3_limit instead of hard-coded SQLITE_MAX_VARIABLE_NUMBER (#371) (#288) r=rnewman
* Part 1: added limits feature to rusqlite dependencies.
* Part 2: replace references to SQLITE_MAX_VARIABLE_NUMBER with sqlite3_limit.
* Move assertion check for correct number of variables in repeat_values to before call as this is where the variable is defined.
* Part 3: add tests
2017-03-13 09:39:19 -07:00
Richard Newman
30804e033a Use rusqlite 0.10.1. (#367) r=nalexander 2017-03-09 12:30:36 -08:00
Richard Newman
7bcf311db9 Pre: move SQLValueType to core, because it's so central.
Yes, this isn't tidy... but in order to be really tidy we'd need to
split up db into parts that don't depend on a particular SQLite library.
2017-03-08 17:41:49 -08:00
Richard Newman
70b112801c Implement projection and querying. (#353) r=nalexander
* Add a failing test for EDN parsing '…'.
* Expose a SQLValueType trait to get value_type_tag values out of a ValueType.
* Add accessors to FindSpec.
* Implement querying.
* Implement rudimentary projection.
* Export mentat_db::new_connection.
* Export symbols from mentat.
* Add rudimentary end-to-end query tests.
2017-03-06 14:40:10 -08:00
Nick Alexander
f86b24001f Add top-level Conn. Fixes #296. (#342) r=rnewman
* Add top-level `Conn`. Fixes #296.

This is a little different than the API rnewman and I originally
discussed in https://public.etherpad-mozilla.org/p/db-conn-thoughts.
A few notes:

- I was led to make a `Schema` instance the thing that is shared,
  rather than a `db::DB`.  It's possible that queries will want to
  know the current transaction at some point (to prevent races, or to
  query historical data), but that can be a future consideration.

- The generation number just allows for a cheap comparison.  I don't
  care to handle races to transact just yet; the long term plan might
  be to make embedding applications responsible for avoiding races, or
  we might handle queuing transactions and yielding report futures in
  Mentat itself.

- The sharing of the partition maps is a little more subtle than
  expected.  Partition maps are volatile: a successful Mentat
  transaction always advances the :db.part/tx partition, so it's not
  worth passing references around.  This means that consumers must
  clone in order to maintain just a single clone per transaction.

Clean some cruft.

* Review comments.
2017-03-03 15:03:59 -08:00
Nick Alexander
dcd9bcb1ce Extract partial storage abstraction; use error-chain throughout. Fixes #328. r=rnewman (#341)
* Pre: Drop unneeded tx0 from search results.

* Pre: Don't require a schema in some of the DB code.

The idea is to separate the transaction applying code, which is
schema-aware, from the concrete storage code, which is just concerned
with getting bits onto disk.

* Pre: Only reference Schema, not DB, in debug module.

This is part of a larger separation of the volatile PartitionMap,
which is modified every transaction, from the stable Schema, which is
infrequently modified.

* Pre: Fix indentation.

* Extract part of DB to new SchemaTypeChecking trait.

* Extract part of DB to new PartitionMapping trait.

* Pre: Don't expect :db.part/tx partition to advance when tx fails.

This fails right now, because we allocate tx IDs even when we shouldn't.

* Sketch a db interface without DB.

* Add ValueParseError; use error-chain in tx-parser.

This can be simplified when
https://github.com/Marwes/combine/issues/86 makes it to a published
release, but this unblocks us for now.  This converts the `combine`
error type `ParseError<&'a [edn::Value]>` to a type with owned
`Vec<edn::Value>` collections, re-using `edn::Value::Vector` for
making them `Display`.

* Pre: Accept Borrow<Schema> instead of just &Schema in debug module.

This makes it easy to use Rc<Schema> or Arc<Schema> without inserting
&* sigils throughout the code.

* Use error-chain in query-parser.

There are a few things to point out here:

- the fine grained error types have been flattened into one crate-wide
  error type; it's pretty easy to regain the granularity as needed.

- edn::ParseError is automatically lifted to
  mentat_query_parser::errors::Error;

- we use mentat_parser_utils::ValueParser to maintain parsing error
  information from `combine`.

* Patch up top-level.

* Review comment: Only `borrow()` once.
2017-02-24 15:33:48 -08:00
Joe Walker
40bca2df6d Remove most uses of use foo::* 2017-02-23 14:09:54 +00:00
Richard Newman
a10f68fdb7 Mark every project as being part of the workspace. r=nalexander
This allows `cargo test --all` to work.
2017-02-20 11:04:08 -08:00
Jordan Santell
bc2b2ec4c8 Change to_namespaced_keyword(s) to return a Result rather than Option to (#333)
reduce error handling throughout db code. Fixes #331. r=nalexander
2017-02-17 16:10:34 -08:00
Jordan Santell
6f67f8563b TypedValue::Keyword now wraps a NamespacedKeyword rather than a String. Fixes #203. r=nalexander" (#329) 2017-02-17 14:07:57 -08:00
Jordan Santell
a59f9583ac Store Idents as NamespacedKeywords, rather than Strings. Fixes #291. (#300)
r=ncalexander
2017-02-17 13:55:36 -08:00
Victor Porof
896d7f8f88 Add a span component to edn::Value, r=ncalexan
Signed-off-by: Victor Porof <victor.porof@gmail.com>
2017-02-17 18:31:26 +01:00
Nick Alexander
16e9740d8a Implement upsert resolution algorithm. (#186, #283). r=rnewman, f=jsantell
* Pre: Implement batch [a v] pair lookup.

* Pre: Add InternSet for sharing ref-counted handles to large values.

* Pre: Derive more for Entity.

* Pre: Return DB from creating; return TxReport from transact.

I explicitly am not supporting opening existing databases yet, let
alone upgrading databases from earlier versions.  That can follow fast
once basic transactions are supported.

* Pre: Parse string temporary ID entities; remove ValueOrLookupRef.

This adds TempId entities, but we can't disambiguate String temporary
IDs from values without the use of the schema, so there's no new value
branch.  Similarly, we can't disambiguate lookup-ref values from two
element list values without a schema, so we remove this entirely.
We'll handle the ambiguity later in the transactor.

* Persist partitions to SQL store; allocate transaction ID. (#186)

* Post: Test upserting with vectors.

This converts an existing test to EDN:
84a80f40f5/test/datomish/db_test.cljc (L193).

* Implement tempid upsert resolution algorithm. (#184)

* Post: Separate Tx out of DB.

This is very preliminary, since we don't have a real connection type
to manage transactions and their metadata yet.

* Post: Comment on implementation choices in the transactor.

* Review comment: Put long use lists on separate lines.

* Review comment: Accept String: Borrow<S> instead of just String.

* Review comment: Address nits.
2017-02-14 16:50:40 -08:00
Richard Newman
a87e5a3ec7 Move Schema from mentat_db to mentat_core, improve API. (#290)
* Move Schema from mentat_db to mentat_core.
* Define SchemaMap in terms of Entid, not i64.
* Add Schema::{is_attribute,identifies_attribute}.
* Add pointer to #291.
* Don't pass around 64-bit pointers to 64-bit integers.
2017-02-13 19:20:20 -08:00
Richard Newman
2e303f4837 Stub out mentat::q_once. (#289) r=nalexander
* Leave a pointer to issue 288.
* Re-export mentat_db::types::DB from mentat_db.
* Parse EDN strings in the query parser.
* Export 'public' API from mentat_query_parser's top level.
* Stub out mentat::q_once.
2017-02-13 10:30:02 -08:00
Jordan Santell
4e81733eed Change expecteddatoms and expectedtransaction to their kebab-case counterparts, for valid EDN style. Fixes #270. r=nalexander (#287) 2017-02-11 12:06:09 -08:00
Jordan Santell
6ce5d526a3 Store a bitfield in temporary search tables and expand to bit flags in the datoms table to investigate performance difference. Fixes #226. r=nalexander (#242) 2017-02-10 12:03:49 -08:00
Jordan Santell
1deed24f42 Remove usage of global imports in db module. Fixes #251. r=nalexander (#253) 2017-02-10 09:09:13 -08:00
Victor Porof
9be487ca7d Run pragmas to configure SQLite store and connection when we open a DB, r=ncalexan (#275)
Signed-off-by: Victor Porof <victor.porof@gmail.com>
2017-02-09 21:23:04 +01:00
Jordan Santell
21f7bdf493 Consolidate Entity::{Add, Retract} to Entity::AddOrRetract. Fixes #255. r=nalexander (#265) 2017-02-08 15:45:09 -08:00
Jordan Santell
688a644bd9 Ensure :db/index true for :db/unique _. Fixes #254. r=nalexander (#267) 2017-02-08 15:26:45 -08:00
Richard Newman
c111d4daff Move TypedValue into mentat_core. r=jsantell,nalexander 2017-02-08 14:20:10 -08:00
Nick Alexander
afafcd64a0 [tx] Start implementing bulk SQL insertion algorithms (#214). r=rnewman,jsantell
* Pre: Add some value conversion tests.

This is follow-up to earlier work.  Turn TypedValue::Keyword into
edn::Value::NamespacedKeyword.  Don't take a reference to
value_type_tag.

* Pre: Add repeat_values.

Requires itertools, so this commit is not stand-alone.

* Pre: Expose the first transaction ID as bootstrap::TX0.

This is handy for testing.

* Pre: Improve debug module.

* Pre: Bump rusqlite version for https://github.com/jgallagher/rusqlite/issues/211.

* Pre: Use itertools.

* Start implementing bulk SQL insertion algorithms. (#214)

This is slightly simpler re-expression of the existing Clojure
implementation.

* Post: Start generic data-driven transaction testing. (#188)

* Review comment: `use ::{SYMBOL}` instead of `use {SYMBOL}`.

* Review comment: Prefer bindings_per_statement to values_per_statement.
2017-02-08 14:04:32 -08:00
Victor Porof
c585715224 Don't depend on num and ordered-float in the db and query crates, r=ncalexan (#223)
Signed-off-by: Victor Porof <victor.porof@gmail.com>
2017-02-08 12:19:16 +01:00
Richard Newman
00c99196a2 Move db::type::{ValueType,Attribute} into a mentat_core crate. 2017-02-03 17:01:30 -08:00
Richard Newman
5b770a54cd Parse basic :find and :where clauses. (#211) r=nalexander
* Make Variable::from_symbol public.
* Implement basic parsing of queries.
* Use pinned dependencies the hard way to fix Travis.
* Bump ordered-float dependency to 0.4.0.
* Error coercions to use ?, and finishing the find interface.
2017-02-02 18:32:00 -08:00
Nick Alexander
81af295948 Start installing SQL schema. (#171) r=rnewman
* Start installing the SQLite store and bootstrapping the datom store.

* Review comment: Decomplect V2_IDENTS.

* Review comment: Decomplect V2_PARTS.

* Review comment: Pre: Expose Clojure's merge on Value instances.

* Review comment: Decomplect V2_SYMBOLIC_SCHEMA.

* Review comment: Decomplect V1_STATEMENTS.

* Review comment: Prefer ? to try!.

* Review comment: Fix typos; format; add TODOs.

* Review comment: Assert that Mentat `Schema` is valid upon creation.

* Review comment: Improve conversion to and from SQL values.

This patch factors the fundamental SQL conversion maps
between (rusqlite::Value, value_type_tag) and (edn::Value, ValueType)
through a new Mentat TypedValue.  (A future patch might rename this
fundamental type mentat::Value.)

To make certain conversion functions infallible, I removed
placeholders for :db.type/{instant,uuid,uri}.  (We could panic
instead, but there's no need to do that right now.)

* Review comment: Always uses bundled SQLite in rusqlite.

This avoids (runtime) failures in Travis CI due to old SQLite
versions.  See 432966ac77.

* Review comment: Move semantics in `from_sql_value_pair`.

* Review comment: DB_EXCISE_BEFORE_T instead of ...BEFORET (no underscore).

* Review comment: Move overview notes to the Wiki.
2017-01-25 16:13:56 -08:00