Commit graph

82 commits

Author SHA1 Message Date
Richard Newman
4d8e179a59
Expose component_attributes on Schema. (#623) r=nalexander
Some parts of the query engine and transactor need to know whether an
attribute is a component attribute, and sometimes want to do so in
a generated SQL query. This is one way to do that.
2018-04-03 14:25:53 -07:00
Richard Newman
6c54e1d370
Support :db/noHistory for attributes. (#622) r=nalexander
At this point we never discard history, but this completes the API support for doing so.
2018-04-03 14:23:46 -07:00
Richard Newman
9cc5cbf288
Rename the helpful variant, AttributeBuilder::new, to AttributeBuilder::helpful. (#625) r=nalexander 2018-04-03 14:23:20 -07:00
Richard Newman
66b892572c
Don't create a CommandExecutor if there are no observers. (#603) (#604) r=emily
* Don't create a CommandExecutor if there are no observers. (#603)
* Don't log if our executor channel goes away. This is routine.
2018-04-03 09:18:22 -07:00
Emily Toop
ab957948b4 Move to using watcher.
Simplify.

This has a watcher collect txid -> AttributeSet mappings each time a
transact occurs. On commit we retrieve those mappings and hand them over
to the observer service, which filters them and packages them up for
dispatch.

Tidy up
2018-03-20 16:27:35 +00:00
Emily Toop
d4365fa4cd Execute commands in a separate thread
Command Queue Executor to watch for new commands and execute on longer running background thread
2018-03-20 16:27:35 +00:00
Emily Toop
9f3d2c08b2 Populate changeset of attributes inside TxReport during transact.
Batch up TxReports for entire transaction.
Notify observers about committed transaction.
Store transaction observer service inside Conn
2018-03-20 16:27:35 +00:00
Emily Toop
8d60f2b3d1 Expose tx observation publicly 2018-03-20 16:27:35 +00:00
Emily Toop
dfa0d3e321 Add TxObservers, Commands and TxObservationService.
These are the base types upon which we will build our transaction observation
2018-03-20 16:27:35 +00:00
Richard Newman
1817ce7c0b Performance and cleanup. r=emily
* Use fixed-size arrays for bootstrap datoms, not vecs.
* Wide-ranging cleanup.

    This commit:
    - Deletes some dead code.
    - Marks some functions only used by tests as cfg(test).
    - Adds pub(crate) to a bunch of functions.
    - Cleans up a few other nits.
2018-03-06 09:03:00 -08:00
Richard Newman
f42ae35b70 Update cache on write. (#566) r=emily
* Use the cache to make constant queries super fast.
* Fix translate tests to match: we no longer generate SQL for many of them!
* Accumulate additions and removals into the cache.
    * Make attribute cache clone-on-write; store it in Metadata.
    * Allow caching of fulltext attributes, interning strings.
2018-03-06 09:01:20 -08:00
Richard Newman
54bd883c65 Follow-up: remove logging and such elsewhere in the codebase. 2018-02-21 11:51:45 -08:00
Richard Newman
e33fe71c47 Rework caching and use it inside the query engine. (#553) r=emily
This puts caching in mentat_db, adds a reverse lookup capability for
unique attributes, and populates bidirectional caches with a single
SQL cursor walk.

Differentiate between begin_read and begin_uncached_read.

Note that we still allow toggling within InProgress, because there might be
transient local state that makes starting a new transaction impossible.
2018-02-21 11:51:45 -08:00
Richard Newman
df3cdb5db6 Allow two datoms in the same transaction to have the same fulltext string. (#565) r=emily 2018-02-21 11:51:45 -08:00
Richard Newman
ae91603bd0 Provide an API for creating truly empty stores (#561) r=grisha
* Part 1: split create_current_version.

* Part 2: add Store::create_empty and Conn::empty.

* Part 3 - Expose 'open_empty' command via CLI
2018-02-16 02:01:00 -08:00
Richard Newman
d11810dca7 Fix warning in tx.rs. 2018-02-09 08:51:13 -08:00
Emily Toop
715d434945
Create generalized in-memory cache for attributes (#525)
* Nit: Alphabetical ordering of imports

* Create Cache and provide functions for calling it

* Get tests working. Move to using NamespacedKeyword over KnownEntid in function signature

* Add is_cached check to caching tests

* Move lazy and add/remove boolean flags to enums

* Move function definitions into generic trait and implement trait for AttributeCache

* Remove lazy cache and generalize cache

* Update tests

* Eager cache becomes simple key value store. AttributeMap handles attribute storing specifics

* Update tests to test presence of correct values in cache

* Move EagerCache, AttributeValueProvider and ValueProvider into mentat_db

* Add test for get_for_entid

* Add test for lookup attribute

* Make caches cloneable. Add value_for alongside values_for

* Use cache in attribute lookups

* Split test for values and value and add cardinality

* address review feedback r=rnewman
2018-02-07 10:56:12 -08:00
Grisha Kruglov
d848d954cf Issue 508 - Iterating transcation processor r=rnewman
Review comments
2018-02-06 12:24:12 -08:00
Richard Newman
e9ed103723 Revert "Make debug structs and functions non-public. r=grisha"
This reverts commit e817f67470.
2018-02-06 11:06:06 -08:00
Richard Newman
e817f67470 Make debug structs and functions non-public. r=grisha 2018-02-06 10:25:42 -08:00
Richard Newman
37a7c9ea48 Validate attributes installed after open. (#538) r=emily
Make AttributeBuilder optionally helpful, fix tests.
2018-02-01 09:29:04 -08:00
Edouard Oger
3bf7459315 Allow customers to assert facts about the current transaction. (#225) r=rnewman
Also move `now` into core, implement microsecond truncation.

This is so we don't return a more granular -- and thus subtly different --
timestamp in a `TxReport` than we put into the store.
2018-01-29 16:46:04 -08:00
Richard Newman
ef9f2d9c51 Don't allow violation of cardinality-one restrictions within a single tx. (#531) r=nalexander 2018-01-23 15:11:38 -08:00
Richard Newman
812f10b3e4 Add an EntityBuilder abstraction. r=nalexander,emily
This includes two other changes:

* Split transact to expose an interface for TermWithTempIds.
* Return TxReport from each InProgress operation, not from commit.
2018-01-23 08:52:09 -08:00
Richard Newman
3d28949add Describe the default core schema, v1 (:db.schema/core). r=nalexander 2018-01-23 08:51:58 -08:00
Richard Newman
4acc6d0658 InProgressRead, KnownEntid. r=nalexander,emily
Improve naming of read-only transactions.
    Implement entid_for_type.
    Simplify get_attribute.
    Name ignored var in algebrizer.
    Comment attribute_for_ident.
    Make KnownEntid a core concept.
    Expose lookup_value_for_attribute.
    Implement HasSchema and a new query encapsulation on Conn.
    Pre: export Queryable.
2018-01-23 08:40:18 -08:00
Richard Newman
6797a606b5 Preliminary work for vocabulary management. r=emily,nalexander
Pre: export AttributeBuilder from mentat_db.
Pre: fix module-level comment for tx/src/entities.rs.
Pre: rename some `to_` conversions to `into_`.
Pre: make AttributeBuilder::unique less verbose.
Pre: split out a HasSchema trait to abstract over Schema.
Pre: rename SchemaMap/schema_map to AttributeMap/attribute_map.
Pre: TypedValue/NamespacedKeyword conversions.
Pre: turn Unique and ValueType into TypedValue::Keyword.
Pre: export IntoResult.
Pre: export NamespacedKeyword from mentat_core.
Pre: use intern_set in tx.
Pre: add InternSet::len.
Pre: comment gardening.
Pre: remove inaccurate TODO from TxReport comment.
2018-01-23 08:25:32 -08:00
Thom
9740cafdbd Automatically remove trailing whitespace from text files. (#527) r=rnewman
This was done using the following shell script:

```
find . -type f -not -path "*target*" \
       '(' -name '*.rs' -o -name '*.md' -o -name '*.toml' ')' -print0 | \
    xargs -0 sed -i '' -E 's/[[:space:]]*$//'
```

Which is admittedly imperfect, but manages to hit everything that was a problem in this repo.
2018-01-19 21:21:04 -06:00
Richard Newman
95e95d735e
Correct an assert relating Datalog projection and SQL column counts. (#519) r=tcsc
* Correct an assert relating Datalog projection and SQL column counts. (#517)
* Fix a comment that shouldn't be a doc comment.
2018-01-17 16:36:15 -06:00
Grisha Kruglov
1589104841
Force SQLite temp files to be stored in memory (#505) r=rnewman (#506)
* Force SQLite temp files to be stored in memory (#505) r=rnewman
2017-12-20 01:52:09 -05:00
Richard Newman
95b9c7f7f5
Atomic multi-tx (#489). r=emily,nalexander
* Pre: rename begin_transaction to begin_tx_application.

* Take an EXCLUSIVE transaction when bootstrapping, and an IMMEDIATE transaction when writing.

This avoids the remote possibility of another write sneaking in the door
while we're preparing to write, avoids us needing to upgrade locks, etc.

  After a BEGIN IMMEDIATE, no other database connection will be able to write
  to the database or do a BEGIN IMMEDIATE or BEGIN EXCLUSIVE. Other processes
  can continue to read from the database, however.

  An exclusive transaction causes EXCLUSIVE locks to be acquired on all
  databases. After a BEGIN EXCLUSIVE, no other database connection except for
  read_uncommitted connections will be able to read the database and no other
  connection without exception will be able to write the database until the
  transaction is complete.

* Hacky implementation of atomic multi-tx.

* Hold the last report, returning the InProgress from each operation.

* Rewrite transact in terms of InProgress.

* Test rollback.

* Remove unused imports.

* Don't use Rc for transaction reports.

* Pre: break out USER0 as a part boundary constant.

* Export TX0 and USER0 from mentat_db. This is for testing.

* Review comments: commenting.

* Test tempid allocation and rollback.
2017-12-05 07:58:24 -08:00
Emily Toop
55588209c2
CLI (#493)
* Create mentat command line.
* Create tools directory containing new crate for mentat_cli.
* Add simple cli with mentat prompt.

* Remove rustc-serialize dependency

* Open DB inside CLI (#452) (#463)

* Open named database OR default to in memory database if no name provided

Rearrange workspace to allow import of mentat crate in cli crate

Create store object inside repl when started for connecting to mentat

Use provided DB name to open connection in store

Accept DB name as command line arg.

Open on CLI start

Implement '.open' command to open desired DB from inside CLI

* Implement Close command to close current DB.
* Closes existing open db and opens new in memory db

* Review comment: Use `combine` to parse arguments.

Move over to using Result rather than enums with err

* Accept and parse EDN Query and Transact commands (#453) (#465)

* Parse query and transact commands

* Implement is_complete for transactions and queries

* Improve query parser. Am still not happy with it though.

There must be some way that I can retain the eof() after the `then` that means I don't have to move the skip on spaces and eof

Make in process command storing clearer.

Add comments around in process commands.
Add alternative commands for transact/t and query/q

* Address review comments r=nalexander.

* Bump rust version number.
* Use `bail` when throwing errors.
* Improve edn parser.
* Remove references to unused `more` flag.
* Improve naming of query and transact commands.

* Send queries and transactions to mentat and output the results (#466)

* Send queries and transactions to mentat and output the results

move outputting query and transaction results out of store and into repl

* Add query and transact commands to help

* Execute queries and transacts passed in at startup

* Address review comments =nalexander.

* Bump rust version number.
* Use `bail` when throwing errors.
* Improve edn parser.
* Remove references to unused `more` flag.
* Improve naming of query and transact commands.

* Execute command line args in order

* Addressing rebase issues

* Exit CLI (#457) (#484) r-rnewman

* Implement exit command for cli tool

* Address review comments r=rnewman

* Include exit commands in help

* Show schema of current DB (#487)

* Fixing rebase issues

* addressing nit

* Match updated dependencies on CLI crate and remove unused import
2017-11-21 16:56:16 +00:00
Richard Newman
c600152d78
Update some dependencies. (#492) r=etoop
* Update some dependencies.

* Update rusqlite to 0.12.

* Update error-chain to a forked version that implements Sync.

* Fix some compiler warnings.

* Remove unused imports in tests.

* Parse errors no longer naturally print with the expected symbol.
2017-11-21 16:24:08 +00:00
Richard Newman
ea0e9d4c7b Allow instants to pass through schema validation. (#481) r=fluffyemily
* Allow instants to pass through schema validation.
* Expand cases in SchemaTypeChecking to catch enum bugs.
2017-06-16 09:15:29 -07:00
Richard Newman
c2ec1a6bdf Pre: move Either to mentat_core::util. 2017-06-15 10:28:02 -07:00
Richard Newman
9a12ced317 Don't allow callers to specify arbitrary new entity IDs. (#447) r=nalexander
This commit adds a check to the partition map that a provided entity ID
has been mentioned (i.e., is present in the start:index range of one of
our partitions).

We introduce a newtype for known entity IDs, using this internally in
the tx expander to track user-provided entids that have passed the above
check (and IDs that we allocate as part of tempid processing). This
newtype is stripped prior to tx assertion.

In order that DB tests can continue to write

  [:db/add 111 :foo/bar 222]

we add an additional fake partition to our test connections, ranging
from 100 to 1000.
2017-06-09 15:45:26 -07:00
Nick Alexander
c165972684 Post: Reject at parse-time reversed attributes in direct notation with bad values.
This is an optimization that trades rejecting inputs earlier at the
cost of expressive error messages.  It should be possible to recover
the error messages, however.

This will reject input like `[:db/{add,retract} v :attribute/_reversed NOT-AN-ENTITY]`.
2017-06-08 10:30:31 -07:00
Nick Alexander
59a710f80f Review comments: another test, add unreversed(). 2017-06-08 10:30:31 -07:00
Nick Alexander
eb220528bf Post: Indent. 2017-06-08 10:30:31 -07:00
Nick Alexander
d88823e7c4 Handle :attribute/_reverse in transactor. Fixes #187
There are two broad approaches:

1) Handle reverse attribute notation dynamically, in the style that
   Datomic does.  This is the most flexible, but it's not a good fit
   given that we produce strongly typed output from the parser.
   Strongly typed input to the transactor has had many benefits, so I
   don't want to roll it back for a relatively unimportant feature
   like reverse notation -- especially not since Mentat does not
   require :db.install/_attribute to modify schema attributes.

2) Handle reverse attribute in the parser itself, so that we can
   produce strongly typed parser output while restricting the input.
   I implemented this first and discovered that it's very difficult to
   give sensible error messages in common cases.

In any case, the bulk of the code is the same between the two
approaches, and I wrote the tests for the dynamic version (with error
output), so that's what I'm rolling with.

This patch preserves the existing indentation, to highlight the
differences.  The next patch will indent.
2017-06-08 10:30:31 -07:00
Nick Alexander
0be78cf956 Pre: Extract entity_*_into_term_* helpers. 2017-06-08 10:30:31 -07:00
Nick Alexander
4b0881a957 Pre: Push bookkeeping into an InProcess struct. 2017-06-08 10:30:31 -07:00
Nick Alexander
05129cefbb Pre: Use ValueType rather than Attribute to convert edn::Value to TypedValue.
This is expedient now, but might require work in the future to achieve
better error messages.
2017-06-08 10:30:31 -07:00
Nick Alexander
2650fe163d Pre: Intern lookup_ref by reference. 2017-06-08 10:30:31 -07:00
Nick Alexander
a4fc04ea86 Pre: Crib map_{left,right} for Either. 2017-06-08 10:30:31 -07:00
Nick Alexander
d1ac752de6 Parse without copying; parse keyword maps using macros.
This is a big commit, but it breaks into two conceptual pieces.  The
first is to "parse without copying".  We replace a stream of an owned
collection of edn::ValueAndSpan and instead have a stream of a
borrowed collection of &edn::ValueAndSpan references.  (Generally,
this is represented as an iterator over a slice, but it can be over
other things too.)  Cloning such iterators is constant time, which
improves on cloning an owned collection of edn::ValueAndSpan, which is
linear time in the length of the collection and additional time
depending on the complexity of the EDN values.

The second conceptual piece is to parse keyword maps using a special
parser and a macro to build the parser implementations.  Before, we
created a new edn::ValueAndSpan::Map to represent a keyword map in
vector form; since we're working with &edn::ValueAndSpan references
now, we can't create an &edn::ValueAndSpan reference with an
appropriate lifetime.  Therefore we generalize the concept of
iteration slightly and turn keyword maps in map form into linear
iterators by flattening the value maps.  This is a potentially
obscuring transformation, so we have to take care to protect against
some failure cases.  (See the comments and the tests in the code.)

After these changes, parsing using `combine` is linear time (and
reasonably fast).
2017-05-18 10:17:13 -07:00
Richard Newman
3d4615fb8c Allow opening a DB. (#462) r=fluffyemily 2017-05-09 09:42:35 -07:00
Richard Newman
daca8def57 UUIDs and instants. Fixes #44, #45, #426, #427. (#438) r=nalexander
* Pre: unused import in translate.rs.

* Part 2: take a dependency on rusqlite for query arguments.

* Part 1: flatten V2 schema into V1. Add UUID and URI.

Bump expected ident and bootstrap datom count in tests.

* Part 5: parse edn::Value::Uuid.

* Part 3: extend ValueType and TypedValue to include Uuid.

* Part 4: add Uuid to query arguments.

* Part 6: extend db to support Uuid.

* Part 8: add a tx-parser test for #f NaN and #uuid.

* Part 7: parse and algebrize UUIDs in queries.

* Part 1: parse #inst in EDN and throughout query engine.

* Part 3: handle instants in db.

* Part 2: instants never matches integers in queries.

* Part 4: use DateTime for tx_instants.

* Add a test for adding and querying UUIDs and instants.

* Review comments.
2017-04-28 20:11:55 -07:00
Nick Alexander
5369f03464 Improve parsing of nested edn::ValueAndSpan streams. r=rnewman (#393)
* Pre: Expose more in edn.

* Pre: Make it easier to work with ValueAndSpan.

with_spans() is a temporary hack, needed only because I don't care to
parse the bootstrap assertions from text right now.

* Part 1a: Add `value_and_span` for parsing nested `edn::ValueAndSpan` instances.

I wasn't able to abstract over `edn::Value` and `edn::ValueAndSpan`;
there are multiple obstacles.  I chose to roll with
`edn::ValueAndSpan` since it exposes the additional span information
that we will want to form good error messages in the future.

* Part 1b: Add keyword_map() parsing an `edn::Value::Vector` into an `edn::Value::map`.

* Part 1c: Add `Log`/`.log(...)` for logging parser progress.

This is a terrible hack, but it sure helps to debug complicated nested
parsers.  I don't even know what a principled approach would look
like; since our parser combinators are so frequently expressed in
code, it's hard to imagine a data-driven interpreter that can help
debug things.

* Part 2: Use `value_and_span` apparatus in tx-parser/.

I break an abstraction boundary by returning a value column
`edn::ValueAndSpan` rather than just an `edn::Value`.  That is, the
transaction processor shouldn't care where the `edn::Value` it is
processing arose -- even we care to track that information we should
bake it into the `Entity` type.  We do this because we need to
dynamically parse the value column to support nested maps, and parsing
requires a full `edn::ValueAndSpan`.  Alternately, we could cheat and
fake the spans when parsing nested maps, but that's potentially
expensive.

* Part 3: Use `value_and_span` apparatus in query-parser/.

* Part 4: Use `value_and_span` apparatus in root crate.

* Review comment: Make Span and SpanPosition Copy.

* Review comment: nits.

* Review comment: Make `or` be `or_exactly`.

I baked the eof checking directly into the parser, rather than using
the skip and eof parsers.  I also took the time to restore some tests
that were mistakenly commented out.

* Review comment: Extract and use def_matches_* macros.

* Review comment: .map() as late as possible.
2017-04-06 10:06:28 -07:00
Richard Newman
a5023c70cb Use Rc for TypedValue, Variable, and query Ident keywords. (#395) r=nalexander
Part 1, core: use Rc for String and Keyword.
Part 2, query: use Rc for Variable.
Part 3, sql: use Rc for args in SQLiteQueryBuilder.
Part 4, query-algebrizer: use Rc.
Part 5, db: use Rc.
Part 6, query-parser: use Rc.
Part 7, query-projector: use Rc.
Part 8, query-translator: use Rc.
Part 9, top level: use Rc.
Part 10: intern Ident and IdentOrKeyword.
2017-04-02 21:38:36 -07:00