Commit graph

269 commits

Author SHA1 Message Date
Richard Newman
909b2a8be5 Refactoring: split up the projector crate. No other code changes. 2018-04-09 10:26:09 -07:00
Richard Newman
3f8464e8ed Implement vocabulary-driven schema upgrades. (#595) r=emily 2018-04-09 09:47:49 -07:00
Richard Newman
a5cda7c3e9 Allow passing a TermBuilder to be transacted by InProgress; add TermBuilder::is_empty. r=emily 2018-04-09 09:47:49 -07:00
Emily Toop
175958e754 Address review comments @rnewman 2018-04-06 10:46:15 +01:00
Emily Toop
9741435026 Add helper functions for FFI. Many of these will go away as we expose the entity builder 2018-04-06 10:46:15 +01:00
Emily Toop
7382e3297d Add QueryBuilder to make querying over FFI easier 2018-04-06 10:46:15 +01:00
Richard Newman
4d8e179a59
Expose component_attributes on Schema. (#623) r=nalexander
Some parts of the query engine and transactor need to know whether an
attribute is a component attribute, and sometimes want to do so in
a generated SQL query. This is one way to do that.
2018-04-03 14:25:53 -07:00
Richard Newman
6c54e1d370
Support :db/noHistory for attributes. (#622) r=nalexander
At this point we never discard history, but this completes the API support for doing so.
2018-04-03 14:23:46 -07:00
Richard Newman
9cc5cbf288
Rename the helpful variant, AttributeBuilder::new, to AttributeBuilder::helpful. (#625) r=nalexander 2018-04-03 14:23:20 -07:00
Emily Toop
9f30fe6295 Create Mentat FFI and expose observers (#574)
* Tidy up and add txid at beginning of transaction

* Add ffi crate and new_store function

* Add register and unregister observer FFI, Store and Conn functions.
Also add android logging facilities

* Add function for fetching entids for attribute strings

* Add functions for iterating through TxReports

* Add sync to ffi boundary

* Move Extern types from submodule to lib in FFI.
For some reason, if these types are in a submodule, even if they are publically used, the functions inside the FFI are not found in
Android. Works for iOS though. To be investigated later....

* Return to passing TxReports to observer function.
Also, remove some debug

* Expose DateTime and Utc publically

* Use Store in observer tests
2018-03-20 19:16:32 +00:00
Emily Toop
ab957948b4 Move to using watcher.
Simplify.

This has a watcher collect txid -> AttributeSet mappings each time a
transact occurs. On commit we retrieve those mappings and hand them over
to the observer service, which filters them and packages them up for
dispatch.

Tidy up
2018-03-20 16:27:35 +00:00
Emily Toop
d4365fa4cd Execute commands in a separate thread
Command Queue Executor to watch for new commands and execute on longer running background thread
2018-03-20 16:27:35 +00:00
Emily Toop
ecc4a7a35a Add tests 2018-03-20 16:27:35 +00:00
Emily Toop
c2e5052877 Allow registration and unregistration of transaction observers from Conn 2018-03-20 16:27:35 +00:00
Emily Toop
9f3d2c08b2 Populate changeset of attributes inside TxReport during transact.
Batch up TxReports for entire transaction.
Notify observers about committed transaction.
Store transaction observer service inside Conn
2018-03-20 16:27:35 +00:00
Richard Newman
833ff92436
Simple aggregates. (#584) r=emily
* Pre: use debugcli in VSCode.
* Pre: wrap subqueries in parentheses in output SQL.
* Pre: add ExistingColumn.

This lets us make reference to columns by name, rather than only
pointing to qualified aliases.

* Pre: add Into for &str to TypedValue.
* Pre: add Store.transact.
* Pre: cleanup.
* Parse and algebrize simple aggregates. (#312)
* Follow-up: print aggregate columns more neatly in the CLI.
* Useful ValueTypeSet helpers.
* Allow for entity inequalities.
* Add 'differ', which is a ref-specialized not-equals.
* Add 'unpermute', a function for getting unique, distinct pairs from bindings.
* Review comments.
* Add 'the' pseudo-aggregation operator.

This allows for a corresponding value to be returned when a query
includes one 'min' or 'max' aggregate.
2018-03-12 15:18:50 -07:00
Richard Newman
f42ae35b70 Update cache on write. (#566) r=emily
* Use the cache to make constant queries super fast.
* Fix translate tests to match: we no longer generate SQL for many of them!
* Accumulate additions and removals into the cache.
    * Make attribute cache clone-on-write; store it in Metadata.
    * Allow caching of fulltext attributes, interning strings.
2018-03-06 09:01:20 -08:00
Richard Newman
bead9752bd Add InProgress.import to load transaction data from a file. r=emily
It's not streaming, but it'll do.
2018-03-06 08:59:30 -08:00
Richard Newman
e33fe71c47 Rework caching and use it inside the query engine. (#553) r=emily
This puts caching in mentat_db, adds a reverse lookup capability for
unique attributes, and populates bidirectional caches with a single
SQL cursor walk.

Differentiate between begin_read and begin_uncached_read.

Note that we still allow toggling within InProgress, because there might be
transient local state that makes starting a new transaction impossible.
2018-02-21 11:51:45 -08:00
Richard Newman
ae91603bd0 Provide an API for creating truly empty stores (#561) r=grisha
* Part 1: split create_current_version.

* Part 2: add Store::create_empty and Conn::empty.

* Part 3 - Expose 'open_empty' command via CLI
2018-02-16 02:01:00 -08:00
Grisha Kruglov
93e5dff9c8
Revised uploader flow (battle-tested); CLI support for sync (#557) r=rnewman 2018-02-16 01:44:28 -08:00
Emily Toop
48ffa20d4c Add retract_kw 2018-02-15 18:47:58 +00:00
Emily Toop
9655c4b85d Add retract to entity builder. (#559) r=rnewman 2018-02-15 09:44:06 -08:00
Richard Newman
01d9b83a9b
Add EntityBuilder.add_kw. (#558) r=emily
* Add EntityBuilder.add_kw.

This allows you to skip your own attribute lookups, at the cost of
potentially doing the work more than once.

Also does value type checking.
2018-02-15 08:22:34 -08:00
Emily Toop
dec86bb2c5
publically expose KnownEntid (#554) 2018-02-14 19:06:50 +00:00
Richard Newman
23e11fabe6
Add a var! macro. (#548)(#550) r=emily 2018-02-14 09:32:37 -08:00
Kit Cambridge
a6341f6fd6 Implement q_prepare with pre-bound variables. r=rnewman 2018-02-07 21:48:05 -08:00
Emily Toop
715d434945
Create generalized in-memory cache for attributes (#525)
* Nit: Alphabetical ordering of imports

* Create Cache and provide functions for calling it

* Get tests working. Move to using NamespacedKeyword over KnownEntid in function signature

* Add is_cached check to caching tests

* Move lazy and add/remove boolean flags to enums

* Move function definitions into generic trait and implement trait for AttributeCache

* Remove lazy cache and generalize cache

* Update tests

* Eager cache becomes simple key value store. AttributeMap handles attribute storing specifics

* Update tests to test presence of correct values in cache

* Move EagerCache, AttributeValueProvider and ValueProvider into mentat_db

* Add test for get_for_entid

* Add test for lookup attribute

* Make caches cloneable. Add value_for alongside values_for

* Use cache in attribute lookups

* Split test for values and value and add cardinality

* address review feedback r=rnewman
2018-02-07 10:56:12 -08:00
Richard Newman
66e6fef75e Define Store, use TabWriter in the CLI for aligning columnar output. (#540) r=emily
* Define Store, which is a simple container for a SQLite connection and a Conn.
  This is a breaking change.
* Return the FindSpec as part of QueryOutput, not just results.
* Switch to using stderr in appropriate places in CLI.
* Print columns in CLI output.
2018-02-01 09:29:07 -08:00
Richard Newman
37a7c9ea48 Validate attributes installed after open. (#538) r=emily
Make AttributeBuilder optionally helpful, fix tests.
2018-02-01 09:29:04 -08:00
Richard Newman
2614f498be Ergonomics improvements, including a kw macro. (#537) r=emily
* Add TypedValue::instant(micros).
* Add From<f64> for TypedValue.
* Add lookup_values_for_attribute to Conn.
* Add q_explain to Queryable.
* Expose an iterator over FindSpec's columns.
* Export edn from mentat crate. Export QueryExecutionResult.
* Implement Display for Variable and Element.
* Introduce a `kw` macro.

    This allows you to write:

    ```rust
    kw!(:foo/bar)
    ```

    instead of

    ```rust
    NamespacedKeyword::new("foo", "bar")
    ```

    … and it's more efficient, too.

Add `mentat::open`, eliminate use of `mentat_db` in some places.
2018-02-01 09:27:23 -08:00
Richard Newman
6ed5413cd4 Implement simple vocabulary management. (#504) r=emily,nalexander
Bump version to 0.5.1 to reflect this change.
2018-01-23 08:52:15 -08:00
Richard Newman
812f10b3e4 Add an EntityBuilder abstraction. r=nalexander,emily
This includes two other changes:

* Split transact to expose an interface for TermWithTempIds.
* Return TxReport from each InProgress operation, not from commit.
2018-01-23 08:52:09 -08:00
Richard Newman
4acc6d0658 InProgressRead, KnownEntid. r=nalexander,emily
Improve naming of read-only transactions.
    Implement entid_for_type.
    Simplify get_attribute.
    Name ignored var in algebrizer.
    Comment attribute_for_ident.
    Make KnownEntid a core concept.
    Expose lookup_value_for_attribute.
    Implement HasSchema and a new query encapsulation on Conn.
    Pre: export Queryable.
2018-01-23 08:40:18 -08:00
Richard Newman
6797a606b5 Preliminary work for vocabulary management. r=emily,nalexander
Pre: export AttributeBuilder from mentat_db.
Pre: fix module-level comment for tx/src/entities.rs.
Pre: rename some `to_` conversions to `into_`.
Pre: make AttributeBuilder::unique less verbose.
Pre: split out a HasSchema trait to abstract over Schema.
Pre: rename SchemaMap/schema_map to AttributeMap/attribute_map.
Pre: TypedValue/NamespacedKeyword conversions.
Pre: turn Unique and ValueType into TypedValue::Keyword.
Pre: export IntoResult.
Pre: export NamespacedKeyword from mentat_core.
Pre: use intern_set in tx.
Pre: add InternSet::len.
Pre: comment gardening.
Pre: remove inaccurate TODO from TxReport comment.
2018-01-23 08:25:32 -08:00
Richard Newman
224570fb45 Switch InProgress to be mutated in place. r=nalexander,emily
This is a breaking change, and involves a very small additional cost
in managing the partition map, but it makes it much more feasible to
implement traits on InProgress: now they don't need to chain back a
new InProgress each time.

Bump version to 0.5 to reflect the change in InProgress.
2018-01-23 08:14:13 -08:00
Emily Toop
8cc7f0b72e
Expose Uuid from the top level crate (#529) 2018-01-22 16:29:25 +00:00
Thom
023fd9b70b
Add a command to the CLI that displays the SQL and query plan for a query. Fixes #428. (#523) r=rnewman 2018-01-19 19:44:39 -05:00
Richard Newman
e8ec59e464
Implement a simple direct lookup API. Fixes #111 (#503) r=grisha
* Add some helpers and refactor how queries are run (once).
* Implement lookup_value_for_attribute.
* Add a multi-value test for lookup_value_for_attribute.
2017-12-11 11:08:10 -08:00
Richard Newman
b7fb44a5a6 Add a comment to InProgress. 2017-12-07 12:19:59 -08:00
Richard Newman
a3b8fd3022 Add results type unwrapping helpers. 2017-12-06 14:47:29 -08:00
Richard Newman
95b9c7f7f5
Atomic multi-tx (#489). r=emily,nalexander
* Pre: rename begin_transaction to begin_tx_application.

* Take an EXCLUSIVE transaction when bootstrapping, and an IMMEDIATE transaction when writing.

This avoids the remote possibility of another write sneaking in the door
while we're preparing to write, avoids us needing to upgrade locks, etc.

  After a BEGIN IMMEDIATE, no other database connection will be able to write
  to the database or do a BEGIN IMMEDIATE or BEGIN EXCLUSIVE. Other processes
  can continue to read from the database, however.

  An exclusive transaction causes EXCLUSIVE locks to be acquired on all
  databases. After a BEGIN EXCLUSIVE, no other database connection except for
  read_uncommitted connections will be able to read the database and no other
  connection without exception will be able to write the database until the
  transaction is complete.

* Hacky implementation of atomic multi-tx.

* Hold the last report, returning the InProgress from each operation.

* Rewrite transact in terms of InProgress.

* Test rollback.

* Remove unused imports.

* Don't use Rc for transaction reports.

* Pre: break out USER0 as a part boundary constant.

* Export TX0 and USER0 from mentat_db. This is for testing.

* Review comments: commenting.

* Test tempid allocation and rollback.
2017-12-05 07:58:24 -08:00
Richard Newman
df90c366af
Partial work from simple aggregates work (#497) r=nalexander
* Pre: make FindQuery, FindSpec, and Element non-Clone.
* Pre: make query translator return a Result.
* Pre: make projection return a Result.
* Pre: refactor query parser in preparation for parsing aggregates.
* Pre: rename PredicateFn -> QueryFunction.
* Pre: expose more about bound variables from CC.
* Pre: move ValueTypeSet to core.
2017-11-30 15:02:07 -08:00
Richard Newman
9a12ced317 Don't allow callers to specify arbitrary new entity IDs. (#447) r=nalexander
This commit adds a check to the partition map that a provided entity ID
has been mentioned (i.e., is present in the start:index range of one of
our partitions).

We introduce a newtype for known entity IDs, using this internally in
the tx expander to track user-provided entids that have passed the above
check (and IDs that we allocate as part of tempid processing). This
newtype is stripped prior to tx assertion.

In order that DB tests can continue to write

  [:db/add 111 :foo/bar 222]

we add an additional fake partition to our test connections, ranging
from 100 to 1000.
2017-06-09 15:45:26 -07:00
Nick Alexander
d1ac752de6 Parse without copying; parse keyword maps using macros.
This is a big commit, but it breaks into two conceptual pieces.  The
first is to "parse without copying".  We replace a stream of an owned
collection of edn::ValueAndSpan and instead have a stream of a
borrowed collection of &edn::ValueAndSpan references.  (Generally,
this is represented as an iterator over a slice, but it can be over
other things too.)  Cloning such iterators is constant time, which
improves on cloning an owned collection of edn::ValueAndSpan, which is
linear time in the length of the collection and additional time
depending on the complexity of the EDN values.

The second conceptual piece is to parse keyword maps using a special
parser and a macro to build the parser implementations.  Before, we
created a new edn::ValueAndSpan::Map to represent a keyword map in
vector form; since we're working with &edn::ValueAndSpan references
now, we can't create an &edn::ValueAndSpan reference with an
appropriate lifetime.  Therefore we generalize the concept of
iteration slightly and turn keyword maps in map form into linear
iterators by flattening the value maps.  This is a potentially
obscuring transformation, so we have to take care to protect against
some failure cases.  (See the comments and the tests in the code.)

After these changes, parsing using `combine` is linear time (and
reasonably fast).
2017-05-18 10:17:13 -07:00
Richard Newman
c95ec13ffe Begin moving web server to a separate crate. (#448) r=bgrins
This doesn't yet introduce a working Cargo.toml for 'mentatweb', but it
does allow RLS to build correctly without errors, and it reduces the
core library's dependency space, which is more important in the short
term.
2017-05-10 02:25:59 -07:00
Richard Newman
059b9d1182 Expose mentat_core::{TypedValue,ValueType} and conn::{Conn,Metadata}. (#443) 2017-05-03 15:49:29 -07:00
Richard Newman
523d5ea5f1 Bump dependency versions. r=bgrins. (#441) 2017-05-03 12:53:16 -07:00
Richard Newman
bc63744aba Add :limit to queries (#420) r=nalexander
* Pre: put query parts in alphabetical order.
* Pre: rename 'input' to 'query' in translate tests.
* Part 1: parse :limit.
* Part 2: validate and escape variable parameters in SQL.
* Part 3: algebrize and translate limits.
2017-04-19 16:16:19 -07:00
Richard Newman
bffefe7e6b Review comments for #418. 2017-04-18 13:50:58 -07:00