Commit graph

666 commits

Author SHA1 Message Date
Emily Toop 9f30fe6295 Create Mentat FFI and expose observers (#574)
* Tidy up and add txid at beginning of transaction

* Add ffi crate and new_store function

* Add register and unregister observer FFI, Store and Conn functions.
Also add android logging facilities

* Add function for fetching entids for attribute strings

* Add functions for iterating through TxReports

* Add sync to ffi boundary

* Move Extern types from submodule to lib in FFI.
For some reason, if these types are in a submodule, even if they are publically used, the functions inside the FFI are not found in
Android. Works for iOS though. To be investigated later....

* Return to passing TxReports to observer function.
Also, remove some debug

* Expose DateTime and Utc publically

* Use Store in observer tests
2018-03-20 19:16:32 +00:00
Emily Toop ab957948b4 Move to using watcher.
Simplify.

This has a watcher collect txid -> AttributeSet mappings each time a
transact occurs. On commit we retrieve those mappings and hand them over
to the observer service, which filters them and packages them up for
dispatch.

Tidy up
2018-03-20 16:27:35 +00:00
Emily Toop d4365fa4cd Execute commands in a separate thread
Command Queue Executor to watch for new commands and execute on longer running background thread
2018-03-20 16:27:35 +00:00
Emily Toop ecc4a7a35a Add tests 2018-03-20 16:27:35 +00:00
Emily Toop c2e5052877 Allow registration and unregistration of transaction observers from Conn 2018-03-20 16:27:35 +00:00
Emily Toop 9f3d2c08b2 Populate changeset of attributes inside TxReport during transact.
Batch up TxReports for entire transaction.
Notify observers about committed transaction.
Store transaction observer service inside Conn
2018-03-20 16:27:35 +00:00
Emily Toop 8d60f2b3d1 Expose tx observation publicly 2018-03-20 16:27:35 +00:00
Emily Toop dfa0d3e321 Add TxObservers, Commands and TxObservationService.
These are the base types upon which we will build our transaction observation
2018-03-20 16:27:35 +00:00
Richard Newman f71b2b207e Expand the README to give a guide to the crates in the repo. 2018-03-19 14:35:18 -07:00
Richard Newman 16a66517e4
Err, don't panic, on unbound variable (#518) (#590) r=emily
* Pre: switch a 'panic' to an 'unreachable'.
* Make candidate_type_column fallible. (#518)
2018-03-15 11:39:24 -07:00
Richard Newman 994a3e65e2
Tests and fixes for aggregates over different or unknown types. (#588) r=emily 2018-03-15 07:14:06 -07:00
Richard Newman df58de52f4
Correctly parse and unescape quotes etc. inside EDN strings. (#434) (#589) 2018-03-15 07:13:27 -07:00
Richard Newman ea52e214af
Small README tweaks. 2018-03-13 04:01:19 +00:00
Richard Newman 833ff92436
Simple aggregates. (#584) r=emily
* Pre: use debugcli in VSCode.
* Pre: wrap subqueries in parentheses in output SQL.
* Pre: add ExistingColumn.

This lets us make reference to columns by name, rather than only
pointing to qualified aliases.

* Pre: add Into for &str to TypedValue.
* Pre: add Store.transact.
* Pre: cleanup.
* Parse and algebrize simple aggregates. (#312)
* Follow-up: print aggregate columns more neatly in the CLI.
* Useful ValueTypeSet helpers.
* Allow for entity inequalities.
* Add 'differ', which is a ref-specialized not-equals.
* Add 'unpermute', a function for getting unique, distinct pairs from bindings.
* Review comments.
* Add 'the' pseudo-aggregation operator.

This allows for a corresponding value to be returned when a query
includes one 'min' or 'max' aggregate.
2018-03-12 15:18:50 -07:00
Richard Newman 46835885e4 Add help for timer command. 2018-03-06 10:49:41 -08:00
Richard Newman 3e615becd8 Fix printing of fractional millisecond timestamps. (#582) r=emily 2018-03-06 10:49:41 -08:00
Richard Newman 1817ce7c0b Performance and cleanup. r=emily
* Use fixed-size arrays for bootstrap datoms, not vecs.
* Wide-ranging cleanup.

    This commit:
    - Deletes some dead code.
    - Marks some functions only used by tests as cfg(test).
    - Adds pub(crate) to a bunch of functions.
    - Cleans up a few other nits.
2018-03-06 09:03:00 -08:00
Richard Newman a0c70a7cd9 Add caching example to Seattle fixture. 2018-03-06 09:02:11 -08:00
Richard Newman f42ae35b70 Update cache on write. (#566) r=emily
* Use the cache to make constant queries super fast.
* Fix translate tests to match: we no longer generate SQL for many of them!
* Accumulate additions and removals into the cache.
    * Make attribute cache clone-on-write; store it in Metadata.
    * Allow caching of fulltext attributes, interning strings.
2018-03-06 09:01:20 -08:00
Richard Newman bead9752bd Add InProgress.import to load transaction data from a file. r=emily
It's not streaming, but it'll do.
2018-03-06 08:59:30 -08:00
Richard Newman 0ca61017a1 Use LTO for release builds. 2018-03-06 08:19:17 -08:00
Richard Newman 4a0b67ab50 Require Rust 0.1.24 or higher. 2018-03-06 08:19:17 -08:00
Richard Newman 9b23cf3945
Speed up EDN parser (fixes #445) (#581) r=nalexander
Fixes from @kevinmehall.

* Prefer character sets over backtracking in the EDN parser.
* Avoid duplicate effort when parsing floats in the EDN parser.
* Clean up duplicate position tracking code.

This turns out to have little performance impact, but makes the grammar
much cleaner.

* Fix EDN work to pass tests with correct numeric precedence.
2018-03-05 20:33:51 -08:00
Richard Newman 30bf827d16
CLI improvements (#577) r=grisha
* Add a prepared query command to CLI.
* Print nanoseconds in the REPL. This is a good problem to have.
* Better CLI timing.
* Use release for 'cargo cli', debug for 'cargo debugcli'.
* Don't enable debug symbols in release builds.
* Clean up CLI code. Fixed order for help.
* Column-align help output.
2018-03-05 12:52:20 -08:00
Richard Newman d46535a7c2
When an attribute is known-fulltext, don't hit AllDatoms. (#576) r=nalexander 2018-03-05 10:09:53 -08:00
Grisha Kruglov 36d455150d
Disable TLS support, add links to issues for TODOs (#573) r=grisha/self
Landing this now to unblock Android builds of mentat until the cross-compilation of dependencies is figured out.
2018-02-28 15:54:46 -08:00
Richard Newman 5e50d2a9b4 Update to rusqlite 0.13. 2018-02-22 11:41:57 -08:00
Richard Newman 39cf26aa76 Add import command. (#456) r=emily 2018-02-21 11:51:45 -08:00
Richard Newman c53da08a00 Add VSCode command to attach to running CLI. 2018-02-21 11:51:45 -08:00
Richard Newman 23ebc3a5fc Add Datomic's Seattle neighborhood test data. 2018-02-21 11:51:45 -08:00
Richard Newman 20d1b45293 Follow-up: replace println_stderr with eprintln. 2018-02-21 11:51:45 -08:00
Richard Newman 54bd883c65 Follow-up: remove logging and such elsewhere in the codebase. 2018-02-21 11:51:45 -08:00
Richard Newman e33fe71c47 Rework caching and use it inside the query engine. (#553) r=emily
This puts caching in mentat_db, adds a reverse lookup capability for
unique attributes, and populates bidirectional caches with a single
SQL cursor walk.

Differentiate between begin_read and begin_uncached_read.

Note that we still allow toggling within InProgress, because there might be
transient local state that makes starting a new transaction impossible.
2018-02-21 11:51:45 -08:00
Richard Newman df3cdb5db6 Allow two datoms in the same transaction to have the same fulltext string. (#565) r=emily 2018-02-21 11:51:45 -08:00
Richard Newman ae91603bd0 Provide an API for creating truly empty stores (#561) r=grisha
* Part 1: split create_current_version.

* Part 2: add Store::create_empty and Conn::empty.

* Part 3 - Expose 'open_empty' command via CLI
2018-02-16 02:01:00 -08:00
Grisha Kruglov 93e5dff9c8
Revised uploader flow (battle-tested); CLI support for sync (#557) r=rnewman 2018-02-16 01:44:28 -08:00
Emily Toop c84db82575
Merge pull request #560 from mozilla/fluffyemily/entity-builder-update
Add retract_kw
2018-02-15 18:48:39 +00:00
Emily Toop 48ffa20d4c Add retract_kw 2018-02-15 18:47:58 +00:00
Emily Toop 9655c4b85d Add retract to entity builder. (#559) r=rnewman 2018-02-15 09:44:06 -08:00
Richard Newman 01d9b83a9b
Add EntityBuilder.add_kw. (#558) r=emily
* Add EntityBuilder.add_kw.

This allows you to skip your own attribute lookups, at the cost of
potentially doing the work more than once.

Also does value type checking.
2018-02-15 08:22:34 -08:00
Richard Newman 0dcb7df1c7
Timing and colorizing in the CLI. (#552) r=grisha
* Add basic coloring of CLI output.
* Add a timer to the CLI.

Toggle it on or off with 'timer on' and 'timer off'.
Output is colorized.

* Add VSCode configuration files.

These allow you to build and run the CLI, build Mentat, or run all tests.
2018-02-15 07:36:18 -08:00
Emily Toop dec86bb2c5
publically expose KnownEntid (#554) 2018-02-14 19:06:50 +00:00
Richard Newman 23e11fabe6
Add a var! macro. (#548)(#550) r=emily 2018-02-14 09:32:37 -08:00
Richard Newman 2ac7a1b1de Add a feature flag to control the use of rusqlite's bundled SQLite. r=emily
You can use this in conjunction with setting SQLITE3_LIB_DIR to control which SQLite is used.

See https://github.com/jgallagher/rusqlite for more.

Also add recent contributors to the authors array.
2018-02-13 08:25:58 -08:00
Grisha Kruglov 84f29676e8
"Unchanged server" uploader flow (#543) r=rnewman
* Remove unused struct from tx_processor

* Derive serialize & deserialize for TypedValue

* First pass of uploader flow + feedback
2018-02-09 09:55:19 -08:00
Richard Newman d11810dca7 Fix warning in tx.rs. 2018-02-09 08:51:13 -08:00
Kit Cambridge a6341f6fd6 Implement q_prepare with pre-bound variables. r=rnewman 2018-02-07 21:48:05 -08:00
Emily Toop 715d434945
Create generalized in-memory cache for attributes (#525)
* Nit: Alphabetical ordering of imports

* Create Cache and provide functions for calling it

* Get tests working. Move to using NamespacedKeyword over KnownEntid in function signature

* Add is_cached check to caching tests

* Move lazy and add/remove boolean flags to enums

* Move function definitions into generic trait and implement trait for AttributeCache

* Remove lazy cache and generalize cache

* Update tests

* Eager cache becomes simple key value store. AttributeMap handles attribute storing specifics

* Update tests to test presence of correct values in cache

* Move EagerCache, AttributeValueProvider and ValueProvider into mentat_db

* Add test for get_for_entid

* Add test for lookup attribute

* Make caches cloneable. Add value_for alongside values_for

* Use cache in attribute lookups

* Split test for values and value and add cardinality

* address review feedback r=rnewman
2018-02-07 10:56:12 -08:00
Grisha Kruglov d848d954cf Issue 508 - Iterating transcation processor r=rnewman
Review comments
2018-02-06 12:24:12 -08:00
Richard Newman e9ed103723 Revert "Make debug structs and functions non-public. r=grisha"
This reverts commit e817f67470.
2018-02-06 11:06:06 -08:00