Commit graph

46 commits

Author SHA1 Message Date
Greg Burd
60c65033b2 Specify dependency versions without patch component unless necessary. 2020-01-23 11:15:49 -05:00
Greg Burd
b2f92b8461 Update to 2018 edition of Rust (1.42). Fix and format code. Update dependencies. Fix tests. 2020-01-16 10:58:21 -05:00
Conrad Dean
3d965fdf6e try fixing build by upgrading rusqlite to 0.19 2019-07-17 10:59:38 -04:00
Grisha Kruglov
b22b29679b
Basic sync support (#563) r=nalexander
* Pre: remove remnants of 'open_empty'

* Pre: Cleanup 'datoms' table after a timeline move

Since timeline move operations use a transactor, they generate a
"phantom" 'tx' and a 'txInstant' assertion. It is "phantom" in a sense
that it was never present in the 'transactions' table, and is entirely
synthetic as far as our database is concerned.
It's an implementational artifact, and we were not cleaning it up.

It becomes a problem when we start inserting transactions after a move.
Once the transactor clashes with the phantom 'tx', it will retract the
phantom 'txInstant' value, leaving the transactions log in an incorrect state.

This patch adds a test for this scenario and elects the easy way out: simply
remove the offending 'txInstant' datom.

* Part 1: Sync without support for side-effects

A "side-effect" is defined here as a mutation of a remote state as part
of the sync.

If, during a sync we determine that a remote state needs to be changed, bail out.

This generally supports different variations of "baton-passing" syncing, where clients
will succeed syncing if each change is non-conflicting.

* Part 2: Support basic "side-effects" syncing

This patch introduces a concept of a follow-up sync. If a sync generated
a "merge transaction" (a regular transaction that contains assertions
necessary for local and remote transaction logs to converge), then
this transaction needs to be uploaded in a follow-up sync.

Generated SyncReport indicates if a follow-up sync is required.

Follow-up sync itself is just a regular sync. If remote state did not change,
it will result in a simple RemoteFastForward. Otherwise, we'll continue
merging and requesting a follow-up.

Schema alterations are explicitly not supported.

As local transactions are rebased on top of remote, following changes happen:
- entids are changed into tempids, letting transactor upsert :db/unique values
- entids for retractions are changed into lookup-refs if we're confident they'll succeed
-- otherwise, retractions are dropped on the floor

* Post: use a macro for more readable tests

* Tolstoy README
2018-09-07 19:18:20 -07:00
Grisha Kruglov
66cc4e14ad Post: use dirs crate, avoiding compile warning about home_dir 2018-08-20 18:23:46 -07:00
Nick Alexander
0b84a0802d Pre: Remove open_empty.
This was a work-around for Tolstoy, which couldn't gracefully handle
syncing a store with a bootstrap transaction.  Tolstoy now handles
that single transaction, so this is no longer necessary.
2018-08-20 18:23:46 -07:00
Grisha Kruglov
1e488d720b Post: Use a single implementation of bail macro 2018-08-09 13:16:05 -07:00
Grisha Kruglov
07beb68c7a Pre: Remove query/ crate 2018-08-09 13:16:05 -07:00
Grisha Kruglov
d0214fad7d Pre: Move core/types.rs into core_traits 2018-08-09 13:16:05 -07:00
Grisha Kruglov
bff24c60b7
Add a top-level "syncable" feature. (#782) r=ncalexan
* Add a top-level "syncable" feature.

Tested with:

cargo test --all
cargo test --all --no-default-features
cargo build --manifest-path tools/cli/Cargo.toml --no-default-features
cargo run --manifest-path tools/cli/Cargo.toml --no-default-features debugcli

Co-authored-by: Nick Alexander <nalexander@mozilla.com>

* Add 'syncable' feature to 'db' crate to conditionally derive serialization for Partition*

This is leading up to syncing with partition support.
2018-07-11 16:26:06 -07:00
Nick Alexander
d82c7f8ef2 Cull unused mentat_parser_utils crate.
With the transition toward parsing with `rust-peg` and away from
`combine`, we're not using some of the many helpers we built to
support our unusual `combine` usage.  They can just go!
2018-06-30 16:21:50 -07:00
Nick Alexander
d31ec28aa8 Patch it all together: use MentatError at top-level.
I elected to keep Tolstoy using `failure::Error`, because Tolstoy
looks rather more like a high-level application (and will continue to
do so for a while) than a production-ready mid- or low-level API.
2018-06-27 15:05:43 -07:00
Nick Alexander
4ea9c78c50 [cli] Part 3: {load,save}_history as appropriate.
It's possible that we should be saving more aggressively -- perhaps
after each entered command -- but we can add that later.
2018-06-22 15:39:29 -07:00
Nick Alexander
c41d728d1d [cli] Part 2: Don't use exit() to terminate the CLI.
It's not possible to do meaningful clean-up (such as saving history)
if we use exit() to quit.  Instead, each handled command returns a
boolean requesting exit.  I elected not to allow ".exit" when
processing commands from the command line; it might be useful to
handle accept that.  In general, though, REPLs that accept "-c
'commands'" on the command line exit after processing those commands,
so I'd rather think more deeply about that model than build in ".exit"
with our existing system.
2018-06-22 15:36:09 -07:00
Nick Alexander
c19337c8bf [cli] Part 1: Bump linefeed; use linefeed::Interface; add "--no-tty" argument.
I don't really understand why we were using `linefeed::Reader`
directly, but reading is not the full set of linefeed features we want
to access.  I think the `linefeed::Interface` should be owned by the
`Repl`, not the `InputReader`, but it's a little awkward to share
access with that configuration, so I'm not going to lift the ownership
until I have a reason to.

I think the "--no-tty" argument might be useful for running inside
Emacs.  Along the way, I made read_stdin() strip the trailing newline,
which agrees with InputReader::read_line().
2018-06-22 15:36:09 -07:00
Nick Alexander
8e2d795778
[cli] Handle line comments in EDN input. (#759) (#761) r=grisha
What was happening is that ["[;", "]"] would get glued to "[; ]",
which of course can never complete.

It would be good to add tests of this, but the existing multi-line
`InputReader` makes that challenging and I don't want to invest the
time to improve it: I expect it to be overhauled as part of a
transition away from parsing with `combine` and toward parsing with
`rust-peg`.
2018-06-22 14:34:16 -07:00
Grisha Kruglov
0adfa6aae6 Convert tools/cli to failure. 2018-06-20 14:42:30 -07:00
Grisha Kruglov
800f404a23 Convert ffi/ to failure.
This is neat, because currently at the FFI boundary we're primarily concerned
with verbalizing our errors. It doesn't matter what 'error' that's wrapped by
Result is then, as long as it can be displayed.

Once we're past the prototyping stage, it might be a good idea to formalize this.
2018-06-20 14:42:21 -07:00
Mario Wenzel
fb7d2357de Don't try to match "key" when not using sqlcipher. (#752, #753) r=nalexander
This causes a runtime error, since `opt_str("key")` isn't recognized.
2018-06-20 13:26:36 -07:00
Nick Alexander
f041dfe509 Bustage fix: Build against Rust 1.25. 2018-06-13 15:28:44 -07:00
Thom
6a1a265894
Add support for using sqlcipher (#737). Fixes #118 2018-06-13 08:49:40 -07:00
Richard Newman
3dc68bcd38 Combine NamespacedKeyword and Keyword. (#689) r=nalexander
* Make properties on NamespacedKeyword/NamespacedSymbol private

* Use only a single String for NamespacedKeyword/NamespacedSymbol

* Review comments.

* Remove unsafe code in namespaced_name.

Benchmarking shows approximately zero change.

* Allow the types of ns and name to differ when constructing a NamespacedName.

* Make symbol namespaces optional.

* Normalize names of keyword/symbol constructors.

This will make the subsequent refactor much less painful.

* Use expect not unwrap.

* Merge Keyword and NamespacedKeyword.
2018-05-11 09:52:17 -07:00
Nick Alexander
cbffe5e545 Use rust-peg for tx parsing.
There are few reasons to do this:

- it's difficult to add symbol interning to combine-based parsers like
  tx-parser -- literally every type changes to reflect the interner,
  and that means every convenience macro we've built needs to chagne.
  It's trivial to add interning to rust-peg-based parsers.

- combine has rolled forward to 3.2, and I spent a similar amount of
  time investigating how to upgrade tx-parser (to take advantage of
  the new parser! macros in combine that I think are necessary for
  adapting to changing types) as I did just converting to rust-peg.

- it's easy to improve the error messages in rust-peg, where-as I have
  tried twice to improve the nested error messages in combine and am
  stumped.

- it's roughly 4x faster to parse strings directly as opposed to
  edn::ValueAndSpan, and it'll be even better when we intern directly.
2018-05-10 10:24:05 -07:00
Richard Newman
e21156a754
Implement simple pull expressions (#638) r=nalexander
* Refactor AttributeCache populator code for use from pull.

* Pre: add to_value_rc to Cloned.

* Pre: add From<StructuredMap> for Binding.

* Pre: clarify Store::open_empty.

* Pre: StructuredMap cleanup.

* Pre: clean up a doc test.

* Split projector crate. Pass schema to projector.

* CLI support for printing bindings.

* Add and use ConjoiningClauses::derive_types_from_find_spec.

* Define pull types.

* Implement pull on top of the attribute cache layer.

* Add pull support to the projector.

* Parse pull expressions.

* Add simple pull support to connection objects.

* Tests for pull.

* Compile with Rust 1.25.

The only choice involved in this commit is that of replacing the
anonymous lifetime '_ with a named lifetime for the cache; since we're
accepting a Known, which includes the cache in question, I think it's
clear that we expect the function to apply to any given cache
lifetime.

* Review comments.

* Bail on unnamed attribute.

* Make assert_parse_failure_contains safe to use.

* Rework query parser to report better errors for pull.

* Test for mixed wildcard and simple attribute.
2018-05-04 12:56:00 -07:00
Nick Alexander
1b66818ac9 Post: Fix CLI bustage. 2018-05-01 16:10:06 -07:00
Nick Alexander
7960b4ccd2 Pre: Get ready to use log in mentat_db.
We don't yet have a logging system for production use, but I'd like to
start experimenting with log, which seems to be (close to) a Rust
standard.  We're already using it in mentat_cli.
2018-05-01 13:46:03 -07:00
Richard Newman
a2e13f624c
Add 'Binding', a structured value type to return from queries. (#657) r=nalexander
Bump to 0.7: breaking change.
2018-04-24 15:08:38 -07:00
Richard Newman
46835885e4 Add help for timer command. 2018-03-06 10:49:41 -08:00
Richard Newman
3e615becd8 Fix printing of fractional millisecond timestamps. (#582) r=emily 2018-03-06 10:49:41 -08:00
Richard Newman
f42ae35b70 Update cache on write. (#566) r=emily
* Use the cache to make constant queries super fast.
* Fix translate tests to match: we no longer generate SQL for many of them!
* Accumulate additions and removals into the cache.
    * Make attribute cache clone-on-write; store it in Metadata.
    * Allow caching of fulltext attributes, interning strings.
2018-03-06 09:01:20 -08:00
Richard Newman
30bf827d16
CLI improvements (#577) r=grisha
* Add a prepared query command to CLI.
* Print nanoseconds in the REPL. This is a good problem to have.
* Better CLI timing.
* Use release for 'cargo cli', debug for 'cargo debugcli'.
* Don't enable debug symbols in release builds.
* Clean up CLI code. Fixed order for help.
* Column-align help output.
2018-03-05 12:52:20 -08:00
Richard Newman
5e50d2a9b4 Update to rusqlite 0.13. 2018-02-22 11:41:57 -08:00
Richard Newman
39cf26aa76 Add import command. (#456) r=emily 2018-02-21 11:51:45 -08:00
Richard Newman
e33fe71c47 Rework caching and use it inside the query engine. (#553) r=emily
This puts caching in mentat_db, adds a reverse lookup capability for
unique attributes, and populates bidirectional caches with a single
SQL cursor walk.

Differentiate between begin_read and begin_uncached_read.

Note that we still allow toggling within InProgress, because there might be
transient local state that makes starting a new transaction impossible.
2018-02-21 11:51:45 -08:00
Richard Newman
ae91603bd0 Provide an API for creating truly empty stores (#561) r=grisha
* Part 1: split create_current_version.

* Part 2: add Store::create_empty and Conn::empty.

* Part 3 - Expose 'open_empty' command via CLI
2018-02-16 02:01:00 -08:00
Grisha Kruglov
93e5dff9c8
Revised uploader flow (battle-tested); CLI support for sync (#557) r=rnewman 2018-02-16 01:44:28 -08:00
Richard Newman
0dcb7df1c7
Timing and colorizing in the CLI. (#552) r=grisha
* Add basic coloring of CLI output.
* Add a timer to the CLI.

Toggle it on or off with 'timer on' and 'timer off'.
Output is colorized.

* Add VSCode configuration files.

These allow you to build and run the CLI, build Mentat, or run all tests.
2018-02-15 07:36:18 -08:00
Richard Newman
2ac7a1b1de Add a feature flag to control the use of rusqlite's bundled SQLite. r=emily
You can use this in conjunction with setting SQLITE3_LIB_DIR to control which SQLite is used.

See https://github.com/jgallagher/rusqlite for more.

Also add recent contributors to the authors array.
2018-02-13 08:25:58 -08:00
Richard Newman
66e6fef75e Define Store, use TabWriter in the CLI for aligning columnar output. (#540) r=emily
* Define Store, which is a simple container for a SQLite connection and a Conn.
  This is a breaking change.
* Return the FindSpec as part of QueryOutput, not just results.
* Switch to using stderr in appropriate places in CLI.
* Print columns in CLI output.
2018-02-01 09:29:07 -08:00
Thom
1fcc3d7e1b
Add support for Ctrl-C aborting the current command on the CLI (#528) r=rnewman 2018-01-21 09:45:40 -05:00
Thom
9740cafdbd Automatically remove trailing whitespace from text files. (#527) r=rnewman
This was done using the following shell script:

```
find . -type f -not -path "*target*" \
       '(' -name '*.rs' -o -name '*.md' -o -name '*.toml' ')' -print0 | \
    xargs -0 sed -i '' -E 's/[[:space:]]*$//'
```

Which is admittedly imperfect, but manages to hit everything that was a problem in this repo.
2018-01-19 21:21:04 -06:00
Thom
023fd9b70b
Add a command to the CLI that displays the SQL and query plan for a query. Fixes #428. (#523) r=rnewman 2018-01-19 19:44:39 -05:00
Thom
579829d091 CLI quality-of-life fixes. (#521) r=rnewman
* CLI: Update linefeed library to latest version
* CLI: Don't store incomplete commands in history
* CLI: add curly braces to word separators
* Review comment: clean up CLI add_history.

Signed-off-by: Thom Chiovoloni <tchiovoloni@mozilla.com>
2018-01-17 22:33:22 -06:00
Emily Toop
2fc4cb5a2d Clear the CLI buffer when an incorrect command is added. (#500) r=rnewman
* Fix issue whereby when an incorrect command was entered, the buffer wasn't cleared and the next command was appended on the end of the incorrect one.

* Cleanup.
2017-12-05 14:10:10 -08:00
Emily Toop
55588209c2
CLI (#493)
* Create mentat command line.
* Create tools directory containing new crate for mentat_cli.
* Add simple cli with mentat prompt.

* Remove rustc-serialize dependency

* Open DB inside CLI (#452) (#463)

* Open named database OR default to in memory database if no name provided

Rearrange workspace to allow import of mentat crate in cli crate

Create store object inside repl when started for connecting to mentat

Use provided DB name to open connection in store

Accept DB name as command line arg.

Open on CLI start

Implement '.open' command to open desired DB from inside CLI

* Implement Close command to close current DB.
* Closes existing open db and opens new in memory db

* Review comment: Use `combine` to parse arguments.

Move over to using Result rather than enums with err

* Accept and parse EDN Query and Transact commands (#453) (#465)

* Parse query and transact commands

* Implement is_complete for transactions and queries

* Improve query parser. Am still not happy with it though.

There must be some way that I can retain the eof() after the `then` that means I don't have to move the skip on spaces and eof

Make in process command storing clearer.

Add comments around in process commands.
Add alternative commands for transact/t and query/q

* Address review comments r=nalexander.

* Bump rust version number.
* Use `bail` when throwing errors.
* Improve edn parser.
* Remove references to unused `more` flag.
* Improve naming of query and transact commands.

* Send queries and transactions to mentat and output the results (#466)

* Send queries and transactions to mentat and output the results

move outputting query and transaction results out of store and into repl

* Add query and transact commands to help

* Execute queries and transacts passed in at startup

* Address review comments =nalexander.

* Bump rust version number.
* Use `bail` when throwing errors.
* Improve edn parser.
* Remove references to unused `more` flag.
* Improve naming of query and transact commands.

* Execute command line args in order

* Addressing rebase issues

* Exit CLI (#457) (#484) r-rnewman

* Implement exit command for cli tool

* Address review comments r=rnewman

* Include exit commands in help

* Show schema of current DB (#487)

* Fixing rebase issues

* addressing nit

* Match updated dependencies on CLI crate and remove unused import
2017-11-21 16:56:16 +00:00
Richard Newman
c95ec13ffe Begin moving web server to a separate crate. (#448) r=bgrins
This doesn't yet introduce a working Cargo.toml for 'mentatweb', but it
does allow RLS to build correctly without errors, and it reduces the
core library's dependency space, which is more important in the short
term.
2017-05-10 02:25:59 -07:00