There's an unfortunate conflation here between implementing the query
parser in `rust-peg` and moving some validation that now happens at
parse time to happen later. The result is that we introduce
`ParsedFindQuery` as a less-processed `FindQuery`, and that we only
use string errors (which is all `rust-peg` supports) instead of the
structured errors in query-parser's errors module. The next commit
will address this, on the road to removing the `query-parser` module
entirely.
* Make properties on NamespacedKeyword/NamespacedSymbol private
* Use only a single String for NamespacedKeyword/NamespacedSymbol
* Review comments.
* Remove unsafe code in namespaced_name.
Benchmarking shows approximately zero change.
* Allow the types of ns and name to differ when constructing a NamespacedName.
* Make symbol namespaces optional.
* Normalize names of keyword/symbol constructors.
This will make the subsequent refactor much less painful.
* Use expect not unwrap.
* Merge Keyword and NamespacedKeyword.
* Refactor AttributeCache populator code for use from pull.
* Pre: add to_value_rc to Cloned.
* Pre: add From<StructuredMap> for Binding.
* Pre: clarify Store::open_empty.
* Pre: StructuredMap cleanup.
* Pre: clean up a doc test.
* Split projector crate. Pass schema to projector.
* CLI support for printing bindings.
* Add and use ConjoiningClauses::derive_types_from_find_spec.
* Define pull types.
* Implement pull on top of the attribute cache layer.
* Add pull support to the projector.
* Parse pull expressions.
* Add simple pull support to connection objects.
* Tests for pull.
* Compile with Rust 1.25.
The only choice involved in this commit is that of replacing the
anonymous lifetime '_ with a named lifetime for the cache; since we're
accepting a Known, which includes the cache in question, I think it's
clear that we expect the function to apply to any given cache
lifetime.
* Review comments.
* Bail on unnamed attribute.
* Make assert_parse_failure_contains safe to use.
* Rework query parser to report better errors for pull.
* Test for mixed wildcard and simple attribute.
* Pre: eliminate some occurrences of Rc, largely through the magic of Into.
* Pre: introduce FromRc to convert between refcounted types.
* Introduce ValueRc as an abstraction over Rc/Arc choice.
* Move Cloned to core.
* Move CString-creation methods to TypedValue.
* Finish transition.
* Pre: use debugcli in VSCode.
* Pre: wrap subqueries in parentheses in output SQL.
* Pre: add ExistingColumn.
This lets us make reference to columns by name, rather than only
pointing to qualified aliases.
* Pre: add Into for &str to TypedValue.
* Pre: add Store.transact.
* Pre: cleanup.
* Parse and algebrize simple aggregates. (#312)
* Follow-up: print aggregate columns more neatly in the CLI.
* Useful ValueTypeSet helpers.
* Allow for entity inequalities.
* Add 'differ', which is a ref-specialized not-equals.
* Add 'unpermute', a function for getting unique, distinct pairs from bindings.
* Review comments.
* Add 'the' pseudo-aggregation operator.
This allows for a corresponding value to be returned when a query
includes one 'min' or 'max' aggregate.
This was done using the following shell script:
```
find . -type f -not -path "*target*" \
'(' -name '*.rs' -o -name '*.md' -o -name '*.toml' ')' -print0 | \
xargs -0 sed -i '' -E 's/[[:space:]]*$//'
```
Which is admittedly imperfect, but manages to hit everything that was a problem in this repo.
* Pre: make FindQuery, FindSpec, and Element non-Clone.
* Pre: make query translator return a Result.
* Pre: make projection return a Result.
* Pre: refactor query parser in preparation for parsing aggregates.
* Pre: rename PredicateFn -> QueryFunction.
* Pre: expose more about bound variables from CC.
* Pre: move ValueTypeSet to core.
* Update some dependencies.
* Update rusqlite to 0.12.
* Update error-chain to a forked version that implements Sync.
* Fix some compiler warnings.
* Remove unused imports in tests.
* Parse errors no longer naturally print with the expected symbol.
Datomic accepts mostly-arbitrary EDN, and it is actually used: for
example, the following are all valid, and all mean different things:
* `(ground 1 ?x)`
* `(ground [1 2 3] [?x ?y ?z])`
* `(ground [[1 2 3] [4 5 6]] [[?x ?y ?z]])`
We could probably introduce new syntax that expresses these patterns
while avoiding collection arguments, but I don't see one right now.
I've elected to support only vectors for simplicity; I'm hoping to
avoid parsing edn::Value in the query-algebrizer.
This is a big commit, but it breaks into two conceptual pieces. The
first is to "parse without copying". We replace a stream of an owned
collection of edn::ValueAndSpan and instead have a stream of a
borrowed collection of &edn::ValueAndSpan references. (Generally,
this is represented as an iterator over a slice, but it can be over
other things too.) Cloning such iterators is constant time, which
improves on cloning an owned collection of edn::ValueAndSpan, which is
linear time in the length of the collection and additional time
depending on the complexity of the EDN values.
The second conceptual piece is to parse keyword maps using a special
parser and a macro to build the parser implementations. Before, we
created a new edn::ValueAndSpan::Map to represent a keyword map in
vector form; since we're working with &edn::ValueAndSpan references
now, we can't create an &edn::ValueAndSpan reference with an
appropriate lifetime. Therefore we generalize the concept of
iteration slightly and turn keyword maps in map form into linear
iterators by flattening the value maps. This is a potentially
obscuring transformation, so we have to take care to protect against
some failure cases. (See the comments and the tests in the code.)
After these changes, parsing using `combine` is linear time (and
reasonably fast).
* Pre: unused import in translate.rs.
* Part 2: take a dependency on rusqlite for query arguments.
* Part 1: flatten V2 schema into V1. Add UUID and URI.
Bump expected ident and bootstrap datom count in tests.
* Part 5: parse edn::Value::Uuid.
* Part 3: extend ValueType and TypedValue to include Uuid.
* Part 4: add Uuid to query arguments.
* Part 6: extend db to support Uuid.
* Part 8: add a tx-parser test for #f NaN and #uuid.
* Part 7: parse and algebrize UUIDs in queries.
* Part 1: parse #inst in EDN and throughout query engine.
* Part 3: handle instants in db.
* Part 2: instants never matches integers in queries.
* Part 4: use DateTime for tx_instants.
* Add a test for adding and querying UUIDs and instants.
* Review comments.
* Part 1 - Parse `not` and `not-join`
* Part 2 - Validate `not` and `not-join` pre-algebrization
* Address review comments rnewman.
* Remove `WhereNotClause` and populate `NotJoin` with `WhereClause`.
* Fix validation for `not` and `not-join`, removing tests that were invalid.
* Address rustification comments.
* Rebase against `rust` branch.
* Part 3 - Add required types for NotJoin.
* Implement `PartialEq` for
`ConjoiningClauses` so `ComputedTable` can be included inside `ColumnConstraint::NotExists`
* Part 4 - Implement `apply_not_join`
* Part 5 - Call `apply_not_join` from inside `apply_clause`
* Part 6 - Translate `not-join` into `NOT EXISTS` SQL
* Address review comments.
* Rename `projected` to `unified` to better describe the fact that we are not projecting any variables.
* Check for presence of each unified var in either `column_bindings` or `input_bindings` and bail if not there.
* Copy over `input_bindings` for each var in `unified`.
* Only copy over the first `column_binding` for each variable in `unified` rather than the whole list.
* Update tests.
* Address review comments.
* Make output from Debug for NotExists more useful
* Clear up misunderstanding. Any single failing clause in the not will cause the entire not to be considered empty
* Address review comments.
* Remove Limit requirement from cc_to_exists.
* Use Entry.or_insert instead of matching on the entry to add to column_bindings.
* Move addition of value_bindings to before apply_clauses on template.
* Tidy up tests with some variable reuse.
* Addressed nits,
* Address review comments.
* Move addition of column_bindings to above apply_clause.
* Update tests.
* Add test to ensure that unbound vars fail
* Improve test for unbound variable to check for correct variable and error
* address nits
* Pre: put query parts in alphabetical order.
* Pre: rename 'input' to 'query' in translate tests.
* Part 1: parse :limit.
* Part 2: validate and escape variable parameters in SQL.
* Part 3: algebrize and translate limits.
We also at this point switch from using `Vec<Variable>` to
`BTreeSet<Variable>`. This allows us to guarantee no duplicates later;
we'll reject duplicates at parse time.
This adds an `:order` keyword to `:find`.
If present, the results of the query will be an ordered set, rather than
an unordered set; rows will appear in an ordered defined by each
`:order` entry.
Each can be one of three things:
- A var, `?x`, meaning "order by ?x ascending".
- A pair, `(asc ?x)`, meaning "order by ?x ascending".
- A pair, `(desc ?x)`, meaning "order by ?x descending".
Values will be ordered in this sequence for asc, and in reverse for desc:
1. Entity IDs, in ascending numerical order.
2. Booleans, false then true.
3. Timestamps, in ascending numerical order.
4. Longs and doubles, intermixed, in ascending numerical order.
5. Strings, in ascending lexicographic order.
6. Keywords, in ascending lexicographic order, considering the entire
ns/name pair as a single string separated by '/'.
Subcommits:
Pre: make bound_value public.
Pre: generalize ErrorKind::UnboundVariable for use in order.
Part 1: parse (direction, var) pairs.
Part 2: parse :order clause into FindQuery.
Part 3: include order variables in algebrized query.
We add order variables to :with, so we can reuse its type tag projection
logic, and so that we can phrase ordering in terms of variables rather
than datoms columns.
Part 4: produce SQL for order clauses.
* Pre: Expose more in edn.
* Pre: Make it easier to work with ValueAndSpan.
with_spans() is a temporary hack, needed only because I don't care to
parse the bootstrap assertions from text right now.
* Part 1a: Add `value_and_span` for parsing nested `edn::ValueAndSpan` instances.
I wasn't able to abstract over `edn::Value` and `edn::ValueAndSpan`;
there are multiple obstacles. I chose to roll with
`edn::ValueAndSpan` since it exposes the additional span information
that we will want to form good error messages in the future.
* Part 1b: Add keyword_map() parsing an `edn::Value::Vector` into an `edn::Value::map`.
* Part 1c: Add `Log`/`.log(...)` for logging parser progress.
This is a terrible hack, but it sure helps to debug complicated nested
parsers. I don't even know what a principled approach would look
like; since our parser combinators are so frequently expressed in
code, it's hard to imagine a data-driven interpreter that can help
debug things.
* Part 2: Use `value_and_span` apparatus in tx-parser/.
I break an abstraction boundary by returning a value column
`edn::ValueAndSpan` rather than just an `edn::Value`. That is, the
transaction processor shouldn't care where the `edn::Value` it is
processing arose -- even we care to track that information we should
bake it into the `Entity` type. We do this because we need to
dynamically parse the value column to support nested maps, and parsing
requires a full `edn::ValueAndSpan`. Alternately, we could cheat and
fake the spans when parsing nested maps, but that's potentially
expensive.
* Part 3: Use `value_and_span` apparatus in query-parser/.
* Part 4: Use `value_and_span` apparatus in root crate.
* Review comment: Make Span and SpanPosition Copy.
* Review comment: nits.
* Review comment: Make `or` be `or_exactly`.
I baked the eof checking directly into the parser, rather than using
the skip and eof parsers. I also took the time to restore some tests
that were mistakenly commented out.
* Review comment: Extract and use def_matches_* macros.
* Review comment: .map() as late as possible.
Part 1, core: use Rc for String and Keyword.
Part 2, query: use Rc for Variable.
Part 3, sql: use Rc for args in SQLiteQueryBuilder.
Part 4, query-algebrizer: use Rc.
Part 5, db: use Rc.
Part 6, query-parser: use Rc.
Part 7, query-projector: use Rc.
Part 8, query-translator: use Rc.
Part 9, top level: use Rc.
Part 10: intern Ident and IdentOrKeyword.
* Pre: Drop unneeded tx0 from search results.
* Pre: Don't require a schema in some of the DB code.
The idea is to separate the transaction applying code, which is
schema-aware, from the concrete storage code, which is just concerned
with getting bits onto disk.
* Pre: Only reference Schema, not DB, in debug module.
This is part of a larger separation of the volatile PartitionMap,
which is modified every transaction, from the stable Schema, which is
infrequently modified.
* Pre: Fix indentation.
* Extract part of DB to new SchemaTypeChecking trait.
* Extract part of DB to new PartitionMapping trait.
* Pre: Don't expect :db.part/tx partition to advance when tx fails.
This fails right now, because we allocate tx IDs even when we shouldn't.
* Sketch a db interface without DB.
* Add ValueParseError; use error-chain in tx-parser.
This can be simplified when
https://github.com/Marwes/combine/issues/86 makes it to a published
release, but this unblocks us for now. This converts the `combine`
error type `ParseError<&'a [edn::Value]>` to a type with owned
`Vec<edn::Value>` collections, re-using `edn::Value::Vector` for
making them `Display`.
* Pre: Accept Borrow<Schema> instead of just &Schema in debug module.
This makes it easy to use Rc<Schema> or Arc<Schema> without inserting
&* sigils throughout the code.
* Use error-chain in query-parser.
There are a few things to point out here:
- the fine grained error types have been flattened into one crate-wide
error type; it's pretty easy to regain the granularity as needed.
- edn::ParseError is automatically lifted to
mentat_query_parser::errors::Error;
- we use mentat_parser_utils::ValueParser to maintain parsing error
information from `combine`.
* Patch up top-level.
* Review comment: Only `borrow()` once.
* Leave a pointer to issue 288.
* Re-export mentat_db::types::DB from mentat_db.
* Parse EDN strings in the query parser.
* Export 'public' API from mentat_query_parser's top level.
* Stub out mentat::q_once.
* Make Variable::from_symbol public.
* Implement basic parsing of queries.
* Use pinned dependencies the hard way to fix Travis.
* Bump ordered-float dependency to 0.4.0.
* Error coercions to use ?, and finishing the find interface.
* Test the mentat_query directory on Travis.
* Export common types from edn.
This allows you to write
use edn::{PlainSymbol,Keyword};
instead of
use edn:🔣:{PlainSymbol,Keyword};
* Add an edn::Value::is_keyword predicate.
* Clean up query, preparing for query-parser.
* Make EDN keywords and symbols take Into<String> arguments.
* Implement parsing of simple :find lists.
* Rustfmt query-parser. Split find and query.
* Review comment: values_to_variables now returns a NotAVariableError on failure.
* Review comment: rename gimme to to_parsed_value.
* Review comment: add comments.
Starting to work out the project layout for sub-crates. The crate inside query-parser/ is "datomish-query-parser" and the core code in src/ depends on it.
