greg/mentat
1
0
Fork 0
Code Issues 294 Pull requests Projects Releases 9 Packages Wiki Activity
mentat/tx-parser/tests/parser.rs

89 lines
3.7 KiB
Rust
Raw Normal View History

Add tx{-parser} crates; start parsing transactions. (#164) r=rnewman This depends on edn and uses the combine parser combinator library.
2017-01-13 00:08:29 +00:00
// Copyright 2016 Mozilla
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may not use
// this file except in compliance with the License. You may obtain a copy of the
// License at http://www.apache.org/licenses/LICENSE-2.0
// Unless required by applicable law or agreed to in writing, software distributed
// under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
// CONDITIONS OF ANY KIND, either express or implied. See the License for the
// specific language governing permissions and limitations under the License.
extern crate edn;
extern crate combine;
extern crate mentat_tx;
extern crate mentat_tx_parser;
use edn::parse;
use edn::symbols::NamespacedKeyword;
Lookup refs, nested vector values, map notation. Fixes #180, fixes #183, fixes #284. (#382) r=rnewman * Pre: Fix error in parser macros. * Pre: Make test unwrapping more verbose. * Pre: Make lookup refs be (lookup-ref a v) in the entity position. This has the advantage of being explicit in all situations and unambiguous at parse-time. This choice agrees with the Clojure implementation but not with Datomic. Datomic treats [a v] as a lookup ref, is ambiguous at parse-time, and is disambiguated in ways I do not understand at transaction time. We mooted making lookup refs [[a v]] and outlawing nested value vectors in transactions, but after implementing that approach I decided it was better to handle lookup refs at parse time and therefore outlawing nested value vectors is not necessary. * Handle lookup refs in the entity and value columns. Fixes #183. * Pre 0a: Use a stack instead of into_iter. * Pre 0b: Dedent. * Pre 0c: Handle `e` after `v`. This allows to use the original `e` while handling `v`. * Explode value lists for :db.cardinality/many attributes. Fixes #284. * Parse and accept map notation. Fixes #180. * Pre: Modernize add() and retract() into one add_or_retract(). * Pre: Add is_collection and is_atom to edn::Value. * Pre: Differentiate atoms from lookup-refs in value position. Initially, I expected to accept arbitrary edn::Value instances in the value position, and to differentiate in the transactor. However, the implementation quickly became a two-stage parser, since we always wanted to parse the resulting value position into some other known thing using the tx-parser. To save calls into the parser and to allow the parser to move forward with a smaller API surface, I push as much of this parsing as possible into the initial parse. * Pre: Modernize entities(). * Pre: Quote edn::Value::Text in Display. * Review comment: Add and use edn::Value::into_atom. * Review comment: Use skip(eof()) throughout. * Review comment: VecDeque instead of Vec. * Review comment: Part 0: Rename TempId to TempIdHandle. * Review comment: Part 1: Differentiate internal and external tempids. This breaks an abstraction boundary by pushing the Internal/External split up to the Entity level in tx/ and tx-parser/. This just makes it easier to explode Entity map notation instances into Entity instances, taking an existing External tempid :db/id or generating a new Internal tempid as appropriate. To do this without breaking the abstraction boundary would require adding flexibility to the transaction processor: we'd need to be able to turn Entity instances into some internal enum and handle the two cases independently. It wouldn't be too hard, but this reduces the combinatorial type explosion.
2017-03-27 23:30:04 +00:00
use mentat_tx::entities::{
AtomOrLookupRefOrVectorOrMapNotation,
Entid,
EntidOrLookupRefOrTempId,
Entity,
OpType,
TempId,
};
Add tx{-parser} crates; start parsing transactions. (#164) r=rnewman This depends on edn and uses the combine parser combinator library.
2017-01-13 00:08:29 +00:00
use mentat_tx_parser::Tx;
UUIDs and instants. Fixes #44, #45, #426, #427. (#438) r=nalexander * Pre: unused import in translate.rs. * Part 2: take a dependency on rusqlite for query arguments. * Part 1: flatten V2 schema into V1. Add UUID and URI. Bump expected ident and bootstrap datom count in tests. * Part 5: parse edn::Value::Uuid. * Part 3: extend ValueType and TypedValue to include Uuid. * Part 4: add Uuid to query arguments. * Part 6: extend db to support Uuid. * Part 8: add a tx-parser test for #f NaN and #uuid. * Part 7: parse and algebrize UUIDs in queries. * Part 1: parse #inst in EDN and throughout query engine. * Part 3: handle instants in db. * Part 2: instants never matches integers in queries. * Part 4: use DateTime for tx_instants. * Add a test for adding and querying UUIDs and instants. * Review comments.
2017-04-29 03:11:55 +00:00
#[test]
fn test_float_and_uuid() {
let expected_uuid = edn::Uuid::parse_str("267bab92-ee39-4ca2-b7f0-1163a85af1fb").expect("valid uuid");
let input = r#"
[[:db/add 101 :test/a #uuid "267bab92-ee39-4ca2-b7f0-1163a85af1fb"]
[:db/add 102 :test/b #f NaN]]
"#;
let edn = parse::value(input).expect("to parse test input");
Parse without copying; parse keyword maps using macros. This is a big commit, but it breaks into two conceptual pieces. The first is to "parse without copying". We replace a stream of an owned collection of edn::ValueAndSpan and instead have a stream of a borrowed collection of &edn::ValueAndSpan references. (Generally, this is represented as an iterator over a slice, but it can be over other things too.) Cloning such iterators is constant time, which improves on cloning an owned collection of edn::ValueAndSpan, which is linear time in the length of the collection and additional time depending on the complexity of the EDN values. The second conceptual piece is to parse keyword maps using a special parser and a macro to build the parser implementations. Before, we created a new edn::ValueAndSpan::Map to represent a keyword map in vector form; since we're working with &edn::ValueAndSpan references now, we can't create an &edn::ValueAndSpan reference with an appropriate lifetime. Therefore we generalize the concept of iteration slightly and turn keyword maps in map form into linear iterators by flattening the value maps. This is a potentially obscuring transformation, so we have to take care to protect against some failure cases. (See the comments and the tests in the code.) After these changes, parsing using `combine` is linear time (and reasonably fast).
2017-05-04 20:40:41 +00:00
let result = Tx::parse(&edn);
UUIDs and instants. Fixes #44, #45, #426, #427. (#438) r=nalexander * Pre: unused import in translate.rs. * Part 2: take a dependency on rusqlite for query arguments. * Part 1: flatten V2 schema into V1. Add UUID and URI. Bump expected ident and bootstrap datom count in tests. * Part 5: parse edn::Value::Uuid. * Part 3: extend ValueType and TypedValue to include Uuid. * Part 4: add Uuid to query arguments. * Part 6: extend db to support Uuid. * Part 8: add a tx-parser test for #f NaN and #uuid. * Part 7: parse and algebrize UUIDs in queries. * Part 1: parse #inst in EDN and throughout query engine. * Part 3: handle instants in db. * Part 2: instants never matches integers in queries. * Part 4: use DateTime for tx_instants. * Add a test for adding and querying UUIDs and instants. * Review comments.
2017-04-29 03:11:55 +00:00
assert_eq!(result.unwrap(),
vec![
Entity::AddOrRetract {
op: OpType::Add,
e: EntidOrLookupRefOrTempId::Entid(Entid::Entid(101)),
a: Entid::Ident(NamespacedKeyword::new("test", "a")),
v: AtomOrLookupRefOrVectorOrMapNotation::Atom(edn::ValueAndSpan::new(edn::SpannedValue::Uuid(expected_uuid), edn::Span(23, 67))),
},
Entity::AddOrRetract {
op: OpType::Add,
e: EntidOrLookupRefOrTempId::Entid(Entid::Entid(102)),
a: Entid::Ident(NamespacedKeyword::new("test", "b")),
v: AtomOrLookupRefOrVectorOrMapNotation::Atom(edn::ValueAndSpan::new(edn::SpannedValue::Float(edn::OrderedFloat(std::f64::NAN)), edn::Span(91, 97))),
},
]);
}
Add tx{-parser} crates; start parsing transactions. (#164) r=rnewman This depends on edn and uses the combine parser combinator library.
2017-01-13 00:08:29 +00:00
#[test]
fn test_entities() {
Implement upsert resolution algorithm. (#186, #283). r=rnewman, f=jsantell * Pre: Implement batch [a v] pair lookup. * Pre: Add InternSet for sharing ref-counted handles to large values. * Pre: Derive more for Entity. * Pre: Return DB from creating; return TxReport from transact. I explicitly am not supporting opening existing databases yet, let alone upgrading databases from earlier versions. That can follow fast once basic transactions are supported. * Pre: Parse string temporary ID entities; remove ValueOrLookupRef. This adds TempId entities, but we can't disambiguate String temporary IDs from values without the use of the schema, so there's no new value branch. Similarly, we can't disambiguate lookup-ref values from two element list values without a schema, so we remove this entirely. We'll handle the ambiguity later in the transactor. * Persist partitions to SQL store; allocate transaction ID. (#186) * Post: Test upserting with vectors. This converts an existing test to EDN: https://github.com/mozilla/mentat/blob/84a80f40f5c888f8452d07bd15f3b5fba49d3963/test/datomish/db_test.cljc#L193. * Implement tempid upsert resolution algorithm. (#184) * Post: Separate Tx out of DB. This is very preliminary, since we don't have a real connection type to manage transactions and their metadata yet. * Post: Comment on implementation choices in the transactor. * Review comment: Put long use lists on separate lines. * Review comment: Accept String: Borrow<S> instead of just String. * Review comment: Address nits.
2017-02-15 00:50:40 +00:00
let input = r#"
[[:db/add 101 :test/a "v"]
[:db/add "tempid" :test/a "v"]
[:db/retract 102 :test/b "w"]]"#;
Add tx{-parser} crates; start parsing transactions. (#164) r=rnewman This depends on edn and uses the combine parser combinator library.
2017-01-13 00:08:29 +00:00
Improve parsing of nested `edn::ValueAndSpan` streams. r=rnewman (#393) * Pre: Expose more in edn. * Pre: Make it easier to work with ValueAndSpan. with_spans() is a temporary hack, needed only because I don't care to parse the bootstrap assertions from text right now. * Part 1a: Add `value_and_span` for parsing nested `edn::ValueAndSpan` instances. I wasn't able to abstract over `edn::Value` and `edn::ValueAndSpan`; there are multiple obstacles. I chose to roll with `edn::ValueAndSpan` since it exposes the additional span information that we will want to form good error messages in the future. * Part 1b: Add keyword_map() parsing an `edn::Value::Vector` into an `edn::Value::map`. * Part 1c: Add `Log`/`.log(...)` for logging parser progress. This is a terrible hack, but it sure helps to debug complicated nested parsers. I don't even know what a principled approach would look like; since our parser combinators are so frequently expressed in code, it's hard to imagine a data-driven interpreter that can help debug things. * Part 2: Use `value_and_span` apparatus in tx-parser/. I break an abstraction boundary by returning a value column `edn::ValueAndSpan` rather than just an `edn::Value`. That is, the transaction processor shouldn't care where the `edn::Value` it is processing arose -- even we care to track that information we should bake it into the `Entity` type. We do this because we need to dynamically parse the value column to support nested maps, and parsing requires a full `edn::ValueAndSpan`. Alternately, we could cheat and fake the spans when parsing nested maps, but that's potentially expensive. * Part 3: Use `value_and_span` apparatus in query-parser/. * Part 4: Use `value_and_span` apparatus in root crate. * Review comment: Make Span and SpanPosition Copy. * Review comment: nits. * Review comment: Make `or` be `or_exactly`. I baked the eof checking directly into the parser, rather than using the skip and eof parsers. I also took the time to restore some tests that were mistakenly commented out. * Review comment: Extract and use def_matches_* macros. * Review comment: .map() as late as possible.
2017-04-06 17:06:28 +00:00
let edn = parse::value(input).expect("to parse test input");
Add tx{-parser} crates; start parsing transactions. (#164) r=rnewman This depends on edn and uses the combine parser combinator library.
2017-01-13 00:08:29 +00:00
Parse without copying; parse keyword maps using macros. This is a big commit, but it breaks into two conceptual pieces. The first is to "parse without copying". We replace a stream of an owned collection of edn::ValueAndSpan and instead have a stream of a borrowed collection of &edn::ValueAndSpan references. (Generally, this is represented as an iterator over a slice, but it can be over other things too.) Cloning such iterators is constant time, which improves on cloning an owned collection of edn::ValueAndSpan, which is linear time in the length of the collection and additional time depending on the complexity of the EDN values. The second conceptual piece is to parse keyword maps using a special parser and a macro to build the parser implementations. Before, we created a new edn::ValueAndSpan::Map to represent a keyword map in vector form; since we're working with &edn::ValueAndSpan references now, we can't create an &edn::ValueAndSpan reference with an appropriate lifetime. Therefore we generalize the concept of iteration slightly and turn keyword maps in map form into linear iterators by flattening the value maps. This is a potentially obscuring transformation, so we have to take care to protect against some failure cases. (See the comments and the tests in the code.) After these changes, parsing using `combine` is linear time (and reasonably fast).
2017-05-04 20:40:41 +00:00
let result = Tx::parse(&edn);
Extract partial storage abstraction; use error-chain throughout. Fixes #328. r=rnewman (#341) * Pre: Drop unneeded tx0 from search results. * Pre: Don't require a schema in some of the DB code. The idea is to separate the transaction applying code, which is schema-aware, from the concrete storage code, which is just concerned with getting bits onto disk. * Pre: Only reference Schema, not DB, in debug module. This is part of a larger separation of the volatile PartitionMap, which is modified every transaction, from the stable Schema, which is infrequently modified. * Pre: Fix indentation. * Extract part of DB to new SchemaTypeChecking trait. * Extract part of DB to new PartitionMapping trait. * Pre: Don't expect :db.part/tx partition to advance when tx fails. This fails right now, because we allocate tx IDs even when we shouldn't. * Sketch a db interface without DB. * Add ValueParseError; use error-chain in tx-parser. This can be simplified when https://github.com/Marwes/combine/issues/86 makes it to a published release, but this unblocks us for now. This converts the `combine` error type `ParseError<&'a [edn::Value]>` to a type with owned `Vec<edn::Value>` collections, re-using `edn::Value::Vector` for making them `Display`. * Pre: Accept Borrow<Schema> instead of just &Schema in debug module. This makes it easy to use Rc<Schema> or Arc<Schema> without inserting &* sigils throughout the code. * Use error-chain in query-parser. There are a few things to point out here: - the fine grained error types have been flattened into one crate-wide error type; it's pretty easy to regain the granularity as needed. - edn::ParseError is automatically lifted to mentat_query_parser::errors::Error; - we use mentat_parser_utils::ValueParser to maintain parsing error information from `combine`. * Patch up top-level. * Review comment: Only `borrow()` once.
2017-02-24 23:32:41 +00:00
assert_eq!(result.unwrap(),
vec![
Consolidate Entity::{Add, Retract} to Entity::AddOrRetract. Fixes #255. r=nalexander (#265)
2017-02-08 23:45:09 +00:00
Entity::AddOrRetract {
Implement upsert resolution algorithm. (#186, #283). r=rnewman, f=jsantell * Pre: Implement batch [a v] pair lookup. * Pre: Add InternSet for sharing ref-counted handles to large values. * Pre: Derive more for Entity. * Pre: Return DB from creating; return TxReport from transact. I explicitly am not supporting opening existing databases yet, let alone upgrading databases from earlier versions. That can follow fast once basic transactions are supported. * Pre: Parse string temporary ID entities; remove ValueOrLookupRef. This adds TempId entities, but we can't disambiguate String temporary IDs from values without the use of the schema, so there's no new value branch. Similarly, we can't disambiguate lookup-ref values from two element list values without a schema, so we remove this entirely. We'll handle the ambiguity later in the transactor. * Persist partitions to SQL store; allocate transaction ID. (#186) * Post: Test upserting with vectors. This converts an existing test to EDN: https://github.com/mozilla/mentat/blob/84a80f40f5c888f8452d07bd15f3b5fba49d3963/test/datomish/db_test.cljc#L193. * Implement tempid upsert resolution algorithm. (#184) * Post: Separate Tx out of DB. This is very preliminary, since we don't have a real connection type to manage transactions and their metadata yet. * Post: Comment on implementation choices in the transactor. * Review comment: Put long use lists on separate lines. * Review comment: Accept String: Borrow<S> instead of just String. * Review comment: Address nits.
2017-02-15 00:50:40 +00:00
op: OpType::Add,
e: EntidOrLookupRefOrTempId::Entid(Entid::Entid(101)),
Add tx{-parser} crates; start parsing transactions. (#164) r=rnewman This depends on edn and uses the combine parser combinator library.
2017-01-13 00:08:29 +00:00
a: Entid::Ident(NamespacedKeyword::new("test", "a")),
Improve parsing of nested `edn::ValueAndSpan` streams. r=rnewman (#393) * Pre: Expose more in edn. * Pre: Make it easier to work with ValueAndSpan. with_spans() is a temporary hack, needed only because I don't care to parse the bootstrap assertions from text right now. * Part 1a: Add `value_and_span` for parsing nested `edn::ValueAndSpan` instances. I wasn't able to abstract over `edn::Value` and `edn::ValueAndSpan`; there are multiple obstacles. I chose to roll with `edn::ValueAndSpan` since it exposes the additional span information that we will want to form good error messages in the future. * Part 1b: Add keyword_map() parsing an `edn::Value::Vector` into an `edn::Value::map`. * Part 1c: Add `Log`/`.log(...)` for logging parser progress. This is a terrible hack, but it sure helps to debug complicated nested parsers. I don't even know what a principled approach would look like; since our parser combinators are so frequently expressed in code, it's hard to imagine a data-driven interpreter that can help debug things. * Part 2: Use `value_and_span` apparatus in tx-parser/. I break an abstraction boundary by returning a value column `edn::ValueAndSpan` rather than just an `edn::Value`. That is, the transaction processor shouldn't care where the `edn::Value` it is processing arose -- even we care to track that information we should bake it into the `Entity` type. We do this because we need to dynamically parse the value column to support nested maps, and parsing requires a full `edn::ValueAndSpan`. Alternately, we could cheat and fake the spans when parsing nested maps, but that's potentially expensive. * Part 3: Use `value_and_span` apparatus in query-parser/. * Part 4: Use `value_and_span` apparatus in root crate. * Review comment: Make Span and SpanPosition Copy. * Review comment: nits. * Review comment: Make `or` be `or_exactly`. I baked the eof checking directly into the parser, rather than using the skip and eof parsers. I also took the time to restore some tests that were mistakenly commented out. * Review comment: Extract and use def_matches_* macros. * Review comment: .map() as late as possible.
2017-04-06 17:06:28 +00:00
v: AtomOrLookupRefOrVectorOrMapNotation::Atom(edn::ValueAndSpan::new(edn::SpannedValue::Text("v".into()), edn::Span(23, 26))),
Implement upsert resolution algorithm. (#186, #283). r=rnewman, f=jsantell * Pre: Implement batch [a v] pair lookup. * Pre: Add InternSet for sharing ref-counted handles to large values. * Pre: Derive more for Entity. * Pre: Return DB from creating; return TxReport from transact. I explicitly am not supporting opening existing databases yet, let alone upgrading databases from earlier versions. That can follow fast once basic transactions are supported. * Pre: Parse string temporary ID entities; remove ValueOrLookupRef. This adds TempId entities, but we can't disambiguate String temporary IDs from values without the use of the schema, so there's no new value branch. Similarly, we can't disambiguate lookup-ref values from two element list values without a schema, so we remove this entirely. We'll handle the ambiguity later in the transactor. * Persist partitions to SQL store; allocate transaction ID. (#186) * Post: Test upserting with vectors. This converts an existing test to EDN: https://github.com/mozilla/mentat/blob/84a80f40f5c888f8452d07bd15f3b5fba49d3963/test/datomish/db_test.cljc#L193. * Implement tempid upsert resolution algorithm. (#184) * Post: Separate Tx out of DB. This is very preliminary, since we don't have a real connection type to manage transactions and their metadata yet. * Post: Comment on implementation choices in the transactor. * Review comment: Put long use lists on separate lines. * Review comment: Accept String: Borrow<S> instead of just String. * Review comment: Address nits.
2017-02-15 00:50:40 +00:00
},
Entity::AddOrRetract {
Consolidate Entity::{Add, Retract} to Entity::AddOrRetract. Fixes #255. r=nalexander (#265)
2017-02-08 23:45:09 +00:00
op: OpType::Add,
Lookup refs, nested vector values, map notation. Fixes #180, fixes #183, fixes #284. (#382) r=rnewman * Pre: Fix error in parser macros. * Pre: Make test unwrapping more verbose. * Pre: Make lookup refs be (lookup-ref a v) in the entity position. This has the advantage of being explicit in all situations and unambiguous at parse-time. This choice agrees with the Clojure implementation but not with Datomic. Datomic treats [a v] as a lookup ref, is ambiguous at parse-time, and is disambiguated in ways I do not understand at transaction time. We mooted making lookup refs [[a v]] and outlawing nested value vectors in transactions, but after implementing that approach I decided it was better to handle lookup refs at parse time and therefore outlawing nested value vectors is not necessary. * Handle lookup refs in the entity and value columns. Fixes #183. * Pre 0a: Use a stack instead of into_iter. * Pre 0b: Dedent. * Pre 0c: Handle `e` after `v`. This allows to use the original `e` while handling `v`. * Explode value lists for :db.cardinality/many attributes. Fixes #284. * Parse and accept map notation. Fixes #180. * Pre: Modernize add() and retract() into one add_or_retract(). * Pre: Add is_collection and is_atom to edn::Value. * Pre: Differentiate atoms from lookup-refs in value position. Initially, I expected to accept arbitrary edn::Value instances in the value position, and to differentiate in the transactor. However, the implementation quickly became a two-stage parser, since we always wanted to parse the resulting value position into some other known thing using the tx-parser. To save calls into the parser and to allow the parser to move forward with a smaller API surface, I push as much of this parsing as possible into the initial parse. * Pre: Modernize entities(). * Pre: Quote edn::Value::Text in Display. * Review comment: Add and use edn::Value::into_atom. * Review comment: Use skip(eof()) throughout. * Review comment: VecDeque instead of Vec. * Review comment: Part 0: Rename TempId to TempIdHandle. * Review comment: Part 1: Differentiate internal and external tempids. This breaks an abstraction boundary by pushing the Internal/External split up to the Entity level in tx/ and tx-parser/. This just makes it easier to explode Entity map notation instances into Entity instances, taking an existing External tempid :db/id or generating a new Internal tempid as appropriate. To do this without breaking the abstraction boundary would require adding flexibility to the transaction processor: we'd need to be able to turn Entity instances into some internal enum and handle the two cases independently. It wouldn't be too hard, but this reduces the combinatorial type explosion.
2017-03-27 23:30:04 +00:00
e: EntidOrLookupRefOrTempId::TempId(TempId::External("tempid".into())),
Implement upsert resolution algorithm. (#186, #283). r=rnewman, f=jsantell * Pre: Implement batch [a v] pair lookup. * Pre: Add InternSet for sharing ref-counted handles to large values. * Pre: Derive more for Entity. * Pre: Return DB from creating; return TxReport from transact. I explicitly am not supporting opening existing databases yet, let alone upgrading databases from earlier versions. That can follow fast once basic transactions are supported. * Pre: Parse string temporary ID entities; remove ValueOrLookupRef. This adds TempId entities, but we can't disambiguate String temporary IDs from values without the use of the schema, so there's no new value branch. Similarly, we can't disambiguate lookup-ref values from two element list values without a schema, so we remove this entirely. We'll handle the ambiguity later in the transactor. * Persist partitions to SQL store; allocate transaction ID. (#186) * Post: Test upserting with vectors. This converts an existing test to EDN: https://github.com/mozilla/mentat/blob/84a80f40f5c888f8452d07bd15f3b5fba49d3963/test/datomish/db_test.cljc#L193. * Implement tempid upsert resolution algorithm. (#184) * Post: Separate Tx out of DB. This is very preliminary, since we don't have a real connection type to manage transactions and their metadata yet. * Post: Comment on implementation choices in the transactor. * Review comment: Put long use lists on separate lines. * Review comment: Accept String: Borrow<S> instead of just String. * Review comment: Address nits.
2017-02-15 00:50:40 +00:00
a: Entid::Ident(NamespacedKeyword::new("test", "a")),
Improve parsing of nested `edn::ValueAndSpan` streams. r=rnewman (#393) * Pre: Expose more in edn. * Pre: Make it easier to work with ValueAndSpan. with_spans() is a temporary hack, needed only because I don't care to parse the bootstrap assertions from text right now. * Part 1a: Add `value_and_span` for parsing nested `edn::ValueAndSpan` instances. I wasn't able to abstract over `edn::Value` and `edn::ValueAndSpan`; there are multiple obstacles. I chose to roll with `edn::ValueAndSpan` since it exposes the additional span information that we will want to form good error messages in the future. * Part 1b: Add keyword_map() parsing an `edn::Value::Vector` into an `edn::Value::map`. * Part 1c: Add `Log`/`.log(...)` for logging parser progress. This is a terrible hack, but it sure helps to debug complicated nested parsers. I don't even know what a principled approach would look like; since our parser combinators are so frequently expressed in code, it's hard to imagine a data-driven interpreter that can help debug things. * Part 2: Use `value_and_span` apparatus in tx-parser/. I break an abstraction boundary by returning a value column `edn::ValueAndSpan` rather than just an `edn::Value`. That is, the transaction processor shouldn't care where the `edn::Value` it is processing arose -- even we care to track that information we should bake it into the `Entity` type. We do this because we need to dynamically parse the value column to support nested maps, and parsing requires a full `edn::ValueAndSpan`. Alternately, we could cheat and fake the spans when parsing nested maps, but that's potentially expensive. * Part 3: Use `value_and_span` apparatus in query-parser/. * Part 4: Use `value_and_span` apparatus in root crate. * Review comment: Make Span and SpanPosition Copy. * Review comment: nits. * Review comment: Make `or` be `or_exactly`. I baked the eof checking directly into the parser, rather than using the skip and eof parsers. I also took the time to restore some tests that were mistakenly commented out. * Review comment: Extract and use def_matches_* macros. * Review comment: .map() as late as possible.
2017-04-06 17:06:28 +00:00
v: AtomOrLookupRefOrVectorOrMapNotation::Atom(edn::ValueAndSpan::new(edn::SpannedValue::Text("v".into()), edn::Span(55, 58))),
Add tx{-parser} crates; start parsing transactions. (#164) r=rnewman This depends on edn and uses the combine parser combinator library.
2017-01-13 00:08:29 +00:00
},
Consolidate Entity::{Add, Retract} to Entity::AddOrRetract. Fixes #255. r=nalexander (#265)
2017-02-08 23:45:09 +00:00
Entity::AddOrRetract {
op: OpType::Retract,
Implement upsert resolution algorithm. (#186, #283). r=rnewman, f=jsantell * Pre: Implement batch [a v] pair lookup. * Pre: Add InternSet for sharing ref-counted handles to large values. * Pre: Derive more for Entity. * Pre: Return DB from creating; return TxReport from transact. I explicitly am not supporting opening existing databases yet, let alone upgrading databases from earlier versions. That can follow fast once basic transactions are supported. * Pre: Parse string temporary ID entities; remove ValueOrLookupRef. This adds TempId entities, but we can't disambiguate String temporary IDs from values without the use of the schema, so there's no new value branch. Similarly, we can't disambiguate lookup-ref values from two element list values without a schema, so we remove this entirely. We'll handle the ambiguity later in the transactor. * Persist partitions to SQL store; allocate transaction ID. (#186) * Post: Test upserting with vectors. This converts an existing test to EDN: https://github.com/mozilla/mentat/blob/84a80f40f5c888f8452d07bd15f3b5fba49d3963/test/datomish/db_test.cljc#L193. * Implement tempid upsert resolution algorithm. (#184) * Post: Separate Tx out of DB. This is very preliminary, since we don't have a real connection type to manage transactions and their metadata yet. * Post: Comment on implementation choices in the transactor. * Review comment: Put long use lists on separate lines. * Review comment: Accept String: Borrow<S> instead of just String. * Review comment: Address nits.
2017-02-15 00:50:40 +00:00
e: EntidOrLookupRefOrTempId::Entid(Entid::Entid(102)),
a: Entid::Ident(NamespacedKeyword::new("test", "b")),
Improve parsing of nested `edn::ValueAndSpan` streams. r=rnewman (#393) * Pre: Expose more in edn. * Pre: Make it easier to work with ValueAndSpan. with_spans() is a temporary hack, needed only because I don't care to parse the bootstrap assertions from text right now. * Part 1a: Add `value_and_span` for parsing nested `edn::ValueAndSpan` instances. I wasn't able to abstract over `edn::Value` and `edn::ValueAndSpan`; there are multiple obstacles. I chose to roll with `edn::ValueAndSpan` since it exposes the additional span information that we will want to form good error messages in the future. * Part 1b: Add keyword_map() parsing an `edn::Value::Vector` into an `edn::Value::map`. * Part 1c: Add `Log`/`.log(...)` for logging parser progress. This is a terrible hack, but it sure helps to debug complicated nested parsers. I don't even know what a principled approach would look like; since our parser combinators are so frequently expressed in code, it's hard to imagine a data-driven interpreter that can help debug things. * Part 2: Use `value_and_span` apparatus in tx-parser/. I break an abstraction boundary by returning a value column `edn::ValueAndSpan` rather than just an `edn::Value`. That is, the transaction processor shouldn't care where the `edn::Value` it is processing arose -- even we care to track that information we should bake it into the `Entity` type. We do this because we need to dynamically parse the value column to support nested maps, and parsing requires a full `edn::ValueAndSpan`. Alternately, we could cheat and fake the spans when parsing nested maps, but that's potentially expensive. * Part 3: Use `value_and_span` apparatus in query-parser/. * Part 4: Use `value_and_span` apparatus in root crate. * Review comment: Make Span and SpanPosition Copy. * Review comment: nits. * Review comment: Make `or` be `or_exactly`. I baked the eof checking directly into the parser, rather than using the skip and eof parsers. I also took the time to restore some tests that were mistakenly commented out. * Review comment: Extract and use def_matches_* macros. * Review comment: .map() as late as possible.
2017-04-06 17:06:28 +00:00
v: AtomOrLookupRefOrVectorOrMapNotation::Atom(edn::ValueAndSpan::new(edn::SpannedValue::Text("w".into()), edn::Span(86, 89))),
Add tx{-parser} crates; start parsing transactions. (#164) r=rnewman This depends on edn and uses the combine parser combinator library.
2017-01-13 00:08:29 +00:00
},
Extract partial storage abstraction; use error-chain throughout. Fixes #328. r=rnewman (#341) * Pre: Drop unneeded tx0 from search results. * Pre: Don't require a schema in some of the DB code. The idea is to separate the transaction applying code, which is schema-aware, from the concrete storage code, which is just concerned with getting bits onto disk. * Pre: Only reference Schema, not DB, in debug module. This is part of a larger separation of the volatile PartitionMap, which is modified every transaction, from the stable Schema, which is infrequently modified. * Pre: Fix indentation. * Extract part of DB to new SchemaTypeChecking trait. * Extract part of DB to new PartitionMapping trait. * Pre: Don't expect :db.part/tx partition to advance when tx fails. This fails right now, because we allocate tx IDs even when we shouldn't. * Sketch a db interface without DB. * Add ValueParseError; use error-chain in tx-parser. This can be simplified when https://github.com/Marwes/combine/issues/86 makes it to a published release, but this unblocks us for now. This converts the `combine` error type `ParseError<&'a [edn::Value]>` to a type with owned `Vec<edn::Value>` collections, re-using `edn::Value::Vector` for making them `Display`. * Pre: Accept Borrow<Schema> instead of just &Schema in debug module. This makes it easy to use Rc<Schema> or Arc<Schema> without inserting &* sigils throughout the code. * Use error-chain in query-parser. There are a few things to point out here: - the fine grained error types have been flattened into one crate-wide error type; it's pretty easy to regain the granularity as needed. - edn::ParseError is automatically lifted to mentat_query_parser::errors::Error; - we use mentat_parser_utils::ValueParser to maintain parsing error information from `combine`. * Patch up top-level. * Review comment: Only `borrow()` once.
2017-02-24 23:32:41 +00:00
]);
Add tx{-parser} crates; start parsing transactions. (#164) r=rnewman This depends on edn and uses the combine parser combinator library.
2017-01-13 00:08:29 +00:00
}
// TODO: test error handling in select cases.
Reference in a new issue Copy permalink