Commit graph

74 commits

Author SHA1 Message Date
Nick Alexander
e5917406b4 Add {un}listen{-chan}! to connection. (#61) 2016-10-13 16:11:16 -07:00
Nick Alexander
a8ad79d0e6 Make <transact! run in a critical section. (#80) 2016-10-11 20:32:35 -07:00
Nick Alexander
2081ca4563 Pre: Add unlimited-buffer and unblocking-chan?. 2016-10-11 20:32:35 -07:00
Nick Alexander
a89224673a Pre: Standardize util-test file location.
This is the "standard" Clojure layout, and makes the testing integration
in Emacs CIDER happy.
2016-10-11 20:04:57 -07:00
Richard Newman
0789219551 Add order-by exercising to the JS test script. 2016-10-07 20:12:17 -07:00
Richard Newman
b8a3f5fe2b Use scalar and collection find specs in tests.js. 2016-10-07 12:03:17 -07:00
Richard Newman
f775e67912 Add a test for long strings round-tripping through node-sqlite3. Fixes #65. 2016-10-07 11:58:50 -07:00
Richard Newman
e89544beba Implement all four find specs. Fixes #38. r=nalexander 2016-10-07 11:02:35 -07:00
Richard Newman
ce67644fd5 Part 3: expand example Node code. 2016-10-05 12:54:48 -07:00
Nick Alexander
3cd64fb4d8 Review comments. 2016-09-30 17:00:27 -07:00
Nick Alexander
611d44fcce Process lookup-refs in batches. Fixes #25.
This uses a common table expression and multiple SQL calls rather than a
temporary table, since transactions with huge numbers of distinct
lookup-refs are likely to be very rare.

We mark lookup-refs with `lookup-ref`, which is a little awkward because
binding `(let [[a v] lookup-ref] ...)` doesn't directly work, but avoids
some ambiguity present in Datomic and DataScript around interpreting
lookup-refs as multiple value lists.  (Which bit the tests in an earlier
version of this patch!)
2016-09-30 16:47:04 -07:00
Richard Newman
6ab93208cb Part 2: implement complex 'or' translation. Fixes #57. r=nalexander
We implement sql-projection-for-simple-variable-list to allow us to add
a projection to subqueries.
2016-09-29 15:46:15 -07:00
Richard Newman
1296b8090f Allow sets of attributes in fulltext expressions. Fixes #54. r=nalexander 2016-09-26 16:34:37 -07:00
Richard Newman
9587311412 Include deps.cljs giving externs for Node.js consumers; normalize build output.
cljsbuild using leiningen projects that depend on Datomish will
automatically include the externs.
2016-09-22 16:24:37 -07:00
Richard Newman
17d7eaec7b Add a babelified test file, Webpack the add-on, and make the JS API work.
We concatenate a simple setTimeout monkeypatch onto the add-on itself.
2016-09-22 15:59:15 -07:00
Richard Newman
360f7622e8 Add handling of simple schemas. Fixes #53. 2016-09-22 15:59:15 -07:00
Nick Alexander
1a30306314 Move datomish.api into exported namespace. 2016-09-19 12:03:09 -07:00
Richard Newman
1cfbf8498c Introduce js-sqlite as an abstraction wrapper around Sqlite.jsm and promise-sqlite. 2016-09-08 19:11:44 -07:00
Nick Alexander
659c26339c Serialize and de-serialize non-keyword schema values correctly. Fixes #52. 2016-09-07 17:31:30 -07:00
Nick Alexander
362bdb2028 Add :db.fn/retractAttribute and :db.fn/retractEntity. Fixes #46.
There's no distinction made for fulltext attributes, since the values
found by the retractAttributes SELECT are already rowids into the
fulltext_values table and therefore need no additional mapping.
2016-09-06 13:13:20 -07:00
Nick Alexander
cfdce12c1a Pre: Use deftest-db in tests. 2016-09-06 13:13:20 -07:00
Richard Newman
5424e61be4 Review comment: args -> inputs. 2016-09-02 16:23:30 -07:00
Richard Newman
1197764949 Implement :limit and :order-by-vars. Fixes #37.
We'd like this to be part of the query syntax itself, but doing so
requires extending DataScript's parser.

Instead we generalize our `args` to `options`, and take `:limit`
and `:order-by-vars`. The former must be an integer or nil, and the
latter is an array of `[var direction]` pairs.

This commit includes descriptive error messages and tests for success
and failure.
2016-09-02 08:32:45 -07:00
Richard Newman
c89f1f74c5 Add some Tofino-ish tests. 2016-08-30 18:22:22 -07:00
Richard Newman
6fbd63fed2 Review comment: clarity. 2016-08-30 18:22:22 -07:00
Richard Newman
495e5a737e Review comment: check that Datalog vars don't have a namespace.
And fix the only test that uses syntax-quote, which namespace-qualifies bare symbols.
2016-08-30 18:22:22 -07:00
Richard Newman
0f399eafb0 Review comment: better UUID support. 2016-08-30 18:22:22 -07:00
Richard Newman
f225dbe734 Implement get-else. 2016-08-30 18:22:22 -07:00
Richard Newman
38cd30a895 Implement aggregation. Fixes #39. 2016-08-30 18:22:22 -07:00
Richard Newman
7d63c2185d Rework query tests to use a live DB. Fixes #35. 2016-08-30 18:22:11 -07:00
Richard Newman
479a7fd583 Project real values. Fixes #30. 2016-08-30 18:22:11 -07:00
Richard Newman
1c6244db5b Implement type-aware querying. Fixes #14.
* Alter how clauses are concatenated. They now preserve order more accurately.
* Track mappings between vars and extracted type columns.
* Generate type code constraints.
* Push known types down into :not.
* Push known types down into :or.
* Tests and test fixes.
2016-08-30 18:22:11 -07:00
Richard Newman
9136ba7425 Basic test for ds/fulltext?. 2016-08-30 18:22:11 -07:00
Nick Alexander
3775c7c773 Start importing places. This is just about profiling for now. 2016-08-30 18:22:10 -07:00
Nick Alexander
71446a3af5 Completely rewrite main transaction logic to be faster.
This is almost complete; it passes the test suite save for retracting
fulltext datoms correctly.

There's a lot to say about this approach, but I don't have time to give
too many details.  The broad outline is as follows.  We collect datoms
to add and retract in a tx_lookup table.  Depending on flags ("search
value" sv and "search value type tag" svalue_type_tag) we "complete" the
tx_lookup table by joining matching datoms.  This allows us to find
datoms that are present (and should not be added as part of the
transaction, or should be retracted as part of the transaction, or
should be replaced as part of the transaction.  We complete the
tx_lookup (in place!) in two separate INSERTs to avoid a quadratic
two-table walk (explain the queries to observe that both INSERTs walk
the lookup table once and then use the datoms indexes to complete the
matching values).

We could simplify the code by using multiple lookup tables, both for the
two cases of search parameters (eav vs. ea) and for the incomplete and
completed rows.  Right now we differentiate the former with NULL checks,
and the latter by incrementing the added0 column.  It performs well
enough, so I haven't tried to understand the performance of separating
these things.

After the tx_lookup table is completed, we build the transaction from
it; and update the datoms materialized view table as well.  Observe the
careful handling of the "search value" sv parameters to handle replacing
:db.cardinality/one datoms.

Finally, we read the processed transaction back to produce to the API.
This is strictly to match the Datomic API; we might make allow to skip
this, since many consumers will not want to stream this over the wire.

Rough timings show the transactor processing a single >50k datom
transaction in about 3.5s, of which less than 0.5s is spent in the
expensive joins.  Further, repeating the processing of the same
transaction is only about 3.5s again!  That's the worst possible for the
joins, since every single inserted datom will already be present in the
database, making the most expensive join match every row.
2016-08-30 18:22:10 -07:00
Nick Alexander
b59c71cb00 Extract datomish.db.debug from test code, in order to use it during development. 2016-08-30 18:22:10 -07:00
Nick Alexander
a1cc372d43 Fix :db.unique/value, which should be per (a, v) pair, not per v-value. 2016-08-30 18:22:10 -07:00
Nick Alexander
b29e5caec0 Implement parts: Make the DB allocate and persist entity IDs.
This implementation is inefficient because each allocated temporary ID
touches the database, but it's enough to allow to re-open DBs.
2016-08-30 18:22:09 -07:00
Nick Alexander
ff9a75ae09 Tag values with value type tags in SQLite. 2016-08-30 18:22:09 -07:00
Nick Alexander
4d34c820b8 Add d/q; make query minimally schema aware. 2016-08-30 18:22:09 -07:00
Nick Alexander
e7e84e0a90 Add d/{ident,entid} for mapping between keyword idents and integer entids. 2016-08-30 18:22:09 -07:00
Richard Newman
9f97cc8f2d Clarify d/<connect. 2016-08-30 18:22:09 -07:00
Richard Newman
1a54cec1ba Use correct view for fulltext+non-fulltext. 2016-08-05 16:29:21 -07:00
Richard Newman
fc845a9950 Implement basic fulltext binding. r=nalexander 2016-08-05 13:04:06 -07:00
Richard Newman
4c2230929e Turn Source into a protocol. Allow source->from to switch on attribute. r=nalexander 2016-08-05 13:03:48 -07:00
Nick Alexander
c948902c52 Follow-up: Extract datomish.transact.explode.
This required pushing the ID literal out of transact.  I elected to put
them in DB, since literal allocation will end up in IDB eventually.
2016-08-04 17:54:50 -07:00
Nick Alexander
d8c976c3ad Follow-up: split the monolith!
This was a little more tricky than might be expected because the
initialization process uses the transactor to bootstrap the database.
Since Clojure doesn't accept mutually recursive modules, this
necessitated a third module, namely "db-factory", which uses both "db"
and "transact".  While I was here, I started an "api" module, to paper
over the potentially complicated internal module structure for external
consumers.  In time, this "api" module may also grow CLJS-specific JS
transformations.
2016-08-04 17:54:32 -07:00
Nick Alexander
73b155cfdc Follow-up: don't accept schema when creating DB.
This schema was already ignored.
2016-08-04 16:34:48 -07:00
Nick Alexander
44db8116bf Handle nested maps, sequences in maps, and reverse references. 2016-08-04 14:26:20 -07:00
Nick Alexander
d9a8cb0d6a Don't accept user-provided tx values.
This agrees with Datomic.  DataScript allows tx values, possibly to
allow reconstructing DBs from Datom streams, but appears to handle
user-provided tx values in the transactor inconsistently.
2016-08-04 14:26:20 -07:00