Commit graph

19 commits

Author SHA1 Message Date
Nick Alexander
d419554361 Revert "Add Datomic, for testing."
This reverts commit 1b1cc6f18e.

This should never have escaped the lab; Datomic is not a dependency of
Datomish.
2016-08-31 16:17:08 -07:00
Nick Alexander
71446a3af5 Completely rewrite main transaction logic to be faster.
This is almost complete; it passes the test suite save for retracting
fulltext datoms correctly.

There's a lot to say about this approach, but I don't have time to give
too many details.  The broad outline is as follows.  We collect datoms
to add and retract in a tx_lookup table.  Depending on flags ("search
value" sv and "search value type tag" svalue_type_tag) we "complete" the
tx_lookup table by joining matching datoms.  This allows us to find
datoms that are present (and should not be added as part of the
transaction, or should be retracted as part of the transaction, or
should be replaced as part of the transaction.  We complete the
tx_lookup (in place!) in two separate INSERTs to avoid a quadratic
two-table walk (explain the queries to observe that both INSERTs walk
the lookup table once and then use the datoms indexes to complete the
matching values).

We could simplify the code by using multiple lookup tables, both for the
two cases of search parameters (eav vs. ea) and for the incomplete and
completed rows.  Right now we differentiate the former with NULL checks,
and the latter by incrementing the added0 column.  It performs well
enough, so I haven't tried to understand the performance of separating
these things.

After the tx_lookup table is completed, we build the transaction from
it; and update the datoms materialized view table as well.  Observe the
careful handling of the "search value" sv parameters to handle replacing
:db.cardinality/one datoms.

Finally, we read the processed transaction back to produce to the API.
This is strictly to match the Datomic API; we might make allow to skip
this, since many consumers will not want to stream this over the wire.

Rough timings show the transactor processing a single >50k datom
transaction in about 3.5s, of which less than 0.5s is spent in the
expensive joins.  Further, repeating the processing of the same
transaction is only about 3.5s again!  That's the worst possible for the
joins, since every single inserted datom will already be present in the
database, making the most expensive join match every row.
2016-08-30 18:22:10 -07:00
Nick Alexander
1b1cc6f18e Add Datomic, for testing. 2016-08-30 18:22:09 -07:00
Richard Newman
3ab0428ee0 Add some Leiningen plugins to make testing prettier. 2016-08-30 18:11:25 -07:00
Richard Newman
f4344fa28a Bump JVM stack size to 4MB and eliminate tiered compilation.
This allows CLJS compilation to complete for complex go-pair forms.

See <https://github.com/emezeske/lein-cljsbuild/issues/303> for more details.
2016-08-19 12:11:44 -07:00
Richard Newman
3f882ebdaa Bump honeysql to 0.8.0, which includes mainline ClojureScript support. 2016-08-05 13:06:42 -07:00
Nick Alexander
cca5010671 Part 1: Fix testing errors.
Some of these were just typos, but `with-open` was fatally flawed on
CLJS (we couldn't call `.close` at all), and `deftest-async` was hiding
all failures (due to a typo).
2016-07-13 18:19:22 -07:00
Richard Newman
75810b924e Use cljsbuild 1.1.3.
Signed-off-by: Richard Newman <rnewman@twinql.com>
2016-07-12 19:06:37 -07:00
Richard Newman
3aac28aa13 Take a dependency on honeysql for building queries.
Signed-off-by: Richard Newman <rnewman@twinql.com>
2016-07-12 19:05:39 -07:00
Richard Newman
b3b9982740 Take a dependency on DataScript.
Signed-off-by: Richard Newman <rnewman@twinql.com>
2016-07-12 19:05:18 -07:00
Richard Newman
798e05bd9a Use correct project name in defproject.
Signed-off-by: Richard Newman <rnewman@twinql.com>
2016-07-12 19:04:53 -07:00
Nick Alexander
724c37466d Add an SQLite connection abstraction. 2016-07-12 13:56:26 -07:00
Nick Alexander
d5cfbeaa45 Add tempfile and with-tempfile to CLJS.
We already have a nice library like this for CLJ (tempfile); this builds
the same thing for CLJS, using Node.js's tmp.
2016-07-12 13:56:26 -07:00
Nick Alexander
0a312b4f40 Add an async and async testing framework.
This is a well-worn idea: use a `promise-channel` of `[result nil]` or
`[nil error]` pairs.  The `go-pair` and `<?` macros handle catching
exceptions (important, given that synchronous CLJ code expects to throw
rather than return an error promise or similar), allowing code like:
```
(go-pair
  (let [result (<? (pair-chan-fn))]
    (when (not result)
      (throw (Exception. "No result!")))
    (transform result)))
```
to be expressed naturally.  These are the equivalents of `async` and
`await` in JS.

The implementation is complicated by significant incompatibilities
between CLJ and CLJS.  The solution presented here takes care to
separate the macro definitions into CLJ.  Sadly, this requires
namespacing the per-environment symbols explicitly; but we hope to
minimize such code in files like this.

The most significant restriction to this approach is that consumers must
require the transitive dependencies of the macro-defining modules.  See
the included tests (both CLJ and CLJS) for the appropriate
incantations (for pair-chan, core.async, and test).
2016-07-12 13:56:26 -07:00
Nick Alexander
08f7084d92 Add lein-doo to run ClojureScript tests. 2016-07-06 16:42:40 -07:00
Nick Alexander
f2365646d2 Add lein-cljsbuild and adapt datascript's release-js vehicle. 2016-07-06 16:42:37 -07:00
Richard Newman
27d35f030d Take dependencies on a few libraries.
Signed-off-by: Richard Newman <rnewman@twinql.com>
2016-07-05 17:01:27 -07:00
Richard Newman
9e3df19798 Stub code, build, and REPL.
Signed-off-by: Richard Newman <rnewman@twinql.com>
2016-07-05 12:36:55 -07:00
Richard Newman
f7621712df Stub out Leiningen and npm project files.
Signed-off-by: Richard Newman <rnewman@twinql.com>
2016-07-05 12:36:45 -07:00