Commit graph

475 commits

Author SHA1 Message Date
Richard Newman
1e04425287 Dates in and out. 2016-08-30 18:22:21 -07:00
Richard Newman
0cd6da1039 Add :db/doc as a default attribute. 2016-08-30 18:22:21 -07:00
Richard Newman
893f944596 Error out when an attribute could not be interned when transacting. 2016-08-30 18:22:11 -07:00
Richard Newman
7d63c2185d Rework query tests to use a live DB. Fixes #35. 2016-08-30 18:22:11 -07:00
Richard Newman
bdac50e03c Define deftest-db to do async testing with an open DB. 2016-08-30 18:22:11 -07:00
Richard Newman
479a7fd583 Project real values. Fixes #30. 2016-08-30 18:22:11 -07:00
Richard Newman
1c6244db5b Implement type-aware querying. Fixes #14.
* Alter how clauses are concatenated. They now preserve order more accurately.
* Track mappings between vars and extracted type columns.
* Generate type code constraints.
* Push known types down into :not.
* Push known types down into :or.
* Tests and test fixes.
2016-08-30 18:22:11 -07:00
Richard Newman
2529378725 Rewrite <resolve-lookup-refs to work for lookup refs.
Note that `go` (and `go-pair`) don't descend into `for` comprehensions
and other situations in which a fn is created. This commit rewrites to
use nested `loop`s, and also improves use of `<av`.
2016-08-30 18:22:11 -07:00
Richard Newman
86b5a8ea8a Rework <apply-entities to be 40% faster and not blow the stack in CLJS.
* Batch up datoms into a smaller number of queries, improving transact speed by about 50%.
* Restore transacting FTS attributes.
* Implement retraction of freetext datoms.
2016-08-30 18:22:11 -07:00
Richard Newman
9136ba7425 Basic test for ds/fulltext?. 2016-08-30 18:22:11 -07:00
Richard Newman
587959d1ff Simplify id-literal?, avoid some consing. 2016-08-30 18:22:11 -07:00
Richard Newman
b4112bbe52 Symbolicating is not expensive. 2016-08-30 18:22:10 -07:00
Richard Newman
43a96efd32 Memoize and simplify parts of insertion. 2016-08-30 18:22:10 -07:00
Richard Newman
6b480a700d Minor perf improvement: use UNION ALL to populate tx_lookup in a single INSERT. 2016-08-30 18:22:10 -07:00
Richard Newman
f53869ba6f Minor perf improvement: create idx_tx_lookup_added after populating tx_lookup. 2016-08-30 18:22:10 -07:00
Richard Newman
2a55b138f6 Limit number of imported places, not number of imported rows. Default to 1000. 2016-08-30 18:22:10 -07:00
Richard Newman
5836b1867e Don't attempt to add a places visit list of (nil). 2016-08-30 18:22:10 -07:00
Richard Newman
db8d043b27 Places import: add a title import function to exercise lookup refs. 2016-08-30 18:22:10 -07:00
Richard Newman
5ec79f6be3 Places import: LEFT JOIN correctly to pick up unvisited pages. 2016-08-30 18:22:10 -07:00
Richard Newman
f85968de88 Places import: title is the same for each returned row. Don't filter. 2016-08-30 18:22:10 -07:00
Richard Newman
d7018aedaa Remove dependency on test code from places importer. 2016-08-30 18:22:10 -07:00
Nick Alexander
3775c7c773 Start importing places. This is just about profiling for now. 2016-08-30 18:22:10 -07:00
Nick Alexander
71446a3af5 Completely rewrite main transaction logic to be faster.
This is almost complete; it passes the test suite save for retracting
fulltext datoms correctly.

There's a lot to say about this approach, but I don't have time to give
too many details.  The broad outline is as follows.  We collect datoms
to add and retract in a tx_lookup table.  Depending on flags ("search
value" sv and "search value type tag" svalue_type_tag) we "complete" the
tx_lookup table by joining matching datoms.  This allows us to find
datoms that are present (and should not be added as part of the
transaction, or should be retracted as part of the transaction, or
should be replaced as part of the transaction.  We complete the
tx_lookup (in place!) in two separate INSERTs to avoid a quadratic
two-table walk (explain the queries to observe that both INSERTs walk
the lookup table once and then use the datoms indexes to complete the
matching values).

We could simplify the code by using multiple lookup tables, both for the
two cases of search parameters (eav vs. ea) and for the incomplete and
completed rows.  Right now we differentiate the former with NULL checks,
and the latter by incrementing the added0 column.  It performs well
enough, so I haven't tried to understand the performance of separating
these things.

After the tx_lookup table is completed, we build the transaction from
it; and update the datoms materialized view table as well.  Observe the
careful handling of the "search value" sv parameters to handle replacing
:db.cardinality/one datoms.

Finally, we read the processed transaction back to produce to the API.
This is strictly to match the Datomic API; we might make allow to skip
this, since many consumers will not want to stream this over the wire.

Rough timings show the transactor processing a single >50k datom
transaction in about 3.5s, of which less than 0.5s is spent in the
expensive joins.  Further, repeating the processing of the same
transaction is only about 3.5s again!  That's the worst possible for the
joins, since every single inserted datom will already be present in the
database, making the most expensive join match every row.
2016-08-30 18:22:10 -07:00
Nick Alexander
b59c71cb00 Extract datomish.db.debug from test code, in order to use it during development. 2016-08-30 18:22:10 -07:00
Nick Alexander
a1cc372d43 Fix :db.unique/value, which should be per (a, v) pair, not per v-value. 2016-08-30 18:22:10 -07:00
Nick Alexander
1b1cc6f18e Add Datomic, for testing. 2016-08-30 18:22:09 -07:00
Nick Alexander
b29e5caec0 Implement parts: Make the DB allocate and persist entity IDs.
This implementation is inefficient because each allocated temporary ID
touches the database, but it's enough to allow to re-open DBs.
2016-08-30 18:22:09 -07:00
Richard Newman
22fc2ce201 Define <-tagged-SQLite and tagged-SQLite-to-JS to do tag-aware value transforms. 2016-08-30 18:22:09 -07:00
Nick Alexander
ff9a75ae09 Tag values with value type tags in SQLite. 2016-08-30 18:22:09 -07:00
Nick Alexander
4d34c820b8 Add d/q; make query minimally schema aware. 2016-08-30 18:22:09 -07:00
Nick Alexander
0f7c1cad79 Extract IEncodeSQLite protocol and type-aware (but not schema-aware) <-SQLite factory. 2016-08-30 18:22:09 -07:00
Richard Newman
655a6a1461 Raise a specific error when trying to transact invalid sequences.
This specifically checks for things like :db/add foo bar (nil),
which will otherwise fail elsewhere after being exploded.
2016-08-30 18:22:09 -07:00
Nick Alexander
e7e84e0a90 Add d/{ident,entid} for mapping between keyword idents and integer entids. 2016-08-30 18:22:09 -07:00
Richard Newman
dc87d7d557 Correctly handle SQL errors in <?all-rows. Fixes #40. 2016-08-30 18:22:09 -07:00
Richard Newman
14a6a9bebb Configure our JDBC interface to not downcase column names in result sets. 2016-08-30 18:22:09 -07:00
Richard Newman
9f97cc8f2d Clarify d/<connect. 2016-08-30 18:22:09 -07:00
Richard Newman
cf96bf1888 Always use our sql-quoting-style. 2016-08-30 18:22:09 -07:00
Richard Newman
cff49b6df5 Initialize the sqlite connection with WAL and foreign keys.
This somewhat improves performance, which is nice.
2016-08-30 18:22:09 -07:00
Richard Newman
3ab0428ee0 Add some Leiningen plugins to make testing prettier. 2016-08-30 18:11:25 -07:00
Richard Newman
f4344fa28a Bump JVM stack size to 4MB and eliminate tiered compilation.
This allows CLJS compilation to complete for complex go-pair forms.

See <https://github.com/emezeske/lein-cljsbuild/issues/303> for more details.
2016-08-19 12:11:44 -07:00
Richard Newman
1a54cec1ba Use correct view for fulltext+non-fulltext. 2016-08-05 16:29:21 -07:00
Richard Newman
3f882ebdaa Bump honeysql to 0.8.0, which includes mainline ClojureScript support. 2016-08-05 13:06:42 -07:00
Richard Newman
b1e6ffc91c Example query hacking. 2016-08-05 13:04:09 -07:00
Richard Newman
fc845a9950 Implement basic fulltext binding. r=nalexander 2016-08-05 13:04:06 -07:00
Richard Newman
57d8796d07 Split out datomish.query.cc so we can implement clause resolution in more than one namespace. r=nalexander 2016-08-05 13:04:01 -07:00
Richard Newman
fc65f268fa Process Function instances in clause handling. r=nalexander 2016-08-05 13:03:52 -07:00
Richard Newman
4c2230929e Turn Source into a protocol. Allow source->from to switch on attribute. r=nalexander 2016-08-05 13:03:48 -07:00
Nick Alexander
c948902c52 Follow-up: Extract datomish.transact.explode.
This required pushing the ID literal out of transact.  I elected to put
them in DB, since literal allocation will end up in IDB eventually.
2016-08-04 17:54:50 -07:00
Nick Alexander
d8c976c3ad Follow-up: split the monolith!
This was a little more tricky than might be expected because the
initialization process uses the transactor to bootstrap the database.
Since Clojure doesn't accept mutually recursive modules, this
necessitated a third module, namely "db-factory", which uses both "db"
and "transact".  While I was here, I started an "api" module, to paper
over the potentially complicated internal module structure for external
consumers.  In time, this "api" module may also grow CLJS-specific JS
transformations.
2016-08-04 17:54:32 -07:00
Nick Alexander
1853d57cba Follow-up: extract datomish.transact.bootstrap. 2016-08-04 16:37:14 -07:00