mentat/README.md

186 lines
6.9 KiB
Markdown
Raw Normal View History

# Datomish
Datomish is a persistent, embedded knowledge base. It's written in ClojureScript, and draws heavily on [DataScript](https://github.com/tonsky/datascript) and [Datomic](http://datomic.com).
2016-09-22 23:45:52 +00:00
Datomish compiles into a single JavaScript file, and is usable both in Node (on top of `promise_sqlite`) and in Firefox (on top of `Sqlite.jsm`). It also works in pure Clojure on the JVM on top of `jdbc-sqlite`.
## Motivation
Datomish is intended to be a flexible relational (not key-value, not document-oriented) store that doesn't leak its storage schema to users, and doesn't make it hard to grow its domain schema and run arbitrary queries.
Our short-term goal is to build a system that, as the basis for a User Agent Service, can support multiple [Tofino](https://github.com/mozilla/tofino) UX experiments without having a storage engineer do significant data migration, schema work, or revving of special-purpose endpoints.
## Comparison to DataScript
DataScript asks the question: "What if creating a database would be as cheap as creating a Hashmap?"
Datomish is not interested in that. Instead, it's strongly interested in persistence and performance, with very little interest in immutable databases/databases as values or throwaway use.
One might say that Datomish's question is: "What if an SQLite database could store arbitrary relations, for arbitrary consumers, without them having to coordinate an up-front storage-level schema?"
(Note that [domain-level schemas are very valuable](http://martinfowler.com/articles/schemaless/).)
Another possible question would be: "What if we could bake some of the concepts of CQRS and event sourcing into a persistent relational store, such that the transaction log itself were of value to queries?"
Some thought has been given to how databases as values — long-term references to a snapshot of the store at an instant in time — could work in this model. It's not impossible; it simply has different performance characteristics.
Just like DataScript, Datomish speaks Datalog for querying and takes additions and retractions as input to a transaction. Unlike DataScript, Datomish's API is asynchronous.
Unlike DataScript, Datomish exposes free-text indexing, thanks to SQLite.
## Comparison to Datomic
Datomic is a server-side, enterprise-grade data storage system. Datomic has a beautiful conceptual model. It's intended to be backed by a storage cluster, in which it keeps index chunks forever. Index chunks are replicated to peers, allowing it to run queries at the edges. Writes are serialized through a transactor.
Many of these design decisions are inapplicable to deployed desktop software; indeed, the use of multiple JVM processes makes Datomic's use in a small desktop app, or a mobile device, prohibitive.
Datomish is designed for embedding, initially in an Electron app ([Tofino](https://github.com/mozilla/tofino)). It is less concerned with exposing consistent database states outside transaction boundaries, because that's less important here, and dropping some of these requirements allows us to leverage SQLite itself.
## Comparison to SQLite
SQLite is a traditional SQL database in most respects: schemas conflate semantic, structural, and datatype concerns; the main interface with the database is human-first textual queries; sparse and graph-structured data are 'unnatural', if not always inefficient; experimenting with and evolving data models are error-prone and complicated activities; and so on.
Datomish aims to offer many of the advantages of SQLite — single-file use, embeddability, and good performance — while building a more relaxed and expressive data model on top.
## Contributing
Please note that this project is released with a Contributor Code of Conduct.
By participating in this project you agree to abide by its terms.
See [CONTRIBUTING.md](/CONTRIBUTING.md) for further notes.
This project is very new, so we'll probably revise these guidelines. Please
comment on a bug before putting significant effort in if you'd like to
contribute.
## License
At present this code is licensed under MPLv2.0. That license is subject to change prior to external contributions.
2016-09-22 23:45:52 +00:00
Datomish includes some code from DataScript, kindly relicensed by Nikita Prokopov.
## SQLite dependencies
Datomish uses partial indices, which are available in SQLite 3.8.0 and higher.
It also uses FTS4, which is [a compile time option](http://www.sqlite.org/fts3.html#section_2).
2016-09-22 23:45:52 +00:00
## Prep
You'll need [Leiningen](http://leiningen.org).
```
# If you use nvm.
nvm use 6
lein deps
npm install
# If you want a decent REPL.
brew install rlwrap
```
2016-09-22 23:45:52 +00:00
## Running a REPL
2016-09-22 17:38:45 +00:00
### Starting a ClojureScript REPL from the terminal
```
rlwrap lein run -m clojure.main repl.clj
```
### Connecting to a ClojureScript environment from Vim
You'll need `vim-fireplace`. Install using Pathogen.
First, start a Clojure REPL with an nREPL server. Then load our ClojureScript REPL and dependencies. Finally, connect to it from Vim.
```
$ lein repl
nREPL server started on port 62385 on host 127.0.0.1 - nrepl://127.0.0.1:62385
REPL-y 0.3.7, nREPL 0.2.10
Clojure 1.8.0
Java HotSpot(TM) 64-Bit Server VM 1.8.0_60-b27
Docs: (doc function-name-here)
(find-doc "part-of-name-here")
Source: (source function-name-here)
Javadoc: (javadoc java-object-or-class-here)
Exit: Control+D or (exit) or (quit)
Results: Stored in vars *1, *2, *3, an exception in *e
user=> (load-file "repl.clj")
Reading analysis cache for jar:file:/Users/rnewman/.m2/repository/org/clojure/clojurescript/1.9.89/clojurescript-1.9.89.jar!/cljs/core.cljs
Compiling out/cljs/nodejs.cljs
Compiling src/datomish/sqlite.cljs
Compiling src/datomish/core.cljs
ClojureScript Node.js REPL server listening on 57134
Watch compilation log available at: out/watch.log
To quit, type: :cljs/quit
cljs.user=>
```
in Vim, in the working directory:
```
:Piggieback (cljs.repl.node/repl-env)
```
Now you can use `:Eval`, `cqc`, and friends to evaluate code. Fireplace should connect automatically, but if it doesn't:
```
:Connect nrepl://localhost:62385
```
## To run tests in ClojureScript
Run `lein doo node test once`, or `lein doo node` to re-run on file changes.
## To run tests in Clojure
Run `lein test`.
## To run smoketests with the built release library in a Node environment
```
# Build.
lein cljsbuild once release-node
npm run test
```
2016-09-22 23:45:52 +00:00
## To build for Firefox
```
lein cljsbuild once release-browser
```
### To build and run the example Firefox add-on:
2016-09-22 23:45:52 +00:00
```
cd addon
./build.sh
cd release
./run.sh
```
2016-09-22 23:45:52 +00:00
## Preparing an NPM release
The intention is that the `target/release-node/` directory is roughly the shape of an npm-ready JavaScript package.
To generate a require/import-ready `target/release-node/datomish.js`, run
```
2016-09-22 23:45:52 +00:00
lein cljsbuild once release-node
```
To verify that importing into Node.js succeeds, run
```
2016-09-22 23:45:52 +00:00
npm run test
```
Many thanks to ([David Nolen](https://github.com/swannodette)) and ([Nikita Prokopov](https://github.com/tonsky)) for demonstrating how to package ClojureScript for distribution via npm.