greg/mentat
1
0
Fork 0
Code Issues 294 Pull requests Projects Releases 9 Packages Wiki Activity

Rename everything to Project Mentat. r=bgrins

Browse source
This commit is contained in:
Richard Newman 2017-01-09 09:34:10 -08:00
commit 22ebcd65f3
25 changed files with 44 additions and 367 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
.travis.yml
View file

@ -2,5 +2,5 @@ language: rust
script:
- cargo build --verbose
- cargo test --verbose
- cargo test --verbose -p datomish-query-parser
- cargo test --verbose -p datomish-cli
- cargo test --verbose -p mentat_query_parser
- cargo test --verbose -p mentat_cli

8
CONTRIBUTING.md
View file

@ -1,4 +1,4 @@
# How to contribute to Datomish
# How to contribute to Project Mentat
This project is very new, so we'll probably revise these guidelines. Please
comment on a bug before putting significant effort in, if you'd like to
@ -29,10 +29,10 @@ Signed-off-by: Random J Developer <random@developer.example.org>
## Example
* Fork this repo at [github.com/mozilla/datomish](https://github.com/mozilla/datomish#fork-destination-box).
* Fork this repo at [github.com/mozilla/mentat](https://github.com/mozilla/mentat#fork-destination-box).
* Clone your fork locally. Make sure you use the correct clone URL.
```
git clone git@github.com:YOURNAME/datomish.git
git clone git@github.com:YOURNAME/mentat.git
```
Check your remotes:
```
@ -40,7 +40,7 @@ git remote --verbose
```
Make sure you have an upstream remote defined:
```
git remote add upstream https://github.com/mozilla/datomish
git remote add upstream https://github.com/mozilla/mentat
```
* Create a new branch to start working on a bug or feature:

10
Cargo.toml
View file

@ -1,18 +1,18 @@
[package]
name = "datomish"
name = "mentat"
version = "0.4.0"
authors = ["Richard Newman <rnewman@twinql.com>", "Nicholas Alexander <nalexander@mozilla.com>"]
[dependencies]
rusqlite = "0.8.0"
[dependencies.datomish-query-parser]
[dependencies.mentat_query_parser]
path = "query-parser"
[dev-dependencies]
[dev-dependencies.datomish-cli]
[dev-dependencies.mentat_cli]
path = "cli"
[[bin]]
name = "datomish-cli"
path = "cli/src/main.rs"
name = "mentat_cli"
path = "cli/src/main.rs"

30
README.md
View file

@ -1,15 +1,15 @@
# Datomish
# Project Mentat
Datomish is a persistent, embedded knowledge base. It draws heavily on [DataScript](https://github.com/tonsky/datascript) and [Datomic](http://datomic.com).
Project Mentat is a persistent, embedded knowledge base. It draws heavily on [DataScript](https://github.com/tonsky/datascript) and [Datomic](http://datomic.com).
The first version of Datomish [was written in ClojureScript](https://github.com/mozilla/datomish/tree/master), targeting both Node (on top of `promise_sqlite`) and Firefox (on top of `Sqlite.jsm`). It also works in pure Clojure on the JVM on top of `jdbc-sqlite`.
The first version of Project Mentat, named Datomish, [was written in ClojureScript](https://github.com/mozilla/mentat/tree/master), targeting both Node (on top of `promise_sqlite`) and Firefox (on top of `Sqlite.jsm`). It also works in pure Clojure on the JVM on top of `jdbc-sqlite`. The name was changed to avoid confusion with [Datomic](http://datomic.com).
This branch is for rewriting Datomish in Rust, giving us a smaller compiled output, better performance, more type safety, better tooling, and easier deployment into Firefox and mobile platforms.
This branch is for rewriting Mentat in Rust, giving us a smaller compiled output, better performance, more type safety, better tooling, and easier deployment into Firefox and mobile platforms.
## Motivation
Datomish is intended to be a flexible relational (not key-value, not document-oriented) store that doesn't leak its storage schema to users, and doesn't make it hard to grow its domain schema and run arbitrary queries.
Mentat is intended to be a flexible relational (not key-value, not document-oriented) store that doesn't leak its storage schema to users, and doesn't make it hard to grow its domain schema and run arbitrary queries.
Our short-term goal is to build a system that, as the basis for a User Agent Service, can support multiple [Tofino](https://github.com/mozilla/tofino) UX experiments without having a storage engineer do significant data migration, schema work, or revving of special-purpose endpoints.
@ -20,9 +20,9 @@ By abstracting away the storage schema, and by exposing change listeners outside
DataScript asks the question: "What if creating a database would be as cheap as creating a Hashmap?"
Datomish is not interested in that. Instead, it's strongly interested in persistence and performance, with very little interest in immutable databases/databases as values or throwaway use.
Mentat is not interested in that. Instead, it's strongly interested in persistence and performance, with very little interest in immutable databases/databases as values or throwaway use.
One might say that Datomish's question is: "What if an SQLite database could store arbitrary relations, for arbitrary consumers, without them having to coordinate an up-front storage-level schema?"
One might say that Mentat's question is: "What if an SQLite database could store arbitrary relations, for arbitrary consumers, without them having to coordinate an up-front storage-level schema?"
(Note that [domain-level schemas are very valuable](http://martinfowler.com/articles/schemaless/).)
@ -30,9 +30,9 @@ Another possible question would be: "What if we could bake some of the concepts
Some thought has been given to how databases as values — long-term references to a snapshot of the store at an instant in time — could work in this model. It's not impossible; it simply has different performance characteristics.
Just like DataScript, Datomish speaks Datalog for querying and takes additions and retractions as input to a transaction. Unlike DataScript, Datomish's API is asynchronous.
Just like DataScript, Mentat speaks Datalog for querying and takes additions and retractions as input to a transaction. Unlike DataScript, Mentat's API is asynchronous.
Unlike DataScript, Datomish exposes free-text indexing, thanks to SQLite.
Unlike DataScript, Mentat exposes free-text indexing, thanks to SQLite.
## Comparison to Datomic
@ -41,14 +41,14 @@ Datomic is a server-side, enterprise-grade data storage system. Datomic has a be
Many of these design decisions are inapplicable to deployed desktop software; indeed, the use of multiple JVM processes makes Datomic's use in a small desktop app, or a mobile device, prohibitive.
Datomish is designed for embedding, initially in an Electron app ([Tofino](https://github.com/mozilla/tofino)). It is less concerned with exposing consistent database states outside transaction boundaries, because that's less important here, and dropping some of these requirements allows us to leverage SQLite itself.
Mentat is designed for embedding, initially in an Electron app ([Tofino](https://github.com/mozilla/tofino)). It is less concerned with exposing consistent database states outside transaction boundaries, because that's less important here, and dropping some of these requirements allows us to leverage SQLite itself.
## Comparison to SQLite
SQLite is a traditional SQL database in most respects: schemas conflate semantic, structural, and datatype concerns; the main interface with the database is human-first textual queries; sparse and graph-structured data are 'unnatural', if not always inefficient; experimenting with and evolving data models are error-prone and complicated activities; and so on.
Datomish aims to offer many of the advantages of SQLite — single-file use, embeddability, and good performance — while building a more relaxed and expressive data model on top.
Mentat aims to offer many of the advantages of SQLite — single-file use, embeddability, and good performance — while building a more relaxed and expressive data model on top.
## Contributing
@ -78,7 +78,7 @@ To run tests use:
cargo test
# Run tests for the query-parser folder
cargo test -p datomish-query-parser
cargo test -p mentat_query_parser
````
To start the cli use:
@ -87,15 +87,15 @@ To start the cli use:
cargo run
````
For most `cargo` commands you can pass the `-p` argument to run the command just on that package. By convention, the package name will be "datomish-directory-name". So, `cargo build -p datomish-cli` will build just the "cli" folder.
For most `cargo` commands you can pass the `-p` argument to run the command just on that package. By convention, the package name will be "mentat_package_name". So, `cargo build -p mentat_cli` will build just the "cli" folder.
## License
Datomish is currently licensed under the Apache License v2.0. See the `LICENSE` file for details.
Project Mentat is currently licensed under the Apache License v2.0. See the `LICENSE` file for details.
## SQLite dependencies
Datomish uses partial indices, which are available in SQLite 3.8.0 and higher.
Mentat uses partial indices, which are available in SQLite 3.8.0 and higher.
It also uses FTS4, which is [a compile time option](http://www.sqlite.org/fts3.html#section_2).

6
cli/Cargo.toml
View file

@ -1,7 +1,7 @@
[package]
name = "datomish-cli"
name = "mentat_cli"
version = "0.0.1"
[dependencies]
[dependencies.datomish]
path = "../"
[dependencies.mentat]
path = "../"

2
cli/README.md
View file

@ -1,3 +1,3 @@
# datomish-cli
# mentat-cli
Note: this isn't actually doing anything and is just a placeholder to get the project structure in place.

4
cli/src/main.rs
View file

@ -9,11 +9,11 @@
// specific language governing permissions and limitations under the License.
use std::env;
extern crate datomish;
extern crate mentat;
// This is just a placeholder to get the project structure in place.
fn main() {
println!("Loaded {}", datomish::get_name());
println!("Loaded {}", mentat::get_name());
let args: Vec<String> = env::args().collect();
println!("I got {:?} arguments: {:?}.", args.len() - 1, &args[1..]);

27
js/.babelrc
View file

@ -1,27 +0,0 @@
{
"env": {
"production": {
"presets": ["react", "react-optimize"]
},
"development": {
"presets": ["react"]
},
"test": {
"presets": ["react"]
}
},
"only": [
"test/js/**"
],
"plugins": [
"transform-es2015-destructuring",
"transform-es2015-parameters",
"transform-es2015-modules-commonjs",
"transform-async-to-generator",
"transform-object-rest-spread",
"transform-class-properties",
"transform-runtime"
],
"sourceMaps": "inline",
"retainLines": true
}

6
js/addon/.babelrc
View file

@ -1,6 +0,0 @@
{
"presets": ["es2015"],
"plugins": [
"transform-async-to-generator"
]
}

3
js/addon/CREDITS
View file

@ -1,3 +0,0 @@
Icon file is "Line Graph" by Cris Dobbins, from The Noun Project.
https://thenounproject.com/term/line-graph/145324/

3
js/addon/build.sh
View file

@ -1,3 +0,0 @@
cp ../target/release-browser/datomish.js release/
node_modules/.bin/webpack -p
cat src/wrapper.prefix built/index.js > release/index.js

23
js/addon/package.json
View file

@ -1,23 +0,0 @@
{
"name": "datomish-example",
"version": "1.0.0",
"description": "A test add-on for Datomish and Firefox.",
"main": "index.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"author": "",
"license": "MPL-2.0",
"devDependencies": {
"babel": "^6.5.2",
"babel-cli": "^6.14.0",
"babel-core": "^6.14.0",
"babel-loader": "^6.2.5",
"babel-plugin-transform-async-to-generator": "^6.8.0",
"babel-preset-es2015": "^6.14.0",
"webpack": "^1.13.2"
},
"dependencies": {
"babel-polyfill": "^6.13.0"
}
}

2
js/addon/release/README.md
View file

@ -1,2 +0,0 @@
#Datomish Test
An example add-on that loads Datomish on top of Sqlite.jsm.

BIN
js/addon/release/data/datomish-48.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 1.1 KiB

15
js/addon/release/package.json
View file

@ -1,15 +0,0 @@
{
"title": "Datomish Test",
"name": "datomish-test",
"version": "0.0.1",
"description": "An example add-on that loads Datomish on top of Sqlite.jsm.",
"main": "index.js",
"author": "Richard Newman <rnewman@mozilla.com>",
"engines": {
"firefox": ">=48.0a1"
},
"license": "MPL-2.0",
"keywords": [
"jetpack"
]
}

1
js/addon/release/run.sh
View file

@ -1 +0,0 @@
jpm run -b /Applications/FirefoxNightly.app/

92
js/addon/src/index.js
View file

@ -1,92 +0,0 @@
/* Any copyright is dedicated to the Public Domain.
http://creativecommons.org/publicdomain/zero/1.0/ */
var self = require("sdk/self");
var buttons = require('sdk/ui/button/action');
var tabs = require('sdk/tabs');
var datomish = require("datomish.js");
var schema = {
"name": "pages",
"attributes": [
{"name": "page/url",
"type": "string",
"cardinality": "one",
"unique": "identity",
"doc": "A page's URL."},
{"name": "page/title",
"type": "string",
"cardinality": "one",
"fulltext": true,
"doc": "A page's title."},
{"name": "page/content",
"type": "string",
"cardinality": "one", // Simple for now.
"fulltext": true,
"doc": "A snapshot of the page's content. Should be plain text."},
]
};
async function initDB(path) {
let db = await datomish.open(path);
await db.ensureSchema(schema);
return db;
}
async function findURLs(db) {
let query = `[:find ?page ?url ?title :in $ :where [?page :page/url ?url][(get-else $ ?page :page/title "") ?title]]`;
let options = new Object();
options["limit"] = 10;
return datomish.q(db.db(), query, options);
}
async function findPagesMatching(db, string) {
let query =
`[:find ?url ?title
:in $ ?str
:where
[(fulltext $ :any ?str) [[?page]]]
[?page :page/url ?url]
[(get-else $ ?page :page/title "") ?title]]`;
return datomish.q(db.db(), query, {"limit": 10, "inputs": {"str": string}});
}
async function savePage(db, url, title, content) {
let datom = {"db/id": 55, "page/url": url};
if (title) {
datom["page/title"] = title;
}
if (content) {
datom["page/content"] = content;
}
let txResult = await db.transact([datom]);
return txResult;
}
async function handleClick(state) {
let db = await datomish.open("/tmp/testing.db");
await db.ensureSchema(schema);
let txResult = await savePage(db, tabs.activeTab.url, tabs.activeTab.title, "Content goes here");
console.log("Transaction returned " + JSON.stringify(txResult));
console.log("Transaction instant: " + txResult.txInstant);
let results = await findURLs(db);
results = results.map(r => r[1]);
console.log("Query results: " + JSON.stringify(results));
let pages = await findPagesMatching(db, "goes");
console.log("Pages: " + JSON.stringify(pages));
await db.close();
}
var button = buttons.ActionButton({
id: "datomish-save",
label: "Save Page",
icon: "./datomish-48.png",
onClick: handleClick
});

4
js/addon/src/wrapper.prefix
View file

@ -1,4 +0,0 @@
// Monkeypatch.
var { setTimeout } = require("sdk/timers");
this.setTimeout = setTimeout;

21
js/addon/webpack.config.js
View file

@ -1,21 +0,0 @@
module.exports = {
entry: ['babel-polyfill', './src/index.js'],
output: {
filename: 'built/index.js'
},
target: 'webworker',
externals: {
'datomish.js': 'commonjs datomish.js',
'sdk/self': 'commonjs sdk/self',
'sdk/ui/button/action': 'commonjs sdk/ui/button/action',
'sdk/tabs': 'commonjs sdk/tabs'
},
module: {
loaders: [{
test: /\.js?$/,
exclude: /(node_modules)|(wrapper.prefix)/,
loader: 'babel'
}]
}
}

126
js/test/js/tests.js
View file

@ -1,126 +0,0 @@
/* Copyright 2016 Mozilla
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use
* this file except in compliance with the License. You may obtain a copy of the
* License at http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software distributed
* under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
* CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*/
var datomish = require("../../target/release-node/datomish.js");
var schema = {
"name": "pages",
"attributes": [
{"name": "page/url",
"type": "string",
"cardinality": "one",
"unique": "identity",
"doc": "A page's URL."},
{"name": "page/title",
"type": "string",
"cardinality": "one",
"doc": "A page's title."}
]
};
async function testOpen() {
// Open a database.
let path = "/tmp/testing" + Date.now() + ".db";
console.log("Opening " + path);
let db = await datomish.open(path);
// Make sure we have our current schema.
await db.ensureSchema(schema);
// Add some data. Note that we use a temporary ID (the real ID
// will be assigned by Datomish).
let txResult = await db.transact([
{"db/id": datomish.tempid(),
"page/url": "https://mozilla.org/",
"page/title": "Mozilla"}
]);
console.log("Transaction returned " + JSON.stringify(txResult));
console.log("Transaction instant: " + txResult.txInstant);
// A simple query.
let results = await db.q("[:find [?url ...] :in $ :where [?e :page/url ?url]]");
console.log("Known URLs: " + JSON.stringify(results));
// Let's extend our schema. In the real world this would typically happen
// across releases.
schema.attributes.push({"name": "page/visitedAt",
"type": "instant",
"cardinality": "many",
"doc": "A visit to the page."});
await db.ensureSchema(schema);
// Now we can make assertions with the new vocabulary about existing
// entities.
// Note that we simply let Datomish find which page we're talking about by
// URL -- the URL is a unique property -- so we just use a tempid again.
await db.transact([
{"db/id": datomish.tempid(),
"page/url": "https://mozilla.org/",
"page/visitedAt": new Date()}
]);
// When did we most recently visit this page?
let date = (await db.q(
`[:find (max ?date) .
:in $ ?url
:where
[?page :page/url ?url]
[?page :page/visitedAt ?date]]`,
{"inputs": {"url": "https://mozilla.org/"}}));
console.log("Most recent visit: " + date);
// Add some more data about a couple of pages.
let start = Date.now();
let lr = datomish.tempid();
let reddit = datomish.tempid();
let res = await db.transact([
{"db/id": reddit,
"page/url": "http://reddit.com/",
"page/title": "Reddit",
"page/visitedAt": new Date(start)},
{"db/id": lr,
"page/url": "https://longreads.com/",
"page/title": "Longreads: The best longform stories on the web",
"page/visitedAt": (new Date(start + 100))},
// Two visits each.
{"db/id": lr,
"page/visitedAt": (new Date(start + 200))},
{"db/id": reddit,
"page/visitedAt": (new Date(start + 300))}
]);
// These are our new persistent IDs. We can use these directly in later
// queries or transactions
lr = res.tempid(lr);
reddit = res.tempid(reddit);
console.log("Persistent IDs are " + lr + ", " + reddit + ".");
// A query with a limit and order-by. Because we limit to 2, and order
// by most recent visit date first, we won't get mozilla.org in our results.
let recent = await db.q(
`[:find ?url (max ?date)
:in $
:where
[?page :page/url ?url]
[?page :page/visitedAt ?date]]`,
{"limit": 2, "order-by": [["_max_date", "desc"]]});
console.log("Recently visited: " + JSON.stringify(recent));
// Close: we're done!
await db.close();
}
testOpen()
.then((r) => console.log("Done."))
.catch((e) => console.log("Failure: " + e.stack));

BIN
js/test/v1.db
View file

Binary file not shown.

4
query-parser/Cargo.toml
View file

@ -1,3 +1,3 @@
[package]
name = "datomish-query-parser"
version = "0.0.1"
name = "mentat_query_parser"
version = "0.0.1"

6
query-parser/src/lib.rs
View file

@ -10,7 +10,7 @@
// This file is just a stub
pub fn get_name() -> String {
return String::from("datomish-query-parser");
return String::from("mentat-query-parser");
}
#[cfg(test)]
@ -19,6 +19,6 @@ mod tests {
#[test]
fn it_works() {
assert_eq!(String::from("datomish-query-parser"), get_name());
assert_eq!(String::from("mentat-query-parser"), get_name());
}
}
}

10
src/lib.rs
View file

@ -8,18 +8,18 @@
// CONDITIONS OF ANY KIND, either express or implied. See the License for the
// specific language governing permissions and limitations under the License.
extern crate datomish_query_parser;
extern crate mentat_query_parser;
extern crate rusqlite;
use rusqlite::Connection;
pub fn get_name() -> String {
return String::from("datomish");
return String::from("mentat");
}
// Just an example of using a dependency
pub fn get_parser_name() -> String {
return datomish_query_parser::get_name();
return mentat_query_parser::get_name();
}
// Will ultimately not return the sqlite connection directly
@ -33,6 +33,6 @@ mod tests {
#[test]
fn can_import_parser() {
assert_eq!(String::from("datomish-query-parser"), get_parser_name());
assert_eq!(String::from("mentat-query-parser"), get_parser_name());
}
}
}

4
tests/external_test.rs
View file

@ -8,7 +8,7 @@
// CONDITIONS OF ANY KIND, either express or implied. See the License for the
// specific language governing permissions and limitations under the License.
extern crate datomish;
extern crate mentat;
#[test]
fn can_import_sqlite() {
@ -20,7 +20,7 @@ fn can_import_sqlite() {
data: Option<Vec<u8>>
}
let conn = datomish::get_connection();
let conn = mentat::get_connection();
conn.execute("CREATE TABLE person (
id INTEGER PRIMARY KEY,