Rust backlog

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: greg/mentat#98

gburd commented

2020-08-06 16:54:52 +00:00

(Migrated from github.com)

We're starting on a Rust rewrite of the existing Clojure codebase, using the rust branch.

With an eye to parallelism, the following will get us to parity and a bit further, at which point we'll start on the list of more speculative work.

Large portions of this will be easier than the first time around: the ClojureScript implementation required lots of deep thinking that won't need to be repeated.

Starter Projects:

Learning Rust. This is a totally valid use of time.
[@bgrins] Get Rust builds and tests working in CI.
- Land initial code (https://github.com/mozilla/datomish/pull/134)
- Update readme with build and test instructions (https://github.com/mozilla/datomish/pull/135)
[@jsantell] Begin SQLite wrapper
- Survey of Rust SQLite libraries.
- [@bgrins] dependency
- open a DB at a path and run the default pragmas
- exec/query abstractions
- How do we update and deliver SQLite?
[@joewalker] Build an EDN parser.
[@jsantell] Build a Rust node native module. End goal: call a Rust function from JS.
Build a Rust library for Android. Call it via JNI.
Build a Rust library for iOS. Sign it. Call it from Swift.
[@bgrins] Command-line tool. Right now it wouldn't do much, but we could put the skeleton in place and do something like print user_version for a DB path… https://github.com/mozilla/datomish/pull/136
- Evaluate exposing as a crate within a subfolder
[@bgrins] Figure out an out-of-band test solution for Rust. In-code tests are great, but they're going to get really verbose when we have piles of SQL generation tests. (see https://github.com/mozilla/datomish/tree/rust/tests)
SQLite generator libraries a la honeysql: #273
- Research.
- Import and use.
Add a logger, something like https://github.com/slog-rs/slog

Moving on:

Core (the big blocker for much of the rest):
- Core valueType work: type codes and translation, value conversion.
- Core schema work. Basic Rust type definitions. Traits for translations.
- SQLite wrapper: opener, part reading, etc.
- Bootstrapper.
- Conn + DB.
Querying:
- [@rnewman] Datalog query parser: port from DataScript. #160.
- Parsed query (FindRel) -> abstract query (ConjoiningClauses). Porting datomish.query.
  - This will involve building up much of 'Core', above, brick by brick.
- Abstract query -> SQL.
- Abstract query -> projectors.
- Querying history and logs.
Transactor:
- Transactor expansion: schema + shorthand -> datoms.
- Transactor writer.
- Transactor loop.
- Prepared statements/statement caching.
- Excision.
- noHistory.
Schema management module (porting).
Reader pool (query executor).
- Query interruption via closing connection.
- Parallel reads.
- Connection affinity to exploit sqlite page cache.
- Prepared statements for repeated queries.
  - Invalidation when schema contents change.
  - Handling of query transformation in the presence of bound values, which might change between uses of the prepared statement.
Tooling. We can invest in this as much as we want; it's leverage.
- A command-line query tool, just like sqlite3.
- A REPL that's capable of doing programmatic read/write tasks. JS bindings? Rust REPL?
- A developer tool/embedded server: showing query translations and plans, timing query execution, exploring schema…
- GraphQL interface generation from Datomish schema.
- Bulk loading and import. Make this independent by expanding into a datom stream wrt a given schema.
- Export: textual/EDN, RDF?
Transaction listeners over a socket.
- Log dumper standalone tool. Watch writes happening as they occur.
Syncing:
- Transaction log replication (out) and replay (in).
  - Code and algorithms exist.
  - Define how IDs are mapped in and out, and how those might be efficiently represented in chunks.
  - Handle application of noHistory attributes.
  - Handle remote excision.
- Multi-master transaction log handling (syncing).

We're starting on a Rust rewrite of the existing Clojure codebase, using the `rust` branch. With an eye to parallelism, the following will get us to parity and a bit further, at which point we'll start on the list of more speculative work. Large portions of this will be easier than the first time around: the ClojureScript implementation required lots of deep thinking that won't need to be repeated. **Starter Projects**: - [x] Learning Rust. This is a totally valid use of time. - [x] [@bgrins] Get Rust builds and tests working in CI. - [x] Land initial code (https://github.com/mozilla/datomish/pull/134) - [x] Update readme with build and test instructions (https://github.com/mozilla/datomish/pull/135) - [x] [@jsantell] Begin SQLite wrapper - [x] Survey of Rust SQLite libraries. - [x] [@bgrins] dependency - [x] open a DB at a path and run the default pragmas - [x] exec/query abstractions - [x] How do we update and deliver SQLite? - [x] [@joewalker] Build an EDN parser. - [ ] [@jsantell] Build a Rust node native module. End goal: call a Rust function from JS. - [x] Build a Rust library for Android. Call it via JNI. - [x] Build a Rust library for iOS. Sign it. Call it from Swift. - [x] [@bgrins] Command-line tool. Right now it wouldn't do much, but we could put the skeleton in place and do something like print user_version for a DB path… https://github.com/mozilla/datomish/pull/136 - [x] Evaluate exposing as a crate within a subfolder - [x] [@bgrins] Figure out an out-of-band test solution for Rust. In-code tests are great, but they're going to get really verbose when we have piles of SQL generation tests. (see https://github.com/mozilla/datomish/tree/rust/tests) - [x] SQLite generator libraries _a la_ honeysql: #273 - [x] Research. - [x] Import and use. - [ ] Add a logger, something like https://github.com/slog-rs/slog **Moving on**: - [x] Core (the big blocker for much of the rest): - [x] Core valueType work: type codes and translation, value conversion. - [x] Core schema work. Basic Rust type definitions. Traits for translations. - [x] SQLite wrapper: opener, part reading, etc. - [x] Bootstrapper. - [x] Conn + DB. - [ ] Querying: - [x] [@rnewman] Datalog query parser: port from DataScript. #160. - [x] Parsed query (FindRel) -> abstract query (ConjoiningClauses). Porting datomish.query. - This will involve building up much of 'Core', above, brick by brick. - [x] Abstract query -> SQL. - [x] Abstract query -> projectors. - [ ] Querying history and logs. - [ ] Transactor: - [x] Transactor expansion: schema + shorthand -> datoms. - [x] Transactor writer. - [ ] Transactor loop. - [x] Prepared statements/statement caching. - [ ] Excision. - [ ] noHistory. - [x] Schema management module (porting). - [ ] Reader pool (query executor). - [ ] Query interruption via closing connection. - [ ] Parallel reads. - [ ] Connection affinity to exploit sqlite page cache. - [ ] Prepared statements for repeated queries. - [ ] Invalidation when schema contents change. - [ ] Handling of query transformation in the presence of bound values, which might change between uses of the prepared statement. - [ ] Tooling. We can invest in this as much as we want; it's leverage. - [x] A command-line query tool, just like sqlite3. - [ ] A REPL that's capable of doing programmatic read/write tasks. JS bindings? Rust REPL? - [ ] A developer tool/embedded server: showing query translations and plans, timing query execution, exploring schema… - [ ] GraphQL interface generation from Datomish schema. - [x] Bulk loading and import. Make this independent by expanding into a datom stream wrt a given schema. - [ ] Export: textual/EDN, RDF? - [ ] Transaction listeners over a socket. - [ ] Log dumper standalone tool. Watch writes happening as they occur. - [ ] Syncing: - [ ] Transaction log replication (out) and replay (in). - [ ] Code and algorithms exist. - [ ] Define how IDs are mapped in and out, and how those might be efficiently represented in chunks. - [ ] Handle application of noHistory attributes. - [ ] Handle remote excision. - [ ] Multi-master transaction log handling (syncing).

Rust backlog #98