5 Home
Richard Newman edited this page 2017-02-21 10:20:27 -08:00

Welcome to the Project Mentat Wiki!


There are two blog posts that discuss some of the motivations for the project and some of the technical context you might need in order to work on or with Project Mentat.

How it works

At its core, Mentat maintains a set of assertions of the form entity-attribute-value (EAV). The assertions conform to a schema whereby the given attribute constrains the associated value/set of associated values. We call these assertions datoms.

Entities and entids

A Mentat entity is represented by a positive integer. (This agrees with Datomic.) We call such a positive integer an entid.


Datomic partitions the entid space in order to separate core knowledge base entities required for the healthy function of the system from user-defined entities. Datomic also partitions in order to ensure that certain index walks of related entities are efficient. Mentat follows suit, partitioning entids into the following partitions:

  • :db.part/db, for core knowledge base entities;
  • :db.part/user, for user-defined entities;
  • :db.part/tx, for transaction entities.

You almost certainly want to add new entities in the :db.part/user partition.

The entid sequence in a given partition is monotonically increasing, although not necessarily contiguous. That is, it is possible for a specific entid to have never been present in the system, even though its predecessor and successor are present.

Representation of assertions

Mentat assertions are represented as rows in the datoms SQLite table, and each Mentat row representing an assertion is tagged with a numeric representation of :db/valueType.

The tag is used to limit queries, and therefore is placed carefully in the relevant indices to allow searching numeric longs and doubles quickly. The tag is also used to convert SQLite values to the correct Mentat value type on query egress.

The value type tag mapping is currently:

:db/valueType value type tag SQLite storage class examples
:db.type/ref 0 INTEGER 1234
:db.type/boolean 1 INTEGER 0 (false), 1 (true)
:db.type/long 5 INTEGER -4321
:db.type/double 5 REAL -0.369
:db.type/string 10 TEXT arbitrary textual data
:db.type/keyword 13 TEXT :namespaced/keyword

Observe that some Mentat value types share a value type tag: they are differentiated using SQLite's storage class.

Representation as SQL tables

The authoritative table in the SQL store is the transactions table; from its contents, all other tables can be derived. Each assertion in a transaction (see Transacting) is represented as a row in the transactions table, which has columns roughly

e a v value_type_tag added tx

The added column is a boolean flag that is non-0 if the datom was added and 0 if the datom was retracted from the datom store. We index on tx so that we can quickly extract the datoms added or retracted as part of a particular transaction.

The most important table in the SQL store is the datoms table. Queries extract data from the datoms table (see Querying). It is the materialized view of the transactions table, taking into account all transacted additions and retractions. Each asserted assertion is represented as a row in the datoms table, which has columns roughly

e a v value_type_tag tx FLAGS

We (really, SQLite) maintains several indexes and partial indexes to make particular types of queries efficient (at the cost of increasing transaction time and increasing database fragmentation). In particular, we maintain the same set of indexes that Datomic does:

Index Contains
EAVT all datoms
AEVT all datoms
AVET datoms with attributes that have :db/index or :db/unique
VAET datoms with attributes that are :db/type :db.type/ref

The EAVT and AEVT indexes allow to efficiently enumerate entities and attributes, respectively. The AVET index allows to efficiently map attribute-value pairs to matching entities. The VAET index allows to efficiently reverse index.

Representation of metadata as SQL tables

The transactor maintains three metadata tables: idents, schema, and parts. These are materialized views capturing the current state of (the schema part of) the transactions table.

The idents table maintains the set of ident mappings from keyword ident (like :db/ident) to numeric entid (like 1). It looks like

ident entid
:db/ident 1
:db.part/db 2

The schema table maintains the flags and types of the Mentat schema. It looks like

ident attribute v value_type_tag
:db/txInstant :db/cardinality 31 0
:db/txInstant :db/index 1 1
:db/txInstant :db/valueType 25 0

Observe that the value type is represented with a value type tag of 0 (since it's a reference to the entid with ident :db.type/long (in future, :db.type/instant)), but the index is represented with a value type tag of 1 (since it's a boolean). This all may change in future as we make the representation more compact, or make certain operations more efficient.

The parts table maintains the partition ranges and especially the next ID to be allocated in each partition. It looks like

part start idx
:db.part/db 0 38
:db.part/user 65536 65536
:db.part/tx 268435456 268435457

Each transaction that allocates temporary IDs will increment the idx of the appropriate partition. Every transaction allocates a single transaction ID, so the :db.part/tx index should tick up regularly.