Multiprocess support #184

New issue

Open

opened 2020-08-06 16:56:20 +00:00 by gburd · 0 comments

gburd commented

2020-08-06 16:56:20 +00:00

(Migrated from github.com)

I thought I'd write a quick brain-dump about how I could imagine two separate processes collaborating on a single Mentat store.

Firstly, the basics: NFS share workarounds and such. That's relatively well understood.

Secondly, SQLite: SQLite is happy with two processes sharing a database, but if only one is allowed to write, the other must be prepared for its attempts to open a transaction to fail.

Thirdly, Mentat itself: we store schema and part information in memory. We must be prepared to establish a transaction around every read or write (or be prepared to re-do work if we keep our critical section small and the schema changes before we enter it), and within that transaction quickly check or revise our assumptions: e.g., refreshing the parts table or checking the latest transaction ID.

We should be able to make this relatively efficient — we already store a generation number in memory, and the work being done on syncing will increase the amount of metadata we store on disk, so there's a natural place to track changes.

You could imagine this as being equivalent to re-opening the Conn and rebuilding its metadata each time we start a read or write.

In reality the schema is fairly stable, so the only collisions are likely to be in transacting, which is already an exclusive operation.

I thought I'd write a quick brain-dump about how I could imagine two separate processes collaborating on a single Mentat store. Firstly, the basics: NFS share workarounds and such. That's relatively well understood. Secondly, SQLite: SQLite is happy with two processes sharing a database, but if only one is allowed to write, the other must be prepared for its attempts to open a transaction to fail. Thirdly, Mentat itself: we store schema and part information in memory. We must be prepared to establish a transaction around every read or write (or be prepared to re-do work if we keep our critical section small and the schema changes before we enter it), and within that transaction quickly check or revise our assumptions: _e.g._, refreshing the parts table or checking the latest transaction ID. We should be able to make this relatively efficient — we already store a generation number in memory, and the work being done on syncing will increase the amount of metadata we store on disk, so there's a natural place to track changes. You could imagine this as being equivalent to re-opening the `Conn` and rebuilding its metadata each time we start a read or write. In reality the schema is fairly stable, so the only collisions are likely to be in transacting, which is already an exclusive operation.