mirror of
https://github.com/berkeleydb/libdb.git
synced 2024-11-16 17:16:25 +00:00
548 lines
22 KiB
HTML
548 lines
22 KiB
HTML
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
|
||
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
|
||
<html xmlns="http://www.w3.org/1999/xhtml">
|
||
<head>
|
||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
|
||
<title>In-Memory Transaction Example</title>
|
||
<link rel="stylesheet" href="gettingStarted.css" type="text/css" />
|
||
<meta name="generator" content="DocBook XSL Stylesheets V1.73.2" />
|
||
<link rel="start" href="index.html" title="Getting Started with Berkeley DB Transaction Processing" />
|
||
<link rel="up" href="wrapup.html" title="Chapter 6. Summary and Examples" />
|
||
<link rel="prev" href="txnexample_c.html" title="Transaction Example" />
|
||
</head>
|
||
<body>
|
||
<div xmlns="" class="navheader">
|
||
<div class="libver">
|
||
<p>Library Version 11.2.5.3</p>
|
||
</div>
|
||
<table width="100%" summary="Navigation header">
|
||
<tr>
|
||
<th colspan="3" align="center">In-Memory Transaction Example</th>
|
||
</tr>
|
||
<tr>
|
||
<td width="20%" align="left"><a accesskey="p" href="txnexample_c.html">Prev</a> </td>
|
||
<th width="60%" align="center">Chapter 6. Summary and Examples</th>
|
||
<td width="20%" align="right"> </td>
|
||
</tr>
|
||
</table>
|
||
<hr />
|
||
</div>
|
||
<div class="sect1" lang="en" xml:lang="en">
|
||
<div class="titlepage">
|
||
<div>
|
||
<div>
|
||
<h2 class="title" style="clear: both"><a id="inmem_txnexample_c"></a>In-Memory Transaction Example</h2>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<p>
|
||
DB is sometimes used for applications that simply need to cache
|
||
data retrieved from some other location (such as a remote database
|
||
server). DB is also often used in embedded systems.
|
||
</p>
|
||
<p>
|
||
In both cases, applications may want to use transactions for
|
||
atomicity, consistency, and isolation guarantees, but they may also want
|
||
to forgo the durability guarantee entirely. In doing so, they can keep
|
||
their DB environment and databases entirely in-memory so
|
||
as to avoid the performance impact of unneeded disk I/O.
|
||
</p>
|
||
<p>
|
||
To do this:
|
||
</p>
|
||
<div class="itemizedlist">
|
||
<ul type="disc">
|
||
<li>
|
||
<p>
|
||
Refrain from specifying a home directory when you open your
|
||
environment. The exception to this is if you are using the
|
||
<code class="literal">DB_CONFIG</code> configuration file — in
|
||
that case you must identify the environment's home
|
||
directory so that the configuration file can be found.
|
||
</p>
|
||
</li>
|
||
<li>
|
||
<p>
|
||
Configure your environment to back your regions from
|
||
system memory instead of the filesystem.
|
||
</p>
|
||
</li>
|
||
<li>
|
||
<p>
|
||
Configure your logging subsystem such that log files are kept
|
||
entirely in-memory.
|
||
</p>
|
||
</li>
|
||
<li>
|
||
<p>
|
||
Increase the size of your in-memory log buffer so that it
|
||
is large enough to hold the largest set of concurrent write operations.
|
||
</p>
|
||
</li>
|
||
<li>
|
||
<p>
|
||
Increase the size of your in-memory cache so that it can
|
||
hold your entire data set. You do not want your cache to
|
||
page to disk.
|
||
</p>
|
||
</li>
|
||
<li>
|
||
<p>
|
||
Do not specify a file name when you open your database(s).
|
||
</p>
|
||
</li>
|
||
</ul>
|
||
</div>
|
||
<p>
|
||
As an example, this section takes the transaction example provided
|
||
in <a class="xref" href="txnexample_c.html" title="Transaction Example">Transaction Example</a>
|
||
and it updates that example so that the environment, database, log
|
||
files, and regions are all kept entirely in-memory.
|
||
</p>
|
||
<p>
|
||
For illustration purposes, we also modify this example so that
|
||
uncommitted reads are no longer used to enable the
|
||
|
||
<code class="function">countRecords()</code>
|
||
function. Instead, we simply provide a transaction handle to
|
||
|
||
<code class="function">countRecords()</code>
|
||
so as to avoid the self-deadlock. Be aware that using a transaction handle here rather than
|
||
uncommitted reads will work just as well as if we had continued to use uncommitted reads. However,
|
||
the usage of the transaction handle here will
|
||
probably cause more deadlocks than using read-uncommitted does, because more locking is being performed in
|
||
this case.
|
||
</p>
|
||
<p>
|
||
To begin, we simplify the beginning of our example a bit. Because
|
||
we no longer need an environment home directory, we can remove all
|
||
the code that we used to determine path delimiters
|
||
and include the <code class="function">getopt</code> function. We can also
|
||
remove our <code class="function">usage()</code> function because we no
|
||
longer require any command line arguments.
|
||
</p>
|
||
<pre class="programlisting">// File TxnGuideInMemory.cpp
|
||
|
||
// We assume an ANSI-compatible compiler
|
||
#include <db_cxx.h>
|
||
#include <pthread.h>
|
||
#include <iostream>
|
||
|
||
// Run 5 writers threads at a time.
|
||
#define NUMWRITERS 5
|
||
|
||
// Printing of pthread_t is implementation-specific, so we
|
||
// create our own thread IDs for reporting purposes.
|
||
int global_thread_num;
|
||
pthread_mutex_t thread_num_lock;
|
||
|
||
// Forward declarations
|
||
int countRecords(Db *, DbTxn *);
|
||
int openDb(Db **, const char *, const char *, DbEnv *, u_int32_t);
|
||
int usage(void);
|
||
void *writerThread(void *); </pre>
|
||
<p>
|
||
Next, in our <code class="function">main()</code>, we also eliminate some
|
||
variables that this example no longer needs. In particular, we are able to remove
|
||
the
|
||
|
||
<code class="literal">dbHomeDir</code>
|
||
and
|
||
|
||
<code class="literal">fileName</code>
|
||
variables. We also remove all our <code class="function">getopt</code> code.
|
||
</p>
|
||
<pre class="programlisting">int
|
||
main(void)
|
||
{
|
||
// Initialize our handles
|
||
Db *dbp = NULL;
|
||
DbEnv *envp = NULL;
|
||
|
||
pthread_t writerThreads[NUMWRITERS];
|
||
int i;
|
||
u_int32_t envFlags;
|
||
|
||
// Application name
|
||
const char *progName = "TxnGuideInMemory"; </pre>
|
||
<p>
|
||
Next we create our environment as always. However, we add
|
||
<code class="literal">DB_PRIVATE</code> to our environment open flags. This
|
||
flag causes our environment to back regions using our
|
||
application's heap memory rather than by using the filesystem.
|
||
This is the first important step to keeping our DB data
|
||
entirely in-memory.
|
||
</p>
|
||
<p>
|
||
We also remove the <code class="literal">DB_RECOVER</code> flag from the environment open flags. Because our databases,
|
||
logs, and regions are maintained in-memory, there will never be anything to recover.
|
||
</p>
|
||
<p>
|
||
Note that we show the additional code here in
|
||
<strong class="userinput"><code>bold.</code></strong>
|
||
</p>
|
||
<pre class="programlisting"> // Env open flags
|
||
envFlags =
|
||
DB_CREATE | // Create the environment if it does not exist
|
||
DB_INIT_LOCK | // Initialize the locking subsystem
|
||
DB_INIT_LOG | // Initialize the logging subsystem
|
||
DB_INIT_TXN | // Initialize the transactional subsystem. This
|
||
// also turns on logging.
|
||
DB_INIT_MPOOL | // Initialize the memory pool (in-memory cache)
|
||
<strong class="userinput"><code>DB_PRIVATE | // Region files are not backed by the filesystem.
|
||
// Instead, they are backed by heap memory.</code></strong>
|
||
DB_THREAD; // Cause the environment to be free-threaded
|
||
|
||
try {
|
||
// Create the environment
|
||
envp = new DbEnv(0); </pre>
|
||
<p>
|
||
Now we configure our environment to keep the log files in memory,
|
||
increase the log buffer size to 10 MB, and increase our in-memory
|
||
cache to 10 MB. These values should be more than enough for our
|
||
application's workload.
|
||
</p>
|
||
<pre class="programlisting">
|
||
<strong class="userinput">
|
||
<code> // Specify in-memory logging
|
||
envp->log_set_config(DB_LOG_IN_MEMORY, 1);
|
||
|
||
// Specify the size of the in-memory log buffer.
|
||
envp->set_lg_bsize(10 * 1024 * 1024);
|
||
|
||
// Specify the size of the in-memory cache
|
||
envp->set_cachesize(0, 10 * 1024 * 1024, 1); </code>
|
||
</strong>
|
||
</pre>
|
||
<p>
|
||
Next, we open the environment and setup our lock detection. This is
|
||
identical to how the example previously worked, except that we do not
|
||
provide a location for the environment's home directory.
|
||
</p>
|
||
<pre class="programlisting"> // Indicate that we want db to internally perform deadlock
|
||
// detection. Also indicate that the transaction with
|
||
// the fewest number of write locks will receive the
|
||
// deadlock notification in the event of a deadlock.
|
||
envp->set_lk_detect(DB_LOCK_MINWRITE);
|
||
|
||
// Open the environment
|
||
envp->open(<strong class="userinput"><code>NULL</code></strong>, envFlags, 0); </pre>
|
||
<p>
|
||
When we call
|
||
|
||
<span><code class="function">openDb()</code>,</span>
|
||
which is what we use
|
||
to open our database, we no not provide a database filename for the
|
||
third parameter. When the filename is <code class="literal">NULL</code>, the database is not
|
||
backed by the filesystem.
|
||
</p>
|
||
<pre class="programlisting"> // If we had utility threads (for running checkpoints or
|
||
// deadlock detection, for example) we would spawn those
|
||
// here. However, for a simple example such as this,
|
||
// that is not required.
|
||
|
||
// Open the database
|
||
openDb(&dbp, progName, <strong class="userinput"><code>NULL</code></strong>,
|
||
envp, DB_DUPSORT);
|
||
</pre>
|
||
<p>
|
||
After that, our <code class="function">main()</code> function is unchanged,
|
||
except that when we
|
||
|
||
<span>check for exceptions on the database open,</span>
|
||
we change the error message string so as to not reference the database filename.
|
||
</p>
|
||
<pre class="programlisting"> // Initialize a pthread mutex. Used to help provide thread ids.
|
||
(void)pthread_mutex_init(&thread_num_lock, NULL);
|
||
|
||
// Start the writer threads.
|
||
for (i = 0; i < NUMWRITERS; i++)
|
||
(void)pthread_create(
|
||
&writerThreads[i], NULL,
|
||
writerThread,
|
||
(void *)dbp);
|
||
|
||
// Join the writers
|
||
for (i = 0; i < NUMWRITERS; i++)
|
||
(void)pthread_join(writerThreads[i], NULL);
|
||
|
||
} catch(DbException &e) {
|
||
<strong class="userinput"><code>std::cerr << "Error opening database environment: "</code></strong>
|
||
<< std::endl;
|
||
std::cerr << e.what() << std::endl;
|
||
return (EXIT_FAILURE);
|
||
}
|
||
|
||
try {
|
||
// Close our database handle if it was opened.
|
||
if (dbp != NULL)
|
||
dbp->close(0);
|
||
|
||
// Close our environment if it was opened.
|
||
if (envp != NULL)
|
||
envp->close(0);
|
||
} catch(DbException &e) {
|
||
std::cerr << "Error closing database and environment."
|
||
<< std::endl;
|
||
std::cerr << e.what() << std::endl;
|
||
return (EXIT_FAILURE);
|
||
}
|
||
|
||
// Final status message and return.
|
||
|
||
std::cout << "I'm all done." << std::endl;
|
||
return (EXIT_SUCCESS);
|
||
} </pre>
|
||
<p>
|
||
That completes <code class="function">main()</code>. The bulk of our
|
||
<code class="function">writerThread()</code> function implementation is
|
||
unchanged from the initial transaction example, except that we now pass
|
||
<code class="function">countRecords</code> a transaction handle, rather than configuring our
|
||
application to perform uncommitted reads. Both mechanisms work well-enough
|
||
for preventing a self-deadlock. However, the individual count
|
||
in this example will tend to be lower than the counts seen in
|
||
the previous transaction example, because
|
||
<code class="function">countRecords()</code> can no longer see records
|
||
created but not yet committed by other threads.
|
||
</p>
|
||
<pre class="programlisting">// A function that performs a series of writes to a
|
||
// Berkeley DB database. The information written
|
||
// to the database is largely nonsensical, but the
|
||
// mechanism of transactional commit/abort and
|
||
// deadlock detection is illustrated here.
|
||
void *
|
||
writerThread(void *args)
|
||
{
|
||
Db *dbp = (Db *)args;
|
||
DbEnv *envp = dbp->get_env(dbp);
|
||
|
||
int j, thread_num;
|
||
int max_retries = 20; // Max retry on a deadlock
|
||
char *key_strings[] = {"key 1", "key 2", "key 3", "key 4",
|
||
"key 5", "key 6", "key 7", "key 8",
|
||
"key 9", "key 10"};
|
||
|
||
// Get the thread number
|
||
(void)pthread_mutex_lock(&thread_num_lock);
|
||
global_thread_num++;
|
||
thread_num = global_thread_num;
|
||
(void)pthread_mutex_unlock(&thread_num_lock);
|
||
|
||
// Initialize the random number generator
|
||
srand((u_int)pthread_self());
|
||
|
||
// Perform 50 transactions
|
||
for (int i=0; i<50; i++) {
|
||
DbTxn *txn;
|
||
bool retry = true;
|
||
int retry_count = 0;
|
||
// while loop is used for deadlock retries
|
||
while (retry) {
|
||
// try block used for deadlock detection and
|
||
// general db exception handling
|
||
try {
|
||
|
||
// Begin our transaction. We group multiple writes in
|
||
// this thread under a single transaction so as to
|
||
// (1) show that you can atomically perform multiple
|
||
// writes at a time, and (2) to increase the chances
|
||
// of a deadlock occurring so that we can observe our
|
||
// deadlock detection at work.
|
||
|
||
// Normally we would want to avoid the potential for
|
||
// deadlocks, so for this workload the correct thing
|
||
// would be to perform our puts with auto commit. But
|
||
// that would excessively simplify our example, so we
|
||
// do the "wrong" thing here instead.
|
||
txn = NULL;
|
||
envp->txn_begin(NULL, &txn, 0);
|
||
// Perform the database write for this transaction.
|
||
for (j = 0; j < 10; j++) {
|
||
Dbt key, value;
|
||
key.set_data(key_strings[j]);
|
||
key.set_size((strlen(key_strings[j]) + 1) *
|
||
sizeof(char));
|
||
|
||
int payload = rand() + i;
|
||
value.set_data(&payload);
|
||
value.set_size(sizeof(int));
|
||
|
||
// Perform the database put
|
||
dbp->put(txn, &key, &value, 0);
|
||
}
|
||
|
||
// countRecords runs a cursor over the entire database.
|
||
// We do this to illustrate issues of deadlocking
|
||
std::cout << thread_num << " : Found "
|
||
<< countRecords(dbp, <strong class="userinput"><code>txn</code></strong>)
|
||
<< " records in the database." << std::endl;
|
||
|
||
std::cout << thread_num << " : committing txn : " << i
|
||
<< std::endl;
|
||
|
||
// commit
|
||
try {
|
||
txn->commit(0);
|
||
retry = false;
|
||
txn = NULL;
|
||
} catch (DbException &e) {
|
||
std::cout << "Error on txn commit: "
|
||
<< e.what() << std::endl;
|
||
}
|
||
} catch (DbDeadlockException &de) {
|
||
// First thing we MUST do is abort the transaction.
|
||
if (txn != NULL)
|
||
(void)txn->abort();
|
||
|
||
// Now we decide if we want to retry the operation.
|
||
// If we have retried less than max_retries,
|
||
// increment the retry count and goto retry.
|
||
if (retry_count < max_retries) {
|
||
std::cout << "############### Writer " << thread_num
|
||
<< ": Got DB_LOCK_DEADLOCK.\n"
|
||
<< "Retrying write operation."
|
||
<< std::endl;
|
||
retry_count++;
|
||
retry = true;
|
||
} else {
|
||
// Otherwise, just give up.
|
||
std::cerr << "Writer " << thread_num
|
||
<< ": Got DeadLockException and out of "
|
||
<< "retries. Giving up." << std::endl;
|
||
retry = false;
|
||
}
|
||
} catch (DbException &e) {
|
||
std::cerr << "db put failed" << std::endl;
|
||
std::cerr << e.what() << std::endl;
|
||
if (txn != NULL)
|
||
txn->abort();
|
||
retry = false;
|
||
} catch (std::exception &ee) {
|
||
std::cerr << "Unknown exception: " << ee.what() << std::endl;
|
||
return (0);
|
||
}
|
||
}
|
||
}
|
||
return (0);
|
||
} </pre>
|
||
<p>
|
||
Next we update
|
||
|
||
<span><code class="function">countRecords()</code>.</span>
|
||
The only difference
|
||
here is that we no longer specify <code class="literal">DB_READ_UNCOMMITTED</code> when
|
||
we open our cursor. Note that even this minor change is not required.
|
||
If we do not configure our database to support uncommitted reads,
|
||
<code class="literal">DB_READ_UNCOMMITTED</code> on the cursor open will be silently
|
||
ignored. However, we remove the flag anyway from the cursor open so as to
|
||
avoid confusion.
|
||
</p>
|
||
<pre class="programlisting">int
|
||
countRecords(Db *dbp, DbTxn *txn)
|
||
{
|
||
|
||
Dbc *cursorp = NULL;
|
||
int count = 0;
|
||
|
||
try {
|
||
// Get the cursor
|
||
dbp->cursor(txn, &cursorp, <strong class="userinput"><code>0</code></strong>);
|
||
|
||
Dbt key, value;
|
||
while (cursorp->get(&key, &value, DB_NEXT) == 0) {
|
||
count++;
|
||
}
|
||
} catch (DbDeadlockException &de) {
|
||
std::cerr << "countRecords: got deadlock" << std::endl;
|
||
cursorp->close();
|
||
throw de;
|
||
} catch (DbException &e) {
|
||
std::cerr << "countRecords error:" << std::endl;
|
||
std::cerr << e.what() << std::endl;
|
||
}
|
||
|
||
if (cursorp != NULL) {
|
||
try {
|
||
cursorp->close();
|
||
} catch (DbException &e) {
|
||
std::cerr << "countRecords: cursor close failed:" << std::endl;
|
||
std::cerr << e.what() << std::endl;
|
||
}
|
||
}
|
||
|
||
return (count);
|
||
} </pre>
|
||
<p>
|
||
Finally, we update
|
||
|
||
<span><code class="function">openDb()</code>.</span>
|
||
This involves
|
||
removing <code class="literal">DB_READ_UNCOMMITTED</code> from the
|
||
open flags.
|
||
|
||
|
||
</p>
|
||
<pre class="programlisting">// Open a Berkeley DB database
|
||
int
|
||
openDb(Db **dbpp, const char *progname, const char *fileName,
|
||
DbEnv *envp, u_int32_t extraFlags)
|
||
{
|
||
int ret;
|
||
u_int32_t openFlags;
|
||
|
||
try {
|
||
Db *dbp = new Db(envp, 0);
|
||
|
||
// Point to the new'd Db
|
||
*dbpp = dbp;
|
||
|
||
if (extraFlags != 0)
|
||
ret = dbp->set_flags(extraFlags);
|
||
|
||
// Now open the database
|
||
<strong class="userinput"><code>openFlags = DB_CREATE | // Allow database creation
|
||
DB_THREAD |
|
||
DB_AUTO_COMMIT; // Allow auto commit</code></strong>
|
||
|
||
dbp->open(NULL, // Txn pointer
|
||
fileName, // File name
|
||
NULL, // Logical db name
|
||
DB_BTREE, // Database type (using btree)
|
||
openFlags, // Open flags
|
||
0); // File mode. Using defaults
|
||
} catch (DbException &e) {
|
||
std::cerr << progname << ": openDb: db open failed:" << std::endl;
|
||
std::cerr << e.what() << std::endl;
|
||
return (EXIT_FAILURE);
|
||
}
|
||
|
||
return (EXIT_SUCCESS);
|
||
} </pre>
|
||
<p>
|
||
This completes our in-memory transactional example. If you would like to
|
||
experiment with this code, you can find the example in the following
|
||
location in your DB distribution:
|
||
</p>
|
||
<pre class="programlisting"><span class="emphasis"><em>DB_INSTALL</em></span>/examples_cxx/txn_guide</pre>
|
||
</div>
|
||
<div class="navfooter">
|
||
<hr />
|
||
<table width="100%" summary="Navigation footer">
|
||
<tr>
|
||
<td width="40%" align="left"><a accesskey="p" href="txnexample_c.html">Prev</a> </td>
|
||
<td width="20%" align="center">
|
||
<a accesskey="u" href="wrapup.html">Up</a>
|
||
</td>
|
||
<td width="40%" align="right"> </td>
|
||
</tr>
|
||
<tr>
|
||
<td width="40%" align="left" valign="top">Transaction Example </td>
|
||
<td width="20%" align="center">
|
||
<a accesskey="h" href="index.html">Home</a>
|
||
</td>
|
||
<td width="40%" align="right" valign="top"> </td>
|
||
</tr>
|
||
</table>
|
||
</div>
|
||
</body>
|
||
</html>
|