mirror of
https://github.com/berkeleydb/libdb.git
synced 2024-11-16 09:06:25 +00:00
241 lines
13 KiB
HTML
241 lines
13 KiB
HTML
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
|
||
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
|
||
<html xmlns="http://www.w3.org/1999/xhtml">
|
||
<head>
|
||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
|
||
<title>Berkeley DB recoverability</title>
|
||
<link rel="stylesheet" href="gettingStarted.css" type="text/css" />
|
||
<meta name="generator" content="DocBook XSL Stylesheets V1.73.2" />
|
||
<link rel="start" href="index.html" title="Berkeley DB Programmer's Reference Guide" />
|
||
<link rel="up" href="transapp.html" title="Chapter 11. Berkeley DB Transactional Data Store Applications" />
|
||
<link rel="prev" href="transapp_filesys.html" title="Recovery and filesystem operations" />
|
||
<link rel="next" href="transapp_tune.html" title="Transaction tuning" />
|
||
</head>
|
||
<body>
|
||
<div xmlns="" class="navheader">
|
||
<div class="libver">
|
||
<p>Library Version 11.2.5.3</p>
|
||
</div>
|
||
<table width="100%" summary="Navigation header">
|
||
<tr>
|
||
<th colspan="3" align="center">Berkeley DB recoverability</th>
|
||
</tr>
|
||
<tr>
|
||
<td width="20%" align="left"><a accesskey="p" href="transapp_filesys.html">Prev</a> </td>
|
||
<th width="60%" align="center">Chapter 11.
|
||
Berkeley DB Transactional Data Store Applications
|
||
</th>
|
||
<td width="20%" align="right"> <a accesskey="n" href="transapp_tune.html">Next</a></td>
|
||
</tr>
|
||
</table>
|
||
<hr />
|
||
</div>
|
||
<div class="sect1" lang="en" xml:lang="en">
|
||
<div class="titlepage">
|
||
<div>
|
||
<div>
|
||
<h2 class="title" style="clear: both"><a id="transapp_reclimit"></a>Berkeley DB recoverability</h2>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<p>
|
||
Berkeley DB recovery is based on write-ahead logging. This means
|
||
that when a change is made to a database page, a description of the
|
||
change is written into a log file. This description in the log
|
||
file is guaranteed to be written to stable storage before the
|
||
database pages that were changed are written to stable storage.
|
||
This is the fundamental feature of the logging system that makes
|
||
durability and rollback work.
|
||
</p>
|
||
<p>
|
||
If the application or system crashes, the log is reviewed during
|
||
recovery. Any database changes described in the log that were part
|
||
of committed transactions and that were never written to the actual
|
||
database itself are written to the database as part of recovery.
|
||
Any database changes described in the log that were never committed
|
||
and that were written to the actual database itself are backed-out
|
||
of the database as part of recovery. This design allows the
|
||
database to be written lazily, and only blocks from the log file
|
||
have to be forced to disk as part of transaction commit.
|
||
</p>
|
||
<p>
|
||
There are two interfaces that are a concern when considering
|
||
Berkeley DB recoverability:
|
||
</p>
|
||
<div class="orderedlist">
|
||
<ol type="1">
|
||
<li>
|
||
The interface between Berkeley DB and the operating
|
||
system/filesystem.
|
||
</li>
|
||
<li>
|
||
The interface between the operating system/filesystem and the
|
||
underlying stable storage hardware.
|
||
</li>
|
||
</ol>
|
||
</div>
|
||
<p>
|
||
Berkeley DB uses the operating system interfaces and its underlying
|
||
filesystem when writing its files. This means that Berkeley DB can
|
||
fail if the underlying filesystem fails in some unrecoverable way.
|
||
Otherwise, the interface requirements here are simple: The system
|
||
call that Berkeley DB uses to flush data to disk (normally fsync or
|
||
fdatasync), must guarantee that all the information necessary for a
|
||
file's recoverability has been written to stable storage before it
|
||
returns to Berkeley DB, and that no possible application or system
|
||
crash can cause that file to be unrecoverable.
|
||
</p>
|
||
<p>
|
||
In addition, Berkeley DB implicitly uses the interface between the
|
||
operating system and the underlying hardware. The interface
|
||
requirements here are not as simple.
|
||
</p>
|
||
<p>
|
||
First, it is necessary to consider the underlying page size of the
|
||
Berkeley DB databases. The Berkeley DB library performs all
|
||
database writes using the page size specified by the application,
|
||
and Berkeley DB assumes pages are written atomically. This means
|
||
that if the operating system performs filesystem I/O in blocks of
|
||
different sizes than the database page size, it may increase the
|
||
possibility for database corruption. For example, assume that
|
||
Berkeley DB is writing 32KB pages for a database, and the operating
|
||
system does filesystem I/O in 16KB blocks. If the operating system
|
||
writes the first 16KB of the database page successfully, but
|
||
crashes before being able to write the second 16KB of the database,
|
||
the database has been corrupted and this corruption may or may not
|
||
be detected during recovery. For this reason, it may be important
|
||
to select database page sizes that will be written as single block
|
||
transfers by the underlying operating system. If you do not select
|
||
a page size that the underlying operating system will write as a
|
||
single block, you may want to configure the database to use
|
||
checksums (see the <a href="../api_reference/C/dbset_flags.html" class="olink">DB->set_flags()</a> flag for more information). By
|
||
configuring checksums, you guarantee this kind of corruption will
|
||
be detected at the expense of the CPU required to generate the
|
||
checksums. When such an error is detected, the only course of
|
||
recovery is to perform catastrophic recovery to restore the
|
||
database.
|
||
</p>
|
||
<p>
|
||
Second, if you are copying database files (either as part of doing
|
||
a hot backup or creation of a hot failover area), there is an
|
||
additional question related to the page size of the Berkeley DB
|
||
databases. You must copy databases atomically, in units of the
|
||
database page size. In other words, the reads made by the copy
|
||
program must not be interleaved with writes by other threads of
|
||
control, and the copy program must read the databases in multiples
|
||
of the underlying database page size. On Unix systems, this is not
|
||
a problem, as these operating systems already make this guarantee
|
||
and system utilities normally read in power-of-2 sized chunks,
|
||
which are larger than the largest possible Berkeley DB database
|
||
page size. Other operating systems, particularly those based on
|
||
Linux and Windows, do not provide this guarantee and hot backups may
|
||
not be performed on these systems by reading data from the file
|
||
system. The <a href="../api_reference/C/db_hotbackup.html" class="olink">db_hotbackup</a> utility should be used on these
|
||
systems.
|
||
</p>
|
||
<p>
|
||
An additional problem we have seen in this area was in some
|
||
releases of Solaris where the cp utility was implemented using the
|
||
mmap system call rather than the read system call. Because the
|
||
Solaris' mmap system call did not make the same guarantee of read
|
||
atomicity as the read system call, using the cp utility could
|
||
create corrupted copies of the databases. Another problem we have
|
||
seen is implementations of the tar utility doing 10KB block reads
|
||
by default, and even when an output block size was specified to
|
||
that utility, not reading from the underlying databases in
|
||
multiples of the block size. Using the dd utility instead of the
|
||
cp or tar utilities (and specifying an appropriate block size),
|
||
fixes these problems. If you plan to use a system utility to copy
|
||
database files, you may want to use a system call trace utility
|
||
(for example, ktrace or truss) to check for an I/O size smaller
|
||
than or not a multiple of the database page size and system calls
|
||
other than read.
|
||
</p>
|
||
<p>
|
||
Third, it is necessary to consider the behavior of the system's
|
||
underlying stable storage hardware. For example, consider a SCSI
|
||
controller that has been configured to cache data and return to the
|
||
operating system that the data has been written to stable storage,
|
||
when, in fact, it has only been written into the controller RAM
|
||
cache. If power is lost before the controller is able to flush its
|
||
cache to disk, and the controller cache is not stable (that is, the
|
||
writes will not be flushed to disk when power returns), the writes
|
||
will be lost. If the writes include database blocks, there is no
|
||
loss because recovery will correctly update the database. If the
|
||
writes include log file blocks, it is possible that transactions
|
||
that were already committed may not appear in the recovered
|
||
database, although the recovered database will be coherent after a
|
||
crash.
|
||
</p>
|
||
<p>
|
||
If the underlying hardware can fail in any way so that only part of
|
||
the block was written, the failure conditions are the same as those
|
||
described previously for an operating system failure that writes
|
||
only part of a logical database block. In such cases, configuring
|
||
the database for checksums will ensure the corruption is
|
||
detected.
|
||
</p>
|
||
<p>
|
||
For these reasons, it may be important to select hardware that does
|
||
not do partial writes and does not cache data writes (or does not
|
||
return that the data has been written to stable storage until it
|
||
has either been written to stable storage or the actual writing of
|
||
all of the data is guaranteed, barring catastrophic hardware
|
||
failure — that is, your disk drive exploding).
|
||
</p>
|
||
<p>
|
||
If the disk drive on which you are storing your databases explodes,
|
||
you can perform normal Berkeley DB catastrophic recovery, because
|
||
it requires only a snapshot of your databases plus the log files
|
||
you have archived since those snapshots were taken. In this case,
|
||
you should lose no database changes at all.
|
||
</p>
|
||
<p>
|
||
If the disk drive on which you are storing your log files explodes,
|
||
you can also perform catastrophic recovery, but you will lose any
|
||
database changes made as part of transactions committed since your
|
||
last archival of the log files. Alternatively, if your database
|
||
environment and databases are still available after you lose the
|
||
log file disk, you should be able to dump your databases. However,
|
||
you may see an inconsistent snapshot of your data after doing the
|
||
dump, because changes that were part of transactions that were not
|
||
yet committed may appear in the database dump. Depending on the
|
||
value of the data, a reasonable alternative may be to perform both
|
||
the database dump and the catastrophic recovery and then compare
|
||
the databases created by the two methods.
|
||
</p>
|
||
<p>
|
||
Regardless, for these reasons, storing your databases and log files
|
||
on different disks should be considered a safety measure as well as
|
||
a performance enhancement.
|
||
</p>
|
||
<p>
|
||
Finally, you should be aware that Berkeley DB does not protect
|
||
against all cases of stable storage hardware failure, nor does it
|
||
protect against simple hardware misbehavior (for example, a disk
|
||
controller writing incorrect data to the disk). However,
|
||
configuring the database for checksums will ensure that any such
|
||
corruption is detected.
|
||
</p>
|
||
</div>
|
||
<div class="navfooter">
|
||
<hr />
|
||
<table width="100%" summary="Navigation footer">
|
||
<tr>
|
||
<td width="40%" align="left"><a accesskey="p" href="transapp_filesys.html">Prev</a> </td>
|
||
<td width="20%" align="center">
|
||
<a accesskey="u" href="transapp.html">Up</a>
|
||
</td>
|
||
<td width="40%" align="right"> <a accesskey="n" href="transapp_tune.html">Next</a></td>
|
||
</tr>
|
||
<tr>
|
||
<td width="40%" align="left" valign="top">Recovery and filesystem operations </td>
|
||
<td width="20%" align="center">
|
||
<a accesskey="h" href="index.html">Home</a>
|
||
</td>
|
||
<td width="40%" align="right" valign="top"> Transaction tuning</td>
|
||
</tr>
|
||
</table>
|
||
</div>
|
||
</body>
|
||
</html>
|