libdb/docs/programmer_reference/am_misc_bulk.html

252 lines
17 KiB
HTML
Raw Permalink Normal View History

2011-09-13 17:44:24 +00:00
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<title>Retrieving and updating records in bulk</title>
<link rel="stylesheet" href="gettingStarted.css" type="text/css" />
<meta name="generator" content="DocBook XSL Stylesheets V1.73.2" />
<link rel="start" href="index.html" title="Berkeley DB Programmer's Reference Guide" />
<link rel="up" href="am_misc.html" title="Chapter 4.  Access Method Wrapup" />
<link rel="prev" href="am_misc.html" title="Chapter 4.  Access Method Wrapup" />
<link rel="next" href="am_misc_partial.html" title="Partial record storage and retrieval" />
</head>
<body>
<div xmlns="" class="navheader">
<div class="libver">
2012-11-14 21:35:20 +00:00
<p>Library Version 11.2.5.3</p>
2011-09-13 17:44:24 +00:00
</div>
<table width="100%" summary="Navigation header">
<tr>
<th colspan="3" align="center">Retrieving and updating records in bulk</th>
</tr>
<tr>
<td width="20%" align="left"><a accesskey="p" href="am_misc.html">Prev</a> </td>
<th width="60%" align="center">Chapter 4. 
Access Method Wrapup
</th>
<td width="20%" align="right"> <a accesskey="n" href="am_misc_partial.html">Next</a></td>
</tr>
</table>
<hr />
</div>
<div class="sect1" lang="en" xml:lang="en">
<div class="titlepage">
<div>
<div>
<h2 class="title" style="clear: both"><a id="am_misc_bulk"></a>Retrieving and updating records in bulk</h2>
</div>
</div>
</div>
<div class="toc">
<dl>
<dt>
<span class="sect2">
<a href="am_misc_bulk.html#am_misc_bulk_get">Bulk retrieval</a>
</span>
</dt>
<dt>
<span class="sect2">
<a href="am_misc_bulk.html#am_misc_bulk_put">Bulk updates</a>
</span>
</dt>
<dt>
<span class="sect2">
<a href="am_misc_bulk.html#am_misc_bulk_del">Bulk deletes</a>
</span>
</dt>
</dl>
</div>
<p>When retrieving or modifying large numbers of records, the number
of method calls can often dominate performance. Berkeley DB offers bulk get,
put and delete interfaces which can significantly increase performance for some
applications.
</p>
<div class="sect2" lang="en" xml:lang="en">
<div class="titlepage">
<div>
<div>
<h3 class="title"><a id="am_misc_bulk_get"></a>Bulk retrieval</h3>
</div>
</div>
</div>
<p>
To retrieve records in bulk, an application buffer must be specified to
the <a href="../api_reference/C/dbget.html" class="olink">DB-&gt;get()</a> or <a href="../api_reference/C/dbcget.html" class="olink">DBC-&gt;get()</a> methods. This is done in the C API by setting
the <span class="bold"><strong>data</strong></span> and <span class="bold"><strong>ulen</strong></span> fields of the <span class="bold"><strong>data</strong></span> <a href="../api_reference/C/dbt.html" class="olink">DBT</a> to reference an application
buffer, and the
<span class="bold"><strong>flags</strong></span> field of that structure to
<a href="../api_reference/C/dbt.html#dbt_DB_DBT_USERMEM" class="olink">DB_DBT_USERMEM</a>. In the Berkeley DB C++ and Java APIs, the actions
are similar, although there are API-specific methods to set the <a href="../api_reference/C/dbt.html" class="olink">DBT</a>
values. Then, the <a href="../api_reference/C/dbcget.html#dbcget_DB_MULTIPLE" class="olink">DB_MULTIPLE</a> or <a href="../api_reference/C/dbcget.html#dbcget_DB_MULTIPLE_KEY" class="olink">DB_MULTIPLE_KEY</a> flags are
specified to the <a href="../api_reference/C/dbget.html" class="olink">DB-&gt;get()</a> or <a href="../api_reference/C/dbcget.html" class="olink">DBC-&gt;get()</a> methods, which cause multiple
records to be returned in the specified buffer.
</p>
<p>
The difference between <a href="../api_reference/C/dbcget.html#dbcget_DB_MULTIPLE" class="olink">DB_MULTIPLE</a> and <a href="../api_reference/C/dbcget.html#dbcget_DB_MULTIPLE_KEY" class="olink">DB_MULTIPLE_KEY</a> is as
follows: <a href="../api_reference/C/dbcget.html#dbcget_DB_MULTIPLE" class="olink">DB_MULTIPLE</a> returns multiple data items for a single key.
For example, the <a href="../api_reference/C/dbcget.html#dbcget_DB_MULTIPLE" class="olink">DB_MULTIPLE</a> flag would be used to retrieve all of
the duplicate data items for a single key in a single call. The
<a href="../api_reference/C/dbcget.html#dbcget_DB_MULTIPLE_KEY" class="olink">DB_MULTIPLE_KEY</a> flag is used to retrieve multiple key/data pairs,
where each returned key may or may not have duplicate data items.
</p>
<p>
Once the <a href="../api_reference/C/dbget.html" class="olink">DB-&gt;get()</a> or <a href="../api_reference/C/dbcget.html" class="olink">DBC-&gt;get()</a> method has returned, the application will
walk through the buffer handling the returned records. This is
implemented for the C and C++ APIs using four macros:
<a href="../api_reference/C/DB_MULTIPLE_INIT.html" class="olink">DB_MULTIPLE_INIT</a>, <a href="../api_reference/C/DB_MULTIPLE_NEXT.html" class="olink">DB_MULTIPLE_NEXT</a>, <a href="../api_reference/C/DB_MULTIPLE_KEY_NEXT.html" class="olink">DB_MULTIPLE_KEY_NEXT</a>, and
<a href="../api_reference/C/DB_MULTIPLE_RECNO_NEXT.html" class="olink">DB_MULTIPLE_RECNO_NEXT</a>. For the Java API, this is implemented as
three iterator classes:
<a class="ulink" href="../java/com/sleepycat/db/MultipleDataEntry.html" target="_top">MultipleDataEntry</a>,
<a class="ulink" href="../java/com/sleepycat/db/MultipleKeyDataEntry.html" target="_top">MultipleKeyDataEntry</a>,
and <a class="ulink" href="../java/com/sleepycat/db/MultipleRecnoDataEntry.html" target="_top">MultipleRecnoDataEntry</a>.
</p>
<p>
The <a href="../api_reference/C/DB_MULTIPLE_INIT.html" class="olink">DB_MULTIPLE_INIT</a> macro is always called first. It initializes a
local application variable and the <span class="bold"><strong>data</strong></span> <a href="../api_reference/C/dbt.html" class="olink">DBT</a> for stepping through the set of
returned records. Then, the application calls one of the remaining
three macros: <a href="../api_reference/C/DB_MULTIPLE_NEXT.html" class="olink">DB_MULTIPLE_NEXT</a>, <a href="../api_reference/C/DB_MULTIPLE_KEY_NEXT.html" class="olink">DB_MULTIPLE_KEY_NEXT</a>, and
<a href="../api_reference/C/DB_MULTIPLE_RECNO_NEXT.html" class="olink">DB_MULTIPLE_RECNO_NEXT</a>.
</p>
<p>
If the <a href="../api_reference/C/dbcget.html#dbcget_DB_MULTIPLE" class="olink">DB_MULTIPLE</a> flag was specified to the <a href="../api_reference/C/dbget.html" class="olink">DB-&gt;get()</a> or <a href="../api_reference/C/dbcget.html" class="olink">DBC-&gt;get()</a>
method, the application will always call the <a href="../api_reference/C/DB_MULTIPLE_NEXT.html" class="olink">DB_MULTIPLE_NEXT</a> macro.
If the <a href="../api_reference/C/dbcget.html#dbcget_DB_MULTIPLE_KEY" class="olink">DB_MULTIPLE_KEY</a> flag was specified to the <a href="../api_reference/C/dbget.html" class="olink">DB-&gt;get()</a> or <a href="../api_reference/C/dbcget.html" class="olink">DBC-&gt;get()</a>
method, and the underlying database is a Btree or Hash database, the
application will always call the <a href="../api_reference/C/DB_MULTIPLE_KEY_NEXT.html" class="olink">DB_MULTIPLE_KEY_NEXT</a> macro. If the
<a href="../api_reference/C/dbcget.html#dbcget_DB_MULTIPLE_KEY" class="olink">DB_MULTIPLE_KEY</a> flag was specified to the <a href="../api_reference/C/dbget.html" class="olink">DB-&gt;get()</a> or <a href="../api_reference/C/dbcget.html" class="olink">DBC-&gt;get()</a> method,
and the underlying database is a Queue or Recno database, the
application will always call the <a href="../api_reference/C/DB_MULTIPLE_RECNO_NEXT.html" class="olink">DB_MULTIPLE_RECNO_NEXT</a> macro. The
<a href="../api_reference/C/DB_MULTIPLE_NEXT.html" class="olink">DB_MULTIPLE_NEXT</a>, <a href="../api_reference/C/DB_MULTIPLE_KEY_NEXT.html" class="olink">DB_MULTIPLE_KEY_NEXT</a>, and
<a href="../api_reference/C/DB_MULTIPLE_RECNO_NEXT.html" class="olink">DB_MULTIPLE_RECNO_NEXT</a> macros are called repeatedly, until the end of
the returned records is reached. The end of the returned records is
detected by the application's local pointer variable being set to NULL.
</p>
<p>
Note that if you want to use a cursor for bulk retrieval of records in
a Btree database, you should open the cursor using the
<code class="literal">DB_CURSOR_BULK</code> flag. This optimizes the cursor for
bulk retrieval.
</p>
<p>
The following is an example of a routine that displays the contents of
a Btree database using the bulk return interfaces.
</p>
<a id="prog_am22"></a>
<pre class="programlisting">int
rec_display(DB *dbp)
{
DBC *dbcp;
DBT key, data;
size_t retklen, retdlen;
void *retkey, *retdata;
int ret, t_ret;
void *p;
memset(&amp;key, 0, sizeof(key));
memset(&amp;data, 0, sizeof(data));
/* Review the database in 5MB chunks. */
#define BUFFER_LENGTH (5 * 1024 * 1024)
if ((data.data = malloc(BUFFER_LENGTH)) == NULL)
return (errno);
data.ulen = BUFFER_LENGTH;
data.flags = DB_DBT_USERMEM;
/* Acquire a cursor for the database. */
if ((ret = dbp-&gt;cursor(dbp, NULL, &amp;dbcp, DB_CURSOR_BULK))
!= 0) {
dbp-&gt;err(dbp, ret, "DB-&gt;cursor");
free(data.data);
return (ret);
}
for (;;) {
/*
* Acquire the next set of key/data pairs. This code
* does not handle single key/data pairs that won't fit
* in a BUFFER_LENGTH size buffer, instead returning
* DB_BUFFER_SMALL to our caller.
*/
if ((ret = dbcp-&gt;get(dbcp,
&amp;key, &amp;data, DB_MULTIPLE_KEY | DB_NEXT)) != 0) {
if (ret != DB_NOTFOUND)
dbp-&gt;err(dbp, ret, "DBcursor-&gt;get");
break;
}
for (DB_MULTIPLE_INIT(p, &amp;data);;) {
DB_MULTIPLE_KEY_NEXT(p,
&amp;data, retkey, retklen, retdata, retdlen);
if (p == NULL)
break;
printf("key: %.*s, data: %.*s\n",
(int)retklen, (char *)retkey, (int)retdlen,
(char *)retdata);
}
}
if ((t_ret = dbcp-&gt;close(dbcp)) != 0) {
dbp-&gt;err(dbp, ret, "DBcursor-&gt;close");
if (ret == 0)
ret = t_ret;
}
free(data.data);
return (ret);
}</pre>
</div>
<div class="sect2" lang="en" xml:lang="en">
<div class="titlepage">
<div>
<div>
<h3 class="title"><a id="am_misc_bulk_put"></a>Bulk updates</h3>
</div>
</div>
</div>
<p>To put records in bulk with the btree or hash access methods, construct bulk buffers in the <span class="bold"><strong>key</strong></span> and <span class="bold"><strong>data</strong></span> <a href="../api_reference/C/dbt.html" class="olink">DBT</a> using <a href="../api_reference/C/DB_MULTIPLE_WRITE_INIT.html" class="olink">DB_MULTIPLE_WRITE_INIT</a> and <a href="../api_reference/C/DB_MULTIPLE_WRITE_NEXT.html" class="olink">DB_MULTIPLE_WRITE_NEXT</a>. To put records in bulk with the recno or queue access methods, construct bulk buffers in the <span class="bold"><strong>data</strong></span> <a href="../api_reference/C/dbt.html" class="olink">DBT</a> as before, but construct the <span class="bold"><strong>key</strong></span> <a href="../api_reference/C/dbt.html" class="olink">DBT</a> using <a href="../api_reference/C/DB_MULTIPLE_RECNO_WRITE_INIT.html" class="olink">DB_MULTIPLE_RECNO_WRITE_INIT</a> and <a href="../api_reference/C/DB_MULTIPLE_RECNO_WRITE_NEXT.html" class="olink">DB_MULTIPLE_RECNO_WRITE_NEXT</a> with a data size of zero;. In both cases, set the <a href="../api_reference/C/dbcget.html#dbcget_DB_MULTIPLE" class="olink">DB_MULTIPLE</a> flag to <a href="../api_reference/C/dbput.html" class="olink">DB-&gt;put()</a>.</p>
<p>Alternatively, for btree and hash access methods, construct a single bulk buffer in the <span class="bold"><strong>key</strong></span> <a href="../api_reference/C/dbt.html" class="olink">DBT</a> using <a href="../api_reference/C/DB_MULTIPLE_WRITE_INIT.html" class="olink">DB_MULTIPLE_WRITE_INIT</a> and <a href="../api_reference/C/DB_MULTIPLE_KEY_WRITE_NEXT.html" class="olink">DB_MULTIPLE_KEY_WRITE_NEXT</a>. For recno and queue access methods, construct a bulk buffer in the <span class="bold"><strong>key</strong></span> <a href="../api_reference/C/dbt.html" class="olink">DBT</a> using <a href="../api_reference/C/DB_MULTIPLE_RECNO_WRITE_INIT.html" class="olink">DB_MULTIPLE_RECNO_WRITE_INIT</a> and <a href="../api_reference/C/DB_MULTIPLE_RECNO_WRITE_NEXT.html" class="olink">DB_MULTIPLE_RECNO_WRITE_NEXT</a>. In both cases, set the <a href="../api_reference/C/dbcget.html#dbcget_DB_MULTIPLE_KEY" class="olink">DB_MULTIPLE_KEY</a> flag to <a href="../api_reference/C/dbput.html" class="olink">DB-&gt;put()</a>.
</p>
<p>A successful bulk operation is logically equivalent to a loop through each key/data pair, performing a <a href="../api_reference/C/dbput.html" class="olink">DB-&gt;put()</a> for each one.</p>
<p>
</p>
</div>
<div class="sect2" lang="en" xml:lang="en">
<div class="titlepage">
<div>
<div>
<h3 class="title"><a id="am_misc_bulk_del"></a>Bulk deletes</h3>
</div>
</div>
</div>
<p>To delete all records with a specified set of keys with the btree or hash access methods, construct a bulk buffer in the <span class="bold"><strong>key</strong></span> <a href="../api_reference/C/dbt.html" class="olink">DBT</a> using <a href="../api_reference/C/DB_MULTIPLE_WRITE_INIT.html" class="olink">DB_MULTIPLE_WRITE_INIT</a> and <a href="../api_reference/C/DB_MULTIPLE_WRITE_NEXT.html" class="olink">DB_MULTIPLE_WRITE_NEXT</a>. To delete a set of records with the recno or queue access methods, construct the <span class="bold"><strong>key</strong></span> <a href="../api_reference/C/dbt.html" class="olink">DBT</a> using <a href="../api_reference/C/DB_MULTIPLE_RECNO_WRITE_INIT.html" class="olink">DB_MULTIPLE_RECNO_WRITE_INIT</a> and <a href="../api_reference/C/DB_MULTIPLE_RECNO_WRITE_NEXT.html" class="olink">DB_MULTIPLE_RECNO_WRITE_NEXT</a> with a data size of zero. In both cases, set the <a href="../api_reference/C/dbcget.html#dbcget_DB_MULTIPLE" class="olink">DB_MULTIPLE</a> flag to <a href="../api_reference/C/dbdel.html" class="olink">DB-&gt;del()</a>. This is equivalent to calling <a href="../api_reference/C/dbdel.html" class="olink">DB-&gt;del()</a> for each key in the bulk buffer. In particular, if the database supports duplicates, all records with the matching key are deleted.</p>
<p>Alternatively, to delete a specific set of key/data pairs, which may be items within a set of duplicates, there are also two cases depending on whether the access method uses record numbers for keys. For btree and hash access methods, construct a single bulk buffer in the <span class="bold"><strong>key</strong></span> <a href="../api_reference/C/dbt.html" class="olink">DBT</a> using <a href="../api_reference/C/DB_MULTIPLE_WRITE_INIT.html" class="olink">DB_MULTIPLE_WRITE_INIT</a> and <a href="../api_reference/C/DB_MULTIPLE_KEY_WRITE_NEXT.html" class="olink">DB_MULTIPLE_KEY_WRITE_NEXT</a>. For recno and queue access methods, construct a bulk buffer in the <span class="bold"><strong>key</strong></span> <a href="../api_reference/C/dbt.html" class="olink">DBT</a> using <a href="../api_reference/C/DB_MULTIPLE_RECNO_WRITE_INIT.html" class="olink">DB_MULTIPLE_RECNO_WRITE_INIT</a> and <a href="../api_reference/C/DB_MULTIPLE_RECNO_WRITE_NEXT.html" class="olink">DB_MULTIPLE_RECNO_WRITE_NEXT</a>. In both cases, set the <a href="../api_reference/C/dbcget.html#dbcget_DB_MULTIPLE_KEY" class="olink">DB_MULTIPLE_KEY</a> flag to <a href="../api_reference/C/dbdel.html" class="olink">DB-&gt;del()</a>.</p>
<p>A successful bulk operation is logically equivalent to a loop through each key/data pair, performing a <a href="../api_reference/C/dbdel.html" class="olink">DB-&gt;del()</a> for each one.</p>
</div>
</div>
<div class="navfooter">
<hr />
<table width="100%" summary="Navigation footer">
<tr>
<td width="40%" align="left"><a accesskey="p" href="am_misc.html">Prev</a> </td>
<td width="20%" align="center">
<a accesskey="u" href="am_misc.html">Up</a>
</td>
<td width="40%" align="right"> <a accesskey="n" href="am_misc_partial.html">Next</a></td>
</tr>
<tr>
<td width="40%" align="left" valign="top">Chapter 4. 
Access Method Wrapup
 </td>
<td width="20%" align="center">
<a accesskey="h" href="index.html">Home</a>
</td>
<td width="40%" align="right" valign="top"> Partial record storage and retrieval</td>
</tr>
</table>
</div>
</body>
</html>