Informatics

issues with Berkeley DB (BDB) used as Openldap backend

I’ve been using Openldap for some years.
Openldap uses by default the Berkeley DB (BDB) as backend repository.
BDB is an embedded DBMS: it keeps the data safe, so one just don’t have to care ’bout how this is done.

BDB has suffered a lot of changes along its history: it originated at the University of California, Berkeley, was used by Netscape, raised a company called Sleepycat (those were funny times with a funny logo), and it was finally acquired by Oracle in 2006 (end of the party).
I think this last acquisition was catastrophic for BDB documentation: almost all links found on internet are broken, as Oracle first migrated but later changed several times the docs and the links. Now there’s a swamp of mangled configuration and API docs and a jungle of deprecated options that I find impossible to traverse.
Some doc sources:

A few weeks ago I experienced a problem with my configured openldap replication (syncrepl mode) to one of my secondary openldap servers: the synchronization began as expected after I made a bunch (>50000) attribute value modifications on the master, but actually it never ended… or at least it didn’t seem to do so in a reasonable amount of time. I mean, two weeks is not a reasonable amount of time, and I did wait for such a long time for it to end…
Other replicas ended the synchronization after 4 hours.

Openldap seemed to be synchronizing, but the biggest problem was the extraordinarily amount of disk traffic that it generated.
BDB uses logging in order to assure ACID operations and db recovery.
By default, this is achieved using 10 MiB files (named log.0000000xyz) that are used one after another, so a long transaction (as this synchronization) can use about 30-40 MiB per minute… This obliges to a periodic checkpoint and deletion of the remaining and yet unnecessary log files (BDB has tools for doing this: db_checkpoint + db_archive). Problem is that this server was generating about 3k IOPS and 10 MiB/s disk traffic.
This doesn’t seem healthy at all, so I finally decided that there was a problem with the BDB or with the syncrepl’s openldap replication. Either case, I deleted the backend, and tried to make a fresh import of the data.

Imports to BDB can be made using BDB’s own tools (like db_dump and db_load), but in this case, this replica have but a subset of the original, so this was not an option.
Instead, I made an ldif export of the master using the proper ldap user, and tried to make and import on this replica using “slapadd” openldap tool.

The generated ldif file was 120 MiB sized (about 4 million lines and >>50000 entries).
The configuration for the openldap branch includes about 30 indexes. This makes the loading a bit longer, and expands the DB to about 700 MiB of total disk size.

Problem was that the process stopped with this error:

bdb(ou=**********): In-memory log buffer is full (an active transaction spans the buffer)
=> bdb_idl_insert_key: c_get next_dup failed: DB_LOG_BUFFER_FULL: In-memory log buffer is full (-30993)
=> bdb_tool_entry_put: index_entry_add failed: Accessing a corrupted shared library (80)
=> bdb_tool_entry_put: txn_aborted! Accessing a corrupted shared library (80)
slapadd: could not add entry dn="**********" (line=3179771): txn_aborted! Accessing a corrupted shared library (80)

I tried to make the same load with DB_LOG_INMEMORY BDB option on DB_CONFIG:

    set_flags DB_LOG_INMEMORY

but even if this made no use of db log files, and so speed up the process, the error was the same.

I think the error could be related with two factors:
First, there seems to be some bug in my bdb version… “Accessing a corrupted shared library”? how could this possibly be? And more interestingly, __db.* files summed a total of about >4 GiB. “__db.*” files store in disk BDB shared memory regions:

Shared memory regions
Each of the Berkeley DB subsystems within an environment is described by one or more regions. The regions contain all of the per-process and per-thread shared information, including mutexes, that comprise a Berkeley DB environment.
...
These files are named __db.### (for example, __db.001, __db.002 and so on).

(from http://pybsddb.sourceforge.net/ref/env/region.html, and http://docs.oracle.com/cd/E17076_02/html/programmer_reference/env_region.html)

Im my case, an empty BDB configuration (see DB_CONFIG later) results in:

.....24576 Dec 1 11:37 __db.001
1407229952 Dec 1 11:37 __db.002
.890593280 Dec 1 11:37 __db.003
.890593280 Dec 1 11:37 __db.004
.890593280 Dec 1 11:37 __db.005
...1114112 Dec 1 11:37 __db.006
.551944192 Dec 1 11:37 __db.007
.....24576 Dec 1 11:37 __db.008
------------
4632117248 = 4.31 GiB

It’s true that the OS is 64 bits, the installed openldap is 64 bits, and so the BDB should be a 64 bits one… but that value is too near to the 32 bits’ 4 GiB limit to just forget it without mentioning.

# uname -a
Linux 2.6.18-164.9.1 #1 SMP x86_64 x86_64 x86_64 GNU/Linux
# rpm -qa | grep openldap
...
openldap-servers-2.3.43-3
...
# rpm -ql openldap-servers-2.3.43-3
...
/usr/lib64/libslapd_db-4.4.so
/usr/lib64/tls/libslapd_db-4.4.so
...
/usr/sbin/slapadd
...
/usr/sbin/slapd

# file /usr/sbin/slapadd
/usr/sbin/slapadd: ELF 64-bit LSB shared object, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.6.9, stripped
# file /usr/lib64/libslapd_db-4.4.so
/usr/lib64/libslapd_db-4.4.so: ELF 64-bit LSB shared object, AMD x86-64, version 1 (SYSV), stripped

And a file to one db’s file:
# file id2entry.bdb
id2entry.bdb: Berkeley DB (Btree, version 9, native byte-order)

Second possible solution: may be some configuration of buffers and caches could improve performance and avoid the error.
BDB usually uses directories as its DB paths, and the BDB_CONFIG file (if any) found in the directory is read for BDB configuration options of that database.
As it is suggested here, the size of BDB cache defined in the DB_CONFIG file should be the sum of all *.bdb files. In my case they sum 700 MiB, and I think it talks about “set_cachesize“.
Nonetheless, I think I played enough with it without success.
So I played with “set_lg_bsize“, but results didn’t lead to success: this just made the process advance a little further… a mere pair of ldif entries, not more.

In the mean time I tried to ‘slapadd’ while periodically checkpointing, and even loading the ldif split in some parts… Even removed sections from the ldif file in case the problem were caused by particular entries. These efforts didn’t lead to a complete solution.

So I gave up and remembered that now the benefit of quick download without db checking burden is obtained directly with the “-q” parameter on slapadd and slapindex, as the “/etc/openldap/DB_CONFIG.example” file clearly states:
# Note: special DB_CONFIG flags are no longer needed for "quick"
# slapadd(8) or slapindex(8) access (see their -q option).

This way, with slapadd -q option, and “in memory” logging (DB_LOG_INMEMORY), the ldif could be finally loaded after half an hour of processing, and almost no use of __db.* BDB files on disk.

Then I started the openldap server, and came the next step, the problem that started it all: could this replica correctly synchronize itself with the master?
After almost an hour of heavy disk load and 100 KiB/s traffic… it did!
It seemed that the openldap replica traversed all the master ldap tree.
And then I remembered that the ldif didn’t contain entryCSN, contextCSN and entryUUID values…
Next time I’d try to remember to include this operational attributes in my ldif exports.

Anyway, the error showed by BDB remains unexplained…


bdb(ou=**********): In-memory log buffer is full (an active transaction spans the buffer)
=> bdb_idl_insert_key: c_get next_dup failed: DB_LOG_BUFFER_FULL: In-memory log buffer is full (-30993)
=> bdb_tool_entry_put: index_entry_add failed: Accessing a corrupted shared library (80)
=> bdb_tool_entry_put: txn_aborted! Accessing a corrupted shared library (80)
slapadd: could not add entry dn="**********" (line=3179771): txn_aborted! Accessing a corrupted shared library (80)


Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s