Package and RPM Database RepairLesson 2 of 3
The RPM database, signatures and GPG keys
Understand where RPM keeps its state, check package signatures, import keys and recover from an unreadable database or a stale lock.
rpm -Krpm --importrpm -qa gpg-pubkey*rpm --rebuilddbrpm -qa --lastrpm -qidnf check/var/lib/rpmrpmkeysrpm --checksig
After this lesson you can
- Locate the RPM database and explain what is stored there and what is not.
- Check a package's integrity and signature, import the right key and explain a NOKEY result.
- Rebuild the database, and know when a rebuild cannot help and a backup or reinstall is required.
The idea
RPM remembers what is installed in a small database. That database is the answer to “what packages do I have, which files did they bring, and what do they depend on?” Everything from rpm -qf to dnf update reads it, and nothing works well if it is unreadable.
| What | Where | Note |
|---|---|---|
| The database | /var/lib/rpm/rpmdb.sqlite | An SQLite file on RHEL 9 and 10 (RHEL 8 used Berkeley DB). Never edit it by hand |
| Write-ahead files | rpmdb.sqlite-wal, -shm | Present while it is in use; belong with the database |
| Lock | /var/lib/rpm/.rpm.lock | One rpm/dnf at a time |
| Imported GPG keys | In the database as gpg-pubkey-* | The key files themselves are under /etc/pki/rpm-gpg/ |
| DNF’s own history | /var/lib/dnf/history.sqlite | Transactions, separate from the RPM database |
| DNF’s repository cache | /var/cache/dnf/ | Safe to delete: dnf clean all |
The database records installed packages and their file lists, not the files’ contents. It is not a backup of your system, and it is only as trustworthy as the machine it lives on.
Integrity and signatures
Two different checks apply to a package file (.rpm):
- Digests (SHA-256 of the header and payload) detect accidental damage or truncation.
- Signatures (GPG) say who built it. They are checked against a public key you have imported.
rpm -K package.rpm
| Result | Meaning |
|---|---|
digests signatures OK | Intact and signed by a trusted key |
digests OK | Intact, but unsigned (or signature not checked) |
digests SIGNATURES NOT OK (and NOKEY with -Kv) | Signed, but you have not imported the key (older versions print (MISSING KEYS: ...)) |
... BAD / FAILED | Corrupt or modified: do not install |
dnf checks the signatures of packages when the repository has gpgcheck=1 (the default for RHEL repositories) and, with repo_gpgcheck=1, of the repository metadata too. On a mismatch it stops with GPG check FAILED or Public key for ... is not installed.
sudo rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release # trust a key
rpm -qa 'gpg-pubkey*' # what is trusted
Try it
The following outputs are real, from rpm 4.18 on a test package (labapp) and a private database.
A healthy check
rpm -K labapp-1.0-1.noarch.rpm
rpm -qa --last
labapp-1.0-1.noarch.rpm: digests OK
labapp-1.0-1.noarch Wed Nov 4 17:23:28 2026
The test package is unsigned, so the result is digests OK. A package from Red Hat repositories on a system with the key imported ends with digests signatures OK (representative: the wording is ...: digests signatures OK).
A signed package, with and without the key
A test package was signed with a freshly created key (rpmsign --addsign), and checked against an empty key database, then after importing the public key. Real output:
rpm -K signed.rpm
rpm -Kv signed.rpm
rpm --import RPM-GPG-KEY-lab
rpm -qa 'gpg-pubkey*' --qf '%{NAME}-%{VERSION}-%{RELEASE} %{SUMMARY}\n'
rpm -K signed.rpm
signed.rpm: digests SIGNATURES NOT OK
signed.rpm:
Header V4 RSA/SHA512 Signature, key ID 1a087f40: NOKEY
Header SHA256 digest: OK
Header SHA1 digest: OK
Payload SHA256 digest: OK
MD5 digest: OK
gpg-pubkey-1a087f40-6aeb4388 Lab Packager <pkg@lab.example> public key
signed.rpm: digests signatures OK
The short form says only that something is not OK; the verbose form (-Kv) shows what: all the digests are fine (the file is intact) and only the signature has NOKEY, meaning “I cannot judge this signature, I do not have the key”. Importing the key changes the result to digests signatures OK. The key appears in the database as a gpg-pubkey-<keyid>-<date> pseudo-package.
rpm -qa --last lists installed packages newest first with the install time: the quickest way to answer “what changed recently?” after a problem begins.
Rebuilding the database
rpm --rebuilddb; echo "rc=$?"
rpm -qa
rc=0
labapp-1.0-1.noarch
rpm --rebuilddb rebuilds the indexes from the package headers stored in the database. It needs a database that can still be read. It fixes things like a stale index, a missing lock file, or inconsistent index rows. A silent success (no output and exit status 0) is the normal result.
When the file itself is damaged
A real test: the database file was truncated to 3,000 bytes. No indexes and no rebuild can recover headers that are no longer there.
rpm -qa
rpm --rebuilddb
error: sqlite failure: CREATE TABLE IF NOT EXISTS 'Packages' (hnum INTEGER PRIMARY KEY AUTOINCREMENT,blob BLOB NOT NULL): database disk image is malformed
error: cannot open Packages index using sqlite - No such file or directory (2)
error: cannot open Packages database in /var/lib/rpm
database disk image is malformed is SQLite’s way of saying the file is corrupt. This is different from a permission or lock problem, and the repair differs too:
| Error text | Cause | Repair |
|---|---|---|
cannot open Packages database with Permission denied | Mode or SELinux label of /var/lib/rpm | chmod, restorecon -Rv /var/lib/rpm |
waiting for transaction lock | Another rpm/dnf is running | Wait, or find it: ps aux | grep -E 'dnf|rpm'; remove a stale lock only if no process exists |
| Strange query results, missing entries | Inconsistent indexes | rpm --rebuilddb |
database disk image is malformed | The file is damaged | Restore from backup or VM/LVM snapshot; as a last resort rebuild the package list by reinstalling the system’s packages from a known list |
The only protection against the last case is a backup. Copy /var/lib/rpm before large operations on a machine you cannot afford to rebuild (cp -a /var/lib/rpm /root/rpm.backup), and take snapshots before major updates.
An interrupted transaction
If dnf is interrupted (power loss, Ctrl+C, network drop) some packages may be half installed. Signs: dnf check prints problems, rpm -V shows missing files, or the same package appears twice in rpm -q with two versions (duplicates).
sudo dnf check
sudo dnf history | head -5
sudo dnf install -y package # re-run the original operation
sudo dnf history list --reverse | tail -3
The documentation-recommended order is: look for a running process, run dnf check, finish or redo the transaction, and only then reach for rpm --rebuilddb. dnf history list shows transaction IDs; dnf history info ID shows exactly which packages were touched.
Going further
Duplicate packages. A package listed in two versions (rpm -q kernel aside, where several are normal) usually means a transaction stopped between install and removal of the old one. package-cleanup --dupes (from dnf-utils) finds them; dnf remove --duplicates removes the older copies. Beyond the exam: rpmkeys --checksig is the lower-level tool dnf uses, and RHEL 10’s stricter crypto policy can reject packages signed with weak algorithms (update-crypto-policies --show).
Practice
0 of 3 done
Do these on your own system. They get harder. Check each result, then tick it off.
- Warm-up
Which keys does this system trust?
List the GPG public keys that RPM trusts, with their names and when they were imported. Identify the Red Hat release key, and find where on disk the key files for the repositories are kept.
Check your result
rpm -qa 'gpg-pubkey*'lists at least one entry, andrpm -qi gpg-pubkey-...shows aSummarynaming the key owner.ls /etc/pki/rpm-gpg/containsRPM-GPG-KEY-redhat-release.
Hint
Imported keys are stored in the RPM database as pseudo-packages called
gpg-pubkey-*. The filesdnfrefers to withgpgkey=file://...are under/etc/pki/rpm-gpg/.Solution
rpm -qa 'gpg-pubkey*' --qf '%{NAME}-%{VERSION}-%{RELEASE} %{SUMMARY}\n' ls -l /etc/pki/rpm-gpg/ grep -rn gpgkey /etc/yum.repos.d/ | head - Practice
Is this RPM genuine?
Download a package without installing it (
dnf download chrony), check its digests and signature, and explain the result. Then (on a lab system only) remove the Red Hat key from the RPM database, repeat the check and note the new message, and import the key again from/etc/pki/rpm-gpg/.Check your result
- Before:
rpm -K chrony-*.rpmends withdigests signatures OK. - Without the key:
rpm -Ksaysdigests SIGNATURES NOT OKandrpm -KvshowsNOKEYon the signature line. - After
sudo rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release, the first result returns.
Hint
rpm -Kis the short form ofrpm --checksig. The key is the package namedgpg-pubkey-<id>-<date>; remove it withrpm -e. Import brings it back.Solution
cd /tmp && dnf download chrony rpm -K chrony-*.rpm KEY=$(rpm -qa 'gpg-pubkey*' | head -1) sudo rpm -e "$KEY" rpm -K chrony-*.rpm sudo rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release rpm -K chrony-*.rpm - Before:
- Challenge
rpm cannot open its database
Setup (disposable VM only):
sudo cp -a /var/lib/rpm /root/rpm.backup sudo bash -c 'echo > /var/lib/rpm/.rpm.lock; chmod 000 /var/lib/rpm/rpmdb.sqlite-wal 2>/dev/null; true'then simulate a stale transaction: start
sudo dnf install -y treeand pressCtrl+Cin the middle. Next,rpm -qaanddnfbehave oddly. Using evidence, decide what is wrong at each step, repair in the least destructive order (permissions and stale lock,dnf check,rpm --rebuilddb,dnf history), and write down the one situation where only the backup helps.Check your result
rpm -qa | wc -lprints a number,dnf checkprints nothing.dnf history | head -3shows the transaction you interrupted as failed or incomplete, and you can name the next step (dnf history redo/undoordnf installagain).- Your notes say: if
rpmdb.sqliteitself is malformed and--rebuilddbcannot read it, restore from backup or snapshot.
Hint
Restore the permission you broke first (
chmod 600), then check processes holding a lock:ps aux | grep -E 'dnf|rpm'. A real stale transaction shows indnf historyanddnf check.rpm --rebuilddbrebuilds the indexes from the package headers; it cannot invent headers that are unreadable.Solution
ls -l /var/lib/rpm/ sudo chmod 600 /var/lib/rpm/rpmdb.sqlite* ps aux | grep -E '[d]nf|[r]pm' sudo dnf check sudo rpm --rebuilddb rpm -qa | wc -l sudo dnf history | head -5 sudo dnf install -y tree # if the database file is malformed: restore /root/rpm.backup (or a snapshot) and run rpm --rebuilddb