2 of 3

Package and RPM Database RepairLesson 2 of 3

The RPM database, signatures and GPG keys

Understand where RPM keeps its state, check package signatures, import keys and recover from an unreadable database or a stale lock.

  • rpm -K
  • rpm --import
  • rpm -qa gpg-pubkey*
  • rpm --rebuilddb
  • rpm -qa --last
  • rpm -qi
  • dnf check
  • /var/lib/rpm
  • rpmkeys
  • rpm --checksig

40 min3 exercises

After this lesson you can

  • Locate the RPM database and explain what is stored there and what is not.
  • Check a package's integrity and signature, import the right key and explain a NOKEY result.
  • Rebuild the database, and know when a rebuild cannot help and a backup or reinstall is required.

The idea

RPM remembers what is installed in a small database. That database is the answer to “what packages do I have, which files did they bring, and what do they depend on?” Everything from rpm -qf to dnf update reads it, and nothing works well if it is unreadable.

WhatWhereNote
The database/var/lib/rpm/rpmdb.sqliteAn SQLite file on RHEL 9 and 10 (RHEL 8 used Berkeley DB). Never edit it by hand
Write-ahead filesrpmdb.sqlite-wal, -shmPresent while it is in use; belong with the database
Lock/var/lib/rpm/.rpm.lockOne rpm/dnf at a time
Imported GPG keysIn the database as gpg-pubkey-*The key files themselves are under /etc/pki/rpm-gpg/
DNF’s own history/var/lib/dnf/history.sqliteTransactions, separate from the RPM database
DNF’s repository cache/var/cache/dnf/Safe to delete: dnf clean all

The database records installed packages and their file lists, not the files’ contents. It is not a backup of your system, and it is only as trustworthy as the machine it lives on.

Integrity and signatures

Two different checks apply to a package file (.rpm):

  • Digests (SHA-256 of the header and payload) detect accidental damage or truncation.
  • Signatures (GPG) say who built it. They are checked against a public key you have imported.
rpm -K package.rpm
ResultMeaning
digests signatures OKIntact and signed by a trusted key
digests OKIntact, but unsigned (or signature not checked)
digests SIGNATURES NOT OK (and NOKEY with -Kv)Signed, but you have not imported the key (older versions print (MISSING KEYS: ...))
... BAD / FAILEDCorrupt or modified: do not install

dnf checks the signatures of packages when the repository has gpgcheck=1 (the default for RHEL repositories) and, with repo_gpgcheck=1, of the repository metadata too. On a mismatch it stops with GPG check FAILED or Public key for ... is not installed.

sudo rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release     # trust a key
rpm -qa 'gpg-pubkey*'                                              # what is trusted

Try it

The following outputs are real, from rpm 4.18 on a test package (labapp) and a private database.

A healthy check

rpm -K labapp-1.0-1.noarch.rpm
rpm -qa --last
labapp-1.0-1.noarch.rpm: digests OK
labapp-1.0-1.noarch                           Wed Nov  4 17:23:28 2026

The test package is unsigned, so the result is digests OK. A package from Red Hat repositories on a system with the key imported ends with digests signatures OK (representative: the wording is ...: digests signatures OK).

A signed package, with and without the key

A test package was signed with a freshly created key (rpmsign --addsign), and checked against an empty key database, then after importing the public key. Real output:

rpm -K signed.rpm
rpm -Kv signed.rpm
rpm --import RPM-GPG-KEY-lab
rpm -qa 'gpg-pubkey*' --qf '%{NAME}-%{VERSION}-%{RELEASE} %{SUMMARY}\n'
rpm -K signed.rpm
signed.rpm: digests SIGNATURES NOT OK
signed.rpm:
    Header V4 RSA/SHA512 Signature, key ID 1a087f40: NOKEY
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    MD5 digest: OK
gpg-pubkey-1a087f40-6aeb4388 Lab Packager <pkg@lab.example> public key
signed.rpm: digests signatures OK

The short form says only that something is not OK; the verbose form (-Kv) shows what: all the digests are fine (the file is intact) and only the signature has NOKEY, meaning “I cannot judge this signature, I do not have the key”. Importing the key changes the result to digests signatures OK. The key appears in the database as a gpg-pubkey-<keyid>-<date> pseudo-package.

rpm -qa --last lists installed packages newest first with the install time: the quickest way to answer “what changed recently?” after a problem begins.

Rebuilding the database

rpm --rebuilddb; echo "rc=$?"
rpm -qa
rc=0
labapp-1.0-1.noarch

rpm --rebuilddb rebuilds the indexes from the package headers stored in the database. It needs a database that can still be read. It fixes things like a stale index, a missing lock file, or inconsistent index rows. A silent success (no output and exit status 0) is the normal result.

When the file itself is damaged

A real test: the database file was truncated to 3,000 bytes. No indexes and no rebuild can recover headers that are no longer there.

rpm -qa
rpm --rebuilddb
error: sqlite failure: CREATE TABLE IF NOT EXISTS 'Packages' (hnum INTEGER PRIMARY KEY AUTOINCREMENT,blob BLOB NOT NULL): database disk image is malformed
error: cannot open Packages index using sqlite - No such file or directory (2)
error: cannot open Packages database in /var/lib/rpm

database disk image is malformed is SQLite’s way of saying the file is corrupt. This is different from a permission or lock problem, and the repair differs too:

Error textCauseRepair
cannot open Packages database with Permission deniedMode or SELinux label of /var/lib/rpmchmod, restorecon -Rv /var/lib/rpm
waiting for transaction lockAnother rpm/dnf is runningWait, or find it: ps aux | grep -E 'dnf|rpm'; remove a stale lock only if no process exists
Strange query results, missing entriesInconsistent indexesrpm --rebuilddb
database disk image is malformedThe file is damagedRestore from backup or VM/LVM snapshot; as a last resort rebuild the package list by reinstalling the system’s packages from a known list

The only protection against the last case is a backup. Copy /var/lib/rpm before large operations on a machine you cannot afford to rebuild (cp -a /var/lib/rpm /root/rpm.backup), and take snapshots before major updates.

An interrupted transaction

If dnf is interrupted (power loss, Ctrl+C, network drop) some packages may be half installed. Signs: dnf check prints problems, rpm -V shows missing files, or the same package appears twice in rpm -q with two versions (duplicates).

sudo dnf check
sudo dnf history | head -5
sudo dnf install -y package          # re-run the original operation
sudo dnf history list --reverse | tail -3

The documentation-recommended order is: look for a running process, run dnf check, finish or redo the transaction, and only then reach for rpm --rebuilddb. dnf history list shows transaction IDs; dnf history info ID shows exactly which packages were touched.

Going further

Duplicate packages. A package listed in two versions (rpm -q kernel aside, where several are normal) usually means a transaction stopped between install and removal of the old one. package-cleanup --dupes (from dnf-utils) finds them; dnf remove --duplicates removes the older copies. Beyond the exam: rpmkeys --checksig is the lower-level tool dnf uses, and RHEL 10’s stricter crypto policy can reject packages signed with weak algorithms (update-crypto-policies --show).

Practice

0 of 3 done

Do these on your own system. They get harder. Check each result, then tick it off.

  1. Warm-up

    Which keys does this system trust?

    List the GPG public keys that RPM trusts, with their names and when they were imported. Identify the Red Hat release key, and find where on disk the key files for the repositories are kept.

    Check your result
    • rpm -qa 'gpg-pubkey*' lists at least one entry, and rpm -qi gpg-pubkey-... shows a Summary naming the key owner.
    • ls /etc/pki/rpm-gpg/ contains RPM-GPG-KEY-redhat-release.
    Hint

    Imported keys are stored in the RPM database as pseudo-packages called gpg-pubkey-*. The files dnf refers to with gpgkey=file://... are under /etc/pki/rpm-gpg/.

    Solution
    rpm -qa 'gpg-pubkey*' --qf '%{NAME}-%{VERSION}-%{RELEASE}  %{SUMMARY}\n'
    ls -l /etc/pki/rpm-gpg/
    grep -rn gpgkey /etc/yum.repos.d/ | head
  2. Practice

    Is this RPM genuine?

    Download a package without installing it (dnf download chrony), check its digests and signature, and explain the result. Then (on a lab system only) remove the Red Hat key from the RPM database, repeat the check and note the new message, and import the key again from /etc/pki/rpm-gpg/.

    Check your result
    • Before: rpm -K chrony-*.rpm ends with digests signatures OK.
    • Without the key: rpm -K says digests SIGNATURES NOT OK and rpm -Kv shows NOKEY on the signature line.
    • After sudo rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release, the first result returns.
    Hint

    rpm -K is the short form of rpm --checksig. The key is the package named gpg-pubkey-<id>-<date>; remove it with rpm -e. Import brings it back.

    Solution
    cd /tmp && dnf download chrony
    rpm -K chrony-*.rpm
    KEY=$(rpm -qa 'gpg-pubkey*' | head -1)
    sudo rpm -e "$KEY"
    rpm -K chrony-*.rpm
    sudo rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
    rpm -K chrony-*.rpm
  3. Challenge

    rpm cannot open its database

    Setup (disposable VM only):

    sudo cp -a /var/lib/rpm /root/rpm.backup
    sudo bash -c 'echo > /var/lib/rpm/.rpm.lock; chmod 000 /var/lib/rpm/rpmdb.sqlite-wal 2>/dev/null; true'

    then simulate a stale transaction: start sudo dnf install -y tree and press Ctrl+C in the middle. Next, rpm -qa and dnf behave oddly. Using evidence, decide what is wrong at each step, repair in the least destructive order (permissions and stale lock, dnf check, rpm --rebuilddb, dnf history), and write down the one situation where only the backup helps.

    Check your result
    • rpm -qa | wc -l prints a number, dnf check prints nothing.
    • dnf history | head -3 shows the transaction you interrupted as failed or incomplete, and you can name the next step (dnf history redo/undo or dnf install again).
    • Your notes say: if rpmdb.sqlite itself is malformed and --rebuilddb cannot read it, restore from backup or snapshot.
    Hint

    Restore the permission you broke first (chmod 600), then check processes holding a lock: ps aux | grep -E 'dnf|rpm'. A real stale transaction shows in dnf history and dnf check. rpm --rebuilddb rebuilds the indexes from the package headers; it cannot invent headers that are unreadable.

    Solution
    ls -l /var/lib/rpm/
    sudo chmod 600 /var/lib/rpm/rpmdb.sqlite*
    ps aux | grep -E '[d]nf|[r]pm'
    sudo dnf check
    sudo rpm --rebuilddb
    rpm -qa | wc -l
    sudo dnf history | head -5
    sudo dnf install -y tree
    # if the database file is malformed: restore /root/rpm.backup (or a snapshot) and run rpm --rebuilddb