Verifying and repairing installed packages

The idea

Every installed package leaves a record in the RPM database: each file it owns, with its expected size, mode, owner, digest (a SHA-256 hash on current releases) and modification time. Verification compares that record with the files on disk. It does not tell you why something changed, but it tells you what changed, which is the first question in most troubleshooting.

rpm -V PACKAGE      # verify one package
rpm -Va             # verify everything (slow, run it when idle)
rpm -Vf /path       # verify the package that owns this file

No output means everything matches. Output means differences.

Reading a result line

SM5DLUGTP  c /path/to/file

The nine characters are fixed positions; a dot means “matches”, a letter means “differs”.

Pos	Letter	Differs in
1	S	file size
2	M	mode (permissions and type)
3	5	digest (content)
4	D	device major/minor
5	L	symlink target
6	U	owner
7	G	group
8	T	modification time
9	P	capabilities

An optional tenth column adds . or ? for SELinux and other attributes depending on the RPM version. Then the file-type letter: c configuration, d documentation, l licence, g ghost, r readme, blank for an ordinary file. The word missing replaces the nine characters when the file does not exist.

How to judge a line:

Pattern	Usually means
S.5....T. c /etc/...	A configuration file edited on purpose
.M....... /usr/bin/...	Mode changed: chmod, or an installer script; compare with rpm -qlv
SM5....T. /usr/bin/...	Binary replaced or modified: a manually copied update, damage, or tampering
missing d /usr/share/doc/...	Documentation removed; harmless to function
.......T. /usr/lib/...	Only the timestamp: touched or restored, usually harmless
..5...... /etc/... with no c	A non-config file whose content changed: investigate

Configuration (c) differences are expected and should be documented, not “repaired”. Everything else needs an explanation.

Which package owns it

rpm -qf /usr/bin/labapp        # owning package
rpm -ql labapp                 # all files; add -v for modes
rpm -qc labapp                 # configuration files
rpm -qd labapp                 # documentation files
rpm -q --scripts labapp        # install/remove scriptlets (what the package does when it runs)

A file that belongs to no package (file ... is not owned by any package) is something an administrator or a script put there. That is not an error by itself, but it is outside what verification covers.

Repair options

Damage	Restore with	Notes
Modes or owners changed	rpm --setperms PKG and rpm --setugids PKG	Restores attributes without touching content
A binary or library modified or missing	dnf reinstall PKG	Re-extracts all files; keeps modified %config(noreplace) files
One file, nothing else disturbed	Extract it from the package: rpm2cpio PKG.rpm | cpio -idmv ./path	Copy the file out of the temporary directory
A configuration file	Extract the packaged copy, diff, then copy or merge	Reinstall will not restore a modified config
SELinux labels wrong	restorecon -Rv PATH	Labels are not part of rpm’s file digest checks the same way

dnf download PKG fetches the .rpm without installing it, from the same repositories.

Try it

This lesson’s outputs are real, from rpm 4.18 on a package built for the test (labapp, with a binary, a %config(noreplace) file and a document).

Verification after damage

Three changes were made: a line appended to the config, the binary replaced and its mode changed, and the README deleted.

rpm -V labapp; echo "rc=$?"

S.5......  c /etc/labapp/labapp.conf
SM5......    /usr/bin/labapp
missing   d /usr/share/doc/labapp/README
rc=1

The exit code is 1 when something differs, which makes rpm -V usable in scripts. Reading the three lines: the config differs in size and digest (an edit), the binary differs in size, mode and content (replaced), and the doc is missing. Note that the config line has the c type letter and the other two do not (doc has d).

Ask the database

rpm -qf /usr/bin/labapp
rpm -qc labapp
rpm -qd labapp

labapp-1.0-1.noarch
/etc/labapp/labapp.conf
/usr/share/doc/labapp/README

Pull one pristine file out of the package

mkdir ex && cd ex
rpm2cpio ../labapp-1.0-1.noarch.rpm | cpio -idm
ls -l usr/bin/labapp
cat usr/bin/labapp

-rwxr-xr-x 1 root root 26 Nov  4 17:23 usr/bin/labapp
#!/bin/sh
echo labapp 1.0

This gives you the original file and its mode, without installing anything. Use it to diff a config or to restore a single file.

Reinstall

rpm -Uvh --force labapp-1.0-1.noarch.rpm
rpm -V labapp; echo "rc=$?"

labapp-1.0-1                          ########################################
Cleaning up / removing...
labapp-1.0-1                          ########################################
rc=0

(On RHEL you use dnf reinstall labapp, which does the same through the repositories.) The binary and the README came back. A test where only the config had been modified, then reinstalled, produced this:

cat /etc/labapp/labapp.conf

port=8080
mode=prod
mode=debug

The appended mode=debug is still there: the reinstall kept the modified configuration file. That is the noreplace protection doing its job, and it is exactly why a reinstall does not “reset” a configuration. To get the packaged version, extract it (above) and copy it over, or look for a .rpmnew file next to the original (ls /etc/labapp/) when a newer package has different defaults.

When repair does not work

Symptom	Cause	Fix
file ... is not owned by any package	Locally installed file	Not a package problem; check where it came from
dnf reinstall says package not available	Repository lacks that exact version	dnf download PKG-VERSION from another repo, or rpm -Uvh --force the file you have
Reinstall succeeded, config still wrong	%config(noreplace) kept your file	Extract and copy the packaged file
Binary works, SELinux denies it	Label wrong, not the content	restorecon -Rv, ls -Z
rpm -V shows .......T. on many files	Timestamps only (copy/restore)	Usually ignorable
Cannot reinstall, no repository	Offline	Boot installation media or use the local repository from the next lesson

Going further

Beyond the exam. rpm -Va is a point-in-time comparison against a database on the same machine. If an attacker can write to /var/lib/rpm, the comparison can be made to say anything; AIDE with a baseline kept elsewhere (see the file integrity lesson) covers that case. Use rpm -V for accidents, AIDE for intrusion.