← Back

Domain 5, Round 2: Package Management Internals — States, Locks, Failures, and Recovery

Mar 17, 2026

This is Round 2 of Domain 5 in my Linux Mastery: Road to Cloud series. If Round 1 was about learning what package management tools exist and how to use them, Round 2 is about understanding what’s actually happening underneath — where files live, what states packages go through, what breaks and why, and how to recover when things go wrong.

By the end of this round, the goal isn’t to memorize every command and path cold. The goal is to know what layer to look at when something breaks, what vocabulary to use, and what questions to ask. The paths and exact syntax stick naturally through real use — not through memorization.

What Round 2 Actually Covers

Round 1 covered the basics: installing, removing, updating packages with APT, and understanding what dpkg is. Round 2 goes deeper into five areas:

APT internals — what actually happens when you run apt update, where the indexes live, how APT decides which version to install, and what the cache is doing.

dpkg internals — the package state machine (unpacked → configured → installed), the status database, what the info directory contains, and how to read status codes properly.

Failure states and recovery — lock files, partial installs, half-configured packages, broken dependencies, and what to do when apt refuses to run.

Repository and security management — GPG keys, sources.list structure, the modern way to add third-party repos, pinning, and priority logic.

Real operational awareness — unattended-upgrades, network failure behavior, stale metadata, and what changes after a libc upgrade on a running server.

The Filesystem Layer — Paths You Need to Know

Before touching commands, I walked these paths on my actual server. Theory becomes real the moment you see what’s actually inside these directories.

1. /var/lib/apt/lists/

ls /var/lib/apt/lists/

This is where APT stores the package indexes downloaded by apt update. The files are named after the repo URLs they came from. When you search for a package or check what version is available, APT reads from here — not from the internet. If these files are stale or a repo was unreachable during your last update, APT is working from old data without telling you.

2. /var/cache/apt/archives/

ls /var/cache/apt/archives/
du -sh /var/cache/apt/archives/

Every .deb file APT downloads before installing sits here first. On a server that has been running for a while, this can quietly consume significant disk space. This is what apt clean wipes. Run apt autoclean instead if you want to only remove packages that can no longer be downloaded (i.e., outdated versions).

3. /etc/apt/sources.list and /etc/apt/sources.list.d/

cat /etc/apt/sources.list
ls /etc/apt/sources.list.d/

Your repo configuration lives here. The sources.list.d/ directory is where every third-party tool you add drops its own file — Docker, Nginx official, GitHub CLI, all of them. This is where repo debt accumulates. If you’ve added repos and forgotten about them, this is where you find them. Always prefer adding repos via sources.list.d/ rather than editing sources.list directly — it keeps things isolated and easier to remove cleanly.

4. /etc/apt/trusted.gpg.d/

ls /etc/apt/trusted.gpg.d/

The signing keys for your repos live here, one .gpg or .asc file per trusted repository. APT refuses to use a repo if the key isn’t here. The old way was apt-key add, which dumped everything into a single shared keyring — a security problem because any trusted key could sign packages from any repo. The modern approach stores keys separately, scoped to the repo they’re for.

5. /var/lib/dpkg/status — the most important file in this domain

grep -A 10 "^Package: nginx" /var/lib/dpkg/status

This file is the single source of truth for what dpkg believes is installed on your system. Every package, its version, its dependencies, and its current state is recorded here. APT reads this file too. If this gets corrupted, your entire package management system breaks. It’s worth knowing it exists and what it looks like — not to edit it manually, but to understand why certain dpkg commands behave the way they do.

6. /var/lib/dpkg/info/

ls /var/lib/dpkg/info/ | grep nginx
cat /var/lib/dpkg/info/nginx.list | head -20

For every installed package there’s a set of files in here: a .list file containing every file the package owns, and scripts like .postinst, .prerm, .postrm that run at install/remove time. When you run dpkg -L nginx, it reads from here. When you run dpkg -S /usr/sbin/nginx to find which package owns a file, it searches these lists.

7. /var/lib/dpkg/lock and /var/lib/apt/lists/lock

ls -la /var/lib/dpkg/lock
lsof /var/lib/dpkg/lock

These lock files prevent two apt/dpkg processes from running simultaneously and corrupting the package database. The most common junior mistake is to kill the process holding the lock to “free it up.” If that process was in the middle of unpacking or configuring a package, you’ve now left your system in a broken half-installed state. Always use lsof to identify what’s holding the lock, then wait for it to finish — or verify it’s genuinely dead before doing anything else.

8. /var/log/apt/history.log

tail -50 /var/log/apt/history.log

Every install, remove, and upgrade is logged here with a timestamp and the exact package versions. When something breaks after an upgrade and you need to know what changed, this is the first place to look. On a production server this log is your audit trail.

9. /var/log/dpkg.log

tail -50 /var/log/dpkg.log

This is the lower-level record of every dpkg operation, including the phase transitions — unpacked, half-configured, configured, installed. If a package got stuck in a partial state, the exact phase where it failed is recorded here. This is where you look when apt install exits mid-way and leaves something broken.

10. /etc/apt/preferences.d/

ls /etc/apt/preferences.d/
cat /etc/apt/preferences 2>/dev/null || echo "no preferences file"

APT pinning configuration lives here. On most servers this directory is empty, which means APT uses its default priority system (500 for standard repos). The moment you need to freeze a package at a specific version on a production server — or force APT to prefer one repo over another — this is where you configure it.

The Commands Layer — 32 Commands Organized by Purpose

These aren’t commands to memorize. They’re commands to run at least once, read the output of, and understand what question each one answers. That’s enough.

Inspection — Reading System State

apt-cache policy <package>        # installed version, candidate version, repo priorities
apt-cache show <package>          # full package metadata
apt-cache madison <package>       # all available versions from all repos
apt-cache rdepends <package>      # what other packages depend on this one
dpkg -l                           # list all packages with status codes
dpkg -l | grep "^rc"              # find removed-but-not-purged packages
dpkg -L <package>                 # list all files owned by a package
dpkg -S <file>                    # which package owns this file
dpkg --get-selections              # export current package selections
apt-mark showhold                  # list packages currently on hold
apt list --upgradable              # show what has pending updates

State Management — Changing Package State

apt-mark hold <package>           # freeze a package at its current version
apt-mark unhold <package>         # release the freeze
apt install -f                     # fix broken dependency state
apt --fix-broken install           # same thing, APT layer version
dpkg --configure -a                # finish configuring any unpacked packages
apt install --reinstall <package>  # reinstall without removing first
apt purge <package>               # remove package AND its config files
apt autoremove                     # remove packages no longer needed by anything

Troubleshooting — Diagnosing Broken States

lsof /var/lib/dpkg/lock           # who is holding the dpkg lock
lsof /var/lib/apt/lists/lock      # who is holding the apt lists lock
fuser /var/lib/dpkg/lock          # alternative to lsof for finding lock holder
apt-get --simulate upgrade         # dry run — shows what would change without doing it
dpkg --audit                       # find partially installed or broken packages

Repository and Version Management

apt install <package>=<version>   # install a specific version
dpkg -i <file>.deb                # install a manually downloaded .deb file
dpkg -c <file>.deb                # inspect contents of a .deb without installing
dpkg-reconfigure <package>        # re-run a package's configuration wizard
apt full-upgrade                   # upgrade including handling dependency changes
ls /etc/apt/trusted.gpg.d/        # check which repo signing keys are trusted

Understanding Package States

This is one of the most important concepts in Round 2. dpkg doesn’t install a package in one step — it goes through phases, and failures can happen between them.

The normal flow looks like this:

config-files   → package was removed but config files kept
unpacked       → files extracted, post-install script not yet run
half-configured → post-install script started but didn't complete
installed      → fully done

When you see half-configured in a log, it doesn’t automatically mean something is broken. During a healthy install, dpkg logs half-configured as a checkpoint while the configuration script is running — you’ll see it followed immediately by installed. That’s normal.

It only means something is broken if the log ends on half-configured or unpacked without ever reaching installed. In that case, run:

sudo dpkg --configure -a

That command attempts to finish configuring everything that’s stuck in an incomplete state.

APT Priority System

When the same package exists in multiple repos at different versions, APT doesn’t pick randomly. It uses a priority system:

  • 100 — an already-installed package version
  • 500 — packages from a standard configured repo (default)
  • 990 — packages from a repo marked as the target release
  • 1000+ — pinned packages that APT will install even if it means a downgrade
  • Below 0 — APT will never automatically select this version

The candidate version is the version APT has selected to install based on these priorities. You can inspect it with:

apt-cache policy <package>

Look for the line marked Candidate:. If you want to override APT’s choice — for example, to keep a specific version on a production server — you configure that in /etc/apt/preferences.d/.

Lock Files — The Most Common Operational Failure

The single most frequent package management failure on real servers is this error:

E: Could not get lock /var/lib/dpkg/lock-frontend

This happens when another process is already using APT or dpkg — most commonly unattended-upgrades running in the background. The correct response is:

# Step 1 — find what's holding the lock
sudo lsof /var/lib/dpkg/lock
sudo lsof /var/lib/dpkg/lock-frontend

# Step 2 — if it's a legitimate process, wait for it to finish
# Step 3 — if the process is genuinely dead (e.g. after a crash), THEN you can remove the lock
sudo rm /var/lib/dpkg/lock
sudo rm /var/lib/dpkg/lock-frontend
sudo dpkg --configure -a

The dangerous path is killing the process holding the lock without checking what it was doing. If it was mid-install, you’re now left with a half-unpacked package and a broken dpkg state.

unattended-upgrades — What’s Running in the Background

Ubuntu enables unattended-upgrades by default. It runs automatically and applies security updates without you doing anything. This is the correct behavior for a server — security patches shouldn’t wait for a human to remember to run apt upgrade.

Check if it’s active:

systemctl status unattended-upgrades
cat /var/log/unattended-upgrades/unattended-upgrades.log | tail -50

One thing that catches people off guard: after unattended-upgrades updates a core library like libc, running services are still using the old version in memory until they restart. On a production server you’d run:

sudo needrestart

This tells you which services need restarting to pick up the newly installed library versions. Skipping this step means your services are still running against the old, potentially vulnerable code even though the package is updated on disk.

A Real Incident From the Lab

While exploring the dpkg log during this round, I found entries from an hour earlier that I hadn’t triggered manually:

2026-03-16 17:09:16 status unpacked sudo:amd64 1.9.15p5-3ubuntu5.24.04.2
2026-03-16 17:09:16 status half-configured sudo:amd64 1.9.15p5-3ubuntu5.24.04.2
2026-03-16 17:09:16 status installed sudo:amd64 1.9.15p5-3ubuntu5.24.04.2
2026-03-16 17:09:19 status half-configured libc-bin:amd64 2.39-0ubuntu8.7
2026-03-16 17:09:19 status installed libc-bin:amd64 2.39-0ubuntu8.7

First instinct was that something was wrong. But reading the log properly: every package that went through half-configured also reached installed immediately after. All three packages — sudo, man-db, and libc-bin — completed successfully. The half-configured entries are normal phase transition checkpoints, not failures.

What actually happened: unattended-upgrades ran at 17:09 and applied security patches to sudo and glibc. The log in /var/log/apt/history.log confirmed this. The correct follow-up was to run needrestart to check if any services needed restarting after the libc update.

The lesson: knowing how to read dpkg.log turned a potentially alarming discovery into a five-minute investigation with a clear answer.

Round 2 Self-Test Questions

These are the questions I used to verify my own understanding before moving on. If you can answer these without looking things up, you have a solid Round 2 foundation. You don’t need to recite exact paths from memory — you need to know what layer to look at and what the answer means.

APT Internals & Cache

  1. What is the APT cache and where does it live on disk? What command cleans it, and why would you care about disk space there on a server?
  2. What is the difference between apt update and apt upgrade? What does apt update actually do — what file does it update and where?
  3. What is apt full-upgrade and how does it differ from apt upgrade? When would you use one over the other?
  4. After running apt update, where does APT store the downloaded package index files?
  5. What does apt-cache policy <package> tell you?
  6. What is the difference between apt install and apt-get install? Is one preferred over the other in scripts?

dpkg Internals

  1. What is dpkg, and what is its relationship to APT? Which layer sits on top of which?
  2. What do the status codes in dpkg -l mean — for example ii, rc, iU?
  3. What does dpkg -L <package> do? How is it different from dpkg -S <file>?
  4. You have a .deb file downloaded manually. What command installs it? What command inspects its contents before installing?
  5. What is the dpkg status database and where is it located?
  6. A package shows status rc in dpkg -l. What does that mean and how do you fully purge it?
  7. What is the difference between apt remove and apt purge?

Package States and Recovery

  1. What are the two main phases dpkg goes through when installing a package? What is the system state called when phase one completes but phase two has not?
  2. What is the difference between unpacked and configured in dpkg’s status model?
  3. What does a half-installed state mean? How do you recover from it?
  4. You run apt upgrade and it exits mid-way. What is the first command you should run next?
  5. You see half-configured in your dpkg log. How do you determine if this is a failed state or a normal transition checkpoint?

Lock Files

  1. What are /var/lib/dpkg/lock and /var/lib/apt/lists/lock? What does each one guard against?
  2. You get “could not get lock /var/lib/dpkg/lock.” What is the correct investigation sequence?
  3. A junior admin kills the process holding the dpkg lock. What is the risk versus waiting?
  4. When is it actually safe to manually delete a lock file?

Repository Management

  1. What is the syntax of a single entry in /etc/apt/sources.list? Break down each field.
  2. What is a GPG key in the context of APT repos? What happens if you add a repo without its key?
  3. What is the modern way to add a third-party repo’s signing key? What was the old way and why was it deprecated?
  4. What is a PPA and what are the security implications of adding one?

Dependency Resolution

  1. What does it mean for a package to be “held back”? How do you place a package on hold?
  2. What are orphaned packages? What command removes them and what risk does blindly running it carry?
  3. What is a virtual package in Debian/APT? Give a real example.
  4. What does apt install -f do and when do you need it?

APT Priority and Version Selection

  1. What are the default APT priority values for an installed package, a package from a standard repo, and a pinned package?
  2. What does a priority of 100 vs 500 vs 990 mean in APT’s selection logic?
  3. Without explicit pinning, how does APT decide which version to install when the same package exists in multiple repos?
  4. What is the “candidate version” and how do you inspect it?
  5. What is APT pinning and where is it configured?
  6. What does it mean when APT says a package is “kept back”?

Security and Updates

  1. What is unattended-upgrades? What does it do by default and how do you check if it’s active?
  2. After a libc upgrade runs automatically, what additional step should you take on a production server?
  3. What does apt-get --simulate upgrade do and when is it useful?
  4. What log file records all APT activity and where is it?

Network Failures

  1. You run apt update and one repository returns a connection error. Does APT abort entirely or continue?
  2. What is “stale metadata” in APT? What are the symptoms of running apt upgrade against an outdated index?
  3. A third-party repo goes offline permanently. What problems does this cause and how do you remove it cleanly?
  4. APT reaches the repo server but the Release file signature fails. Why is this more serious than a connection timeout?

Troubleshooting Broken States

  1. You get “dpkg was interrupted, you must manually run dpkg –configure -a.” What caused this and what do you do?
  2. apt upgrade fails mid-way due to a disk full error. What is your recovery sequence?
  3. What does apt install --reinstall <package> do and when would you use it?
  4. A package’s post-installation script fails. How do you find what went wrong?

Snap and Flatpak (Awareness)

  1. What is Snap and how does it differ architecturally from APT packages?
  2. What is a snap “channel”? What is the difference between stable, candidate, beta, and edge?
  3. What is Flatpak and what is Flathub?
  4. What are the main criticisms of Snap packages in production environments?

Practical Scenarios

  1. You need to downgrade a package to a specific older version. Walk through how you’d do it.
  2. You need to find which package provides the command netstat. What command do you use?
  3. A colleague says “just run apt upgrade every week and you’re fine.” What’s wrong with that on a production server?
  4. You want to replicate the exact package set from one server onto a new server. How do you export and import the package list?
  5. What is dpkg-reconfigure and when would you use it?

What Round 2 Competency Actually Means

After completing this round, the benchmark isn’t “can I recite every path and flag from memory.” The benchmark is: when something breaks, do I know what layer to look at, what questions to ask, and what vocabulary to use?

If unattended-upgrades runs in the background and you see unfamiliar entries in your dpkg log, you read them — you don’t panic. If apt refuses to run because of a lock error, you investigate before doing anything — you don’t kill processes blindly. If a package ends up in a half-configured state, you know what command finishes the job.

That’s what Round 2 is building. Round 3 will push into real incident simulation — broken package states thrown at you with no warning, and you diagnose them from scratch.

This post is part of the Linux Mastery: Road to Cloud series documenting my self-directed path from zero to Junior Cloud/DevOps Engineer. All labs run on a live Ubuntu server at limonlab.online.

Scroll to Top