LLI Wiki — My Digital Knowledge Base

Keeping an "errors and corrections" knowledge base: entries, runbooks and payoff

One entry per error (symptom, cause, fix, prevention), runbooks for recurring incidents and why it pays off.

The same error solved three times by three people (or by me, six months apart) without anyone remembering the fix: that is the hidden cost of operations. My countermeasure is one file: an "errors and corrections" knowledge base, versioned with the code, where every recurring error has its entry.

One entry per error

Short and concrete. Each entry has a stable id (ERR-042) and five fields:

Field Content Generic example
Symptom / signature The exact message or behaviour, easy to search Login failed for user '<token-identified principal>'
Context Where and when it happens Batch task run from a managed identity
Root cause Specific, not "permissions problem" The identity was never created as a user in the target database
Selected fix Actionable, verified CREATE USER ... FROM EXTERNAL PROVIDER then a minimal role
Prevention An explicit rule, including what not to do again Check the user in the deployment script

A Markdown table with one row per error is enough. Add the verification evidence (test, log) and, if an approach was rejected, say so: that stops it from being proposed again.

| ID      | Signature                  | Root cause                      | Fix                              | Prevention                         |
|---------|----------------------------|---------------------------------|----------------------------------|------------------------------------|
| ERR-042 | "Login failed ... token"   | User missing from the database  | Create the user from Entra       | Check step in the deployment       |

The lifecycle

flowchart LR
    A["Incident"] --> B["Search the signature in the base"]
    B --> C{"Found?"}
    C -->|yes| D["Apply the confirmed fix"]
    C -->|no| E["Diagnose and fix"]
    E --> F["Write the test"]
    F --> G["Add an entry with the next id"]
    D --> H["Update the entry if behaviour changed"]
    G --> I["Move the prevention into the repo rules"]

Rules I impose on myself:

Runbooks for recurring incidents

When an incident keeps coming back, an entry is no longer enough: write a short runbook chaining diagnosis and action. The format I keep:

  1. Trigger (which alert, which symptom).
  2. Read-only checks, with copy-paste queries.
  3. Decision tree: transient, third-party change, defect.
  4. Action and verification of its result.
  5. Who to notify.

The runbook points to the error entries; the entries point back to the runbook. See also the triage method described in another article.

Why it pays off

Lessons learned

Mastodon LLIMastodon SocialFosstodon