Triaging a failed scheduled run: a six-step method
Where to read status and logs, group errors, tell transient from defect, rerun safely and know when to open a ticket.
A red mail at 7 am: "119 failures". The reflex is to rerun or to panic. After a few bad experiences, mine is to apply the same method, in the same order, every time. It works for any scheduled workload: Batch task, pipeline, nightly job, work queue.
The principle
Diagnosing never needs a write. You read, group, decide, and only then act. Rerunning a destructive script (database drop, switchover, SQL corrections) is always its owner's call, never a reflex.
flowchart TD
A["Alert or red mail"] --> B["Read the status table, not the mail"]
B --> C["Group by error pattern, server, time window"]
C --> D{"Known pattern?"}
D -->|yes| E["Apply the documented fix"]
D -->|no| F["Check what changed underneath"]
F --> G{"Transient?"}
G -->|yes| H{"Safe to rerun?"}
H -->|yes| I["Rerun, verify the result"]
H -->|no| J["Ask the owner"]
G -->|no| K{"Someone else's change?"}
K -->|yes| L["Tell the owner"]
K -->|no| M["Defect: test then fix, add to the errors base"]
1. Where is the status?
The mail summarises, the status table details: one row per run, with the full error text. Every workload should write its state (start, end, message) to a central table. That is the truth; the process exit code is not: a Batch task can finish "successfully" while the maintenance it drove failed.
2. Where are the logs?
- Local run: a log folder per application and environment (
.log,.err,.warn). - Pipeline: logs published as build artifacts.
- Batch: only the wrapper log is uploaded to storage at task completion;
stdoutandstderrstay on the node. - Retention: tasks and node files are deleted after a few days, storage logs are purged, Log Analytics keeps 30 or 90 days. Fetch what you need before it is gone.
3. Group the errors
This is the step that changes everything. Normalise the message, then GROUP BY:
SELECT CASE
WHEN ErrorMessage LIKE '%was not found%' THEN 'NOT FOUND'
WHEN ErrorMessage LIKE '%timeout%' THEN 'TIMEOUT'
WHEN ErrorMessage LIKE '%login failed%' THEN 'AUTH'
ELSE LEFT(ErrorMessage, 60)
END AS pattern,
COUNT(*) AS runs, COUNT(DISTINCT ServerName) AS servers,
MIN(StartTime) AS first_seen, MAX(StartTime) AS last_seen
FROM dbo.RunStatus
WHERE Status = 'Failed' AND StartTime >= DATEADD(DAY, -1, SYSUTCDATETIME())
GROUP BY CASE
WHEN ErrorMessage LIKE '%was not found%' THEN 'NOT FOUND'
WHEN ErrorMessage LIKE '%timeout%' THEN 'TIMEOUT'
WHEN ErrorMessage LIKE '%login failed%' THEN 'AUTH'
ELSE LEFT(ErrorMessage, 60) END
ORDER BY runs DESC;
Golden rule: one pattern on one server within minutes is one cause, not N failures. A hundred failures in six minutes is typically one deleted server.
4. Transient or not?
Check, in this order, what changed underneath:
- The estate: resource deleted, renamed, failed over? Resource Graph, activity log, last inventory collection.
- The database itself: schema or permission change.
- The platform: Azure maintenance, firewall, quota.
- The code: last deployed version and its changelog.
| Signal | Probably |
|---|---|
| Timeout, dropped connection, failover in progress | Transient |
| Same error, same database, every night | A problem of that database, not the platform |
| "Not found" right after a deletion | Inventory lag |
| Sudden authentication error | Permission or identity change |
5. Rerunning safely
A rerun is safe if it is idempotent and you know what it touches. Before rerunning: is the previous run really finished? A killed process leaves its row open (InProgress, Running); do not rerun on top of it without closing it by hand, previewing first, during the day.
6. When to open a ticket
- A third party changed something (deletion, permission, firewall): tell its owner; there is nothing to fix on your side.
- A defect in your code: fix with its test, then an entry in the errors base if it can come back.
- A persistent platform failure: ticket to the vendor, with the pattern, the window and the evidence.
The final report is one line per pattern: count, servers, window, cause and evidence, transient or not, action, owner.
Lessons learned
- A "success" from the scheduler proves nothing: read the status table.
- Mails and tables do not always use the same words; trust the table.
- A failure every night on the same database is a problem of that database.
- A document that misled me (log name, parameter) gets fixed on the spot.