Monitoring Azure Batch with KQL and Grafana
KQL queries over AzureDiagnostics for failed, active and pending tasks, success rate, tasks per job and per application, and how to use them in Grafana.
A scheduled job that fails at night with nobody noticing is worse than no job at all. For Azure Batch I monitor with KQL queries over the diagnostic logs, displayed in Grafana. Here are the queries I use: failed tasks, active tasks, stuck tasks, success rate, tasks per job and failures per application.
Prerequisite: send diagnostics to Log Analytics
On the Batch account, a diagnostic setting sends service logs to a Log Analytics workspace. In my case they land in the AzureDiagnostics table (the "Azure Diagnostics" mode). The events I care about are identified by OperationName:
| OperationName | Meaning |
|---|---|
TaskStartEvent |
a task starts on a node |
TaskCompleteEvent |
a task finishes (success or failure) |
TaskScheduleFailEvent |
the task could not be scheduled on a node |
Columns used: ResourceProvider == "MICROSOFT.BATCH", jobId_s, id_s (task id), and the result inside AdditionalFields (executionInfo_result: Success or Failure). Exact column names depend on what your workspace ingested: run a take 10 before building a dashboard.
Count each task once
First pitfall: a task can emit several events (retries, later job cleanup). To count cleanly I keep the last real completion event per task:
let tasks = materialize(
AzureDiagnostics
| where TimeGenerated > ago(30d)
| where ResourceProvider == "MICROSOFT.BATCH"
| where OperationName == "TaskCompleteEvent"
| where isnotnull(executionInfo_startTime_t) // real execution
| extend Result = tostring(parse_json(tostring(AdditionalFields)).executionInfo_result)
| summarize arg_max(TimeGenerated, Result) by Resource, jobId_s, id_s);
The executionInfo_startTime_t filter drops completion events without an execution (for example a task ended by job cleanup, which shows as a failure although it succeeded days earlier).
Success rate
tasks
| summarize Total = count(),
Success = countif(Result == "Success"),
Failed = countif(Result == "Failure")
| extend SuccessRate = round(100.0 * Success / Total, 2)
For a daily trend, add by Day = bin(TimeGenerated, 1d) to the summarize (dropping the deduplication or keeping it per day).
Failed tasks with details
AzureDiagnostics
| where $__timeFilter(TimeGenerated)
| where ResourceProvider == "MICROSOFT.BATCH" and OperationName == "TaskCompleteEvent"
| extend AF = parse_json(tostring(AdditionalFields))
| where tostring(AF.executionInfo_result) == "Failure"
| project TimeGenerated, Job = jobId_s, TaskId = id_s,
ExitCode = tostring(AF.executionInfo_exitCode)
| order by TimeGenerated desc
The exit code column may vary with ingestion; if it is missing, just show the full row.
Active tasks: started without a completion
AzureDiagnostics
| where $__timeFilter(TimeGenerated)
| where ResourceProvider == "MICROSOFT.BATCH" and OperationName == "TaskStartEvent"
| join kind=leftanti (
AzureDiagnostics
| where TimeGenerated > ago(7d)
| where OperationName == "TaskCompleteEvent"
| project id_s
) on id_s
| summarize StartTime = max(TimeGenerated) by TaskId = id_s, Job = jobId_s
| order by StartTime desc
Pitfall: the lookback window for completions (7 days) must be wider than the start window, otherwise a task that finished outside the displayed range counts as active.
Pending tasks: scheduling failures
I look at each task's last known event; if it is a scheduling failure, the task is still waiting for a node:
AzureDiagnostics
| where TimeGenerated > ago(7d)
| where ResourceProvider == "MICROSOFT.BATCH"
| where OperationName in ("TaskStartEvent", "TaskCompleteEvent", "TaskScheduleFailEvent")
| summarize FailCount = countif(OperationName == "TaskScheduleFailEvent"),
Job = any(jobId_s),
arg_max(TimeGenerated, OperationName) by TaskId = id_s
| where OperationName == "TaskScheduleFailEvent"
| extend MinutesSinceLastAttempt = datetime_diff('minute', now(), TimeGenerated)
Tasks per job and failures per application
Per job I keep the full id: truncating it merges distinct jobs.
AzureDiagnostics
| where $__timeFilter(TimeGenerated)
| where ResourceProvider == "MICROSOFT.BATCH" and OperationName == "TaskCompleteEvent"
| extend AF = parse_json(tostring(AdditionalFields))
| summarize Success = countif(tostring(AF.executionInfo_result) == "Success"),
Failed = countif(tostring(AF.executionInfo_result) == "Failure")
by Job = jobId_s
For the application I rely on the task id prefix (naming convention) with startswith, more reliable than a regular expression:
| extend App = case(
id_s startswith "export-", "export",
id_s startswith "maint-", "maintenance",
id_s startswith "backup-", "backup",
"other")
In Grafana
- Azure Monitor datasource, "Logs" query type, on the right workspace (the wrong datasource/workspace pair returns zero rows without any error).
- Bind the query to the time picker with
$__timeFilter(TimeGenerated)and use$__intervalforbin(). - "Day/week/month" panels that define their own window must turn off the dashboard-time option on the target, otherwise the picker range silently caps the query.
- A multi-value variable interpolates as a glob: write
in (${var:singlequote})rather than==.
Lessons learned
- Grafana errors of this kind are silent: "No data" with no message. Replay the query in Log Analytics.
- Ingestion delay exists: do not alert on the last minute.
- Workspace retention limits the "all time" history.