LLI Wiki — My Digital Knowledge Base

Monitoring Azure Batch with KQL and Grafana

KQL queries over AzureDiagnostics for failed, active and pending tasks, success rate, tasks per job and per application, and how to use them in Grafana.

A scheduled job that fails at night with nobody noticing is worse than no job at all. For Azure Batch I monitor with KQL queries over the diagnostic logs, displayed in Grafana. Here are the queries I use: failed tasks, active tasks, stuck tasks, success rate, tasks per job and failures per application.

Prerequisite: send diagnostics to Log Analytics

On the Batch account, a diagnostic setting sends service logs to a Log Analytics workspace. In my case they land in the AzureDiagnostics table (the "Azure Diagnostics" mode). The events I care about are identified by OperationName:

OperationName Meaning
TaskStartEvent a task starts on a node
TaskCompleteEvent a task finishes (success or failure)
TaskScheduleFailEvent the task could not be scheduled on a node

Columns used: ResourceProvider == "MICROSOFT.BATCH", jobId_s, id_s (task id), and the result inside AdditionalFields (executionInfo_result: Success or Failure). Exact column names depend on what your workspace ingested: run a take 10 before building a dashboard.

Count each task once

First pitfall: a task can emit several events (retries, later job cleanup). To count cleanly I keep the last real completion event per task:

let tasks = materialize(
    AzureDiagnostics
    | where TimeGenerated > ago(30d)
    | where ResourceProvider == "MICROSOFT.BATCH"
    | where OperationName == "TaskCompleteEvent"
    | where isnotnull(executionInfo_startTime_t)       // real execution
    | extend Result = tostring(parse_json(tostring(AdditionalFields)).executionInfo_result)
    | summarize arg_max(TimeGenerated, Result) by Resource, jobId_s, id_s);

The executionInfo_startTime_t filter drops completion events without an execution (for example a task ended by job cleanup, which shows as a failure although it succeeded days earlier).

Success rate

tasks
| summarize Total = count(),
            Success = countif(Result == "Success"),
            Failed  = countif(Result == "Failure")
| extend SuccessRate = round(100.0 * Success / Total, 2)

For a daily trend, add by Day = bin(TimeGenerated, 1d) to the summarize (dropping the deduplication or keeping it per day).

Failed tasks with details

AzureDiagnostics
| where $__timeFilter(TimeGenerated)
| where ResourceProvider == "MICROSOFT.BATCH" and OperationName == "TaskCompleteEvent"
| extend AF = parse_json(tostring(AdditionalFields))
| where tostring(AF.executionInfo_result) == "Failure"
| project TimeGenerated, Job = jobId_s, TaskId = id_s,
          ExitCode = tostring(AF.executionInfo_exitCode)
| order by TimeGenerated desc

The exit code column may vary with ingestion; if it is missing, just show the full row.

Active tasks: started without a completion

AzureDiagnostics
| where $__timeFilter(TimeGenerated)
| where ResourceProvider == "MICROSOFT.BATCH" and OperationName == "TaskStartEvent"
| join kind=leftanti (
    AzureDiagnostics
    | where TimeGenerated > ago(7d)
    | where OperationName == "TaskCompleteEvent"
    | project id_s
) on id_s
| summarize StartTime = max(TimeGenerated) by TaskId = id_s, Job = jobId_s
| order by StartTime desc

Pitfall: the lookback window for completions (7 days) must be wider than the start window, otherwise a task that finished outside the displayed range counts as active.

Pending tasks: scheduling failures

I look at each task's last known event; if it is a scheduling failure, the task is still waiting for a node:

AzureDiagnostics
| where TimeGenerated > ago(7d)
| where ResourceProvider == "MICROSOFT.BATCH"
| where OperationName in ("TaskStartEvent", "TaskCompleteEvent", "TaskScheduleFailEvent")
| summarize FailCount = countif(OperationName == "TaskScheduleFailEvent"),
            Job = any(jobId_s),
            arg_max(TimeGenerated, OperationName) by TaskId = id_s
| where OperationName == "TaskScheduleFailEvent"
| extend MinutesSinceLastAttempt = datetime_diff('minute', now(), TimeGenerated)

Tasks per job and failures per application

Per job I keep the full id: truncating it merges distinct jobs.

AzureDiagnostics
| where $__timeFilter(TimeGenerated)
| where ResourceProvider == "MICROSOFT.BATCH" and OperationName == "TaskCompleteEvent"
| extend AF = parse_json(tostring(AdditionalFields))
| summarize Success = countif(tostring(AF.executionInfo_result) == "Success"),
            Failed  = countif(tostring(AF.executionInfo_result) == "Failure")
          by Job = jobId_s

For the application I rely on the task id prefix (naming convention) with startswith, more reliable than a regular expression:

| extend App = case(
    id_s startswith "export-",  "export",
    id_s startswith "maint-",   "maintenance",
    id_s startswith "backup-",  "backup",
    "other")

In Grafana

Lessons learned

Mastodon LLIMastodon SocialFosstodon