LLI Wiki — My Digital Knowledge Base

Grafana and AI: querying monitoring in natural language through an MCP server

Plugging an AI assistant into Grafana through MCP: read-only by default, writes gated by human approval, useful questions, limits, hallucination risks and cost.

"Which databases hit their CPU ceiling last night?" "Which dashboard shows the errors of this job?" Instead of opening Grafana, hunting for the right panel and adjusting the time range, I ask an AI assistant plugged into Grafana through an MCP server. Here is how I frame it, what it does well, and where it can go wrong.

The principle

MCP (Model Context Protocol) lets an assistant call external tools. A Grafana MCP server exposes the Grafana API as tools: search dashboards, read their JSON, query data sources (Prometheus, Loki, Azure Monitor...), list alert rules, look at incidents. The assistant picks the tool, runs the query and summarises the result.

flowchart LR
    U["Me: natural-language question"] --> A["AI assistant"]
    A --> M["Grafana MCP server"]
    M --> G["Grafana API"]
    G --> D["Data sources: metrics, logs, KQL"]
    A -->|"read"| M
    A -.->|"write: ask for approval"| H{"Human approval"}
    H -->|"yes"| M
    H -->|"no"| X["Nothing is changed"]

Read-only by default

This is rule number one. I set up two independent layers:

  1. A Grafana service account with the lowest possible role (viewer), its token kept in the tool's personal configuration, never in a repository. If the AI gets it wrong, Grafana itself refuses to write.
  2. Write tools gated by confirmation on the client side: creating or editing an alert, a silence, a dashboard, an incident. Each call is displayed, I read it and approve, or not.

Reads are free. Writes, especially in production, wait for my explicit approval, every time. Approval given for one write does not carry over to the next.

Useful questions

Question What the assistant does
"Which dashboards cover elastic pools?" searches titles and tags
"Which query feeds this panel?" extracts the KQL or PromQL from the panel
"Which alerts are failing or silenced?" lists rules and their state
"What happened at 3:00 am?" queries logs over the window, summarises errors
"This panel is empty, why?" replays the query, checks the data source and range
"Compare CPU of these two pools over 7 days" runs two queries and compares

Where AI shines: turning an intent into a query, navigating dozens of dashboards, explaining a query you did not write yourself.

Limits and hallucination risks

Cost

The MCP server is free; what costs money is model tokens. A full dashboard JSON or a large log query eats a lot of context. My habits: ask for a summary or one specific property instead of the whole JSON, bound time ranges and row counts, and do not let the assistant "explore" without a target. On the Grafana side, every query against a volume-billed source (Log Analytics, for example) has a cost too: an eager AI can fire many of them.

Lessons learned

Mastodon LLIMastodon SocialFosstodon