Grafana and AI: querying monitoring in natural language through an MCP server
Plugging an AI assistant into Grafana through MCP: read-only by default, writes gated by human approval, useful questions, limits, hallucination risks and cost.
"Which databases hit their CPU ceiling last night?" "Which dashboard shows the errors of this job?" Instead of opening Grafana, hunting for the right panel and adjusting the time range, I ask an AI assistant plugged into Grafana through an MCP server. Here is how I frame it, what it does well, and where it can go wrong.
The principle
MCP (Model Context Protocol) lets an assistant call external tools. A Grafana MCP server exposes the Grafana API as tools: search dashboards, read their JSON, query data sources (Prometheus, Loki, Azure Monitor...), list alert rules, look at incidents. The assistant picks the tool, runs the query and summarises the result.
flowchart LR
U["Me: natural-language question"] --> A["AI assistant"]
A --> M["Grafana MCP server"]
M --> G["Grafana API"]
G --> D["Data sources: metrics, logs, KQL"]
A -->|"read"| M
A -.->|"write: ask for approval"| H{"Human approval"}
H -->|"yes"| M
H -->|"no"| X["Nothing is changed"]
Read-only by default
This is rule number one. I set up two independent layers:
- A Grafana service account with the lowest possible role (viewer), its token kept in the tool's personal configuration, never in a repository. If the AI gets it wrong, Grafana itself refuses to write.
- Write tools gated by confirmation on the client side: creating or editing an alert, a silence, a dashboard, an incident. Each call is displayed, I read it and approve, or not.
Reads are free. Writes, especially in production, wait for my explicit approval, every time. Approval given for one write does not carry over to the next.
Useful questions
| Question | What the assistant does |
|---|---|
| "Which dashboards cover elastic pools?" | searches titles and tags |
| "Which query feeds this panel?" | extracts the KQL or PromQL from the panel |
| "Which alerts are failing or silenced?" | lists rules and their state |
| "What happened at 3:00 am?" | queries logs over the window, summarises errors |
| "This panel is empty, why?" | replays the query, checks the data source and range |
| "Compare CPU of these two pools over 7 days" | runs two queries and compares |
Where AI shines: turning an intent into a query, navigating dozens of dashboards, explaining a query you did not write yourself.
Limits and hallucination risks
- It can invent a metric, column or dashboard name that "sounds right". Countermeasure: have it list the real sources and labels first, and require the executed query to be shown.
- A valid query can answer a different question: wrong time range, wrong time zone (the default is UTC), wrong workspace, filter too broad. An empty result is not proof that nothing happened.
- Numbers need checking before you repeat them in a report: I read the query and sometimes replay it myself in Grafana.
- Its summary is not the data. For an important decision, I look at the chart.
- What it reads enters the assistant's context: server names or error messages can contain sensitive information. Only connect instances whose content you accept to send through the model provider, and treat text it reads (logs, descriptions) as data, never as instructions.
Cost
The MCP server is free; what costs money is model tokens. A full dashboard JSON or a large log query eats a lot of context. My habits: ask for a summary or one specific property instead of the whole JSON, bound time ranges and row counts, and do not let the assistant "explore" without a target. On the Grafana side, every query against a volume-billed source (Log Analytics, for example) has a cost too: an eager AI can fire many of them.
Lessons learned
- The gain is real for exploration and diagnosis; it is small for what I already do in two clicks.
- Read-only by construction (the token's role) beats read-only by promise.
- I keep a human in the loop for every write: the AI proposes, I decide.
- Always ask for the executed query: it is the best defence against a plausible but wrong answer.