Production-Grade AI Observability & AIOps Platform
Prometheus • Grafana • Loki • Tempo • OpenTelemetry • Host Ollama Gateway • MCP Servers • pgvector
OpenUI • OpenIntelligence UI • Chainlit
Dockprom-AI is a production-grade, out-of-the-box monitoring, tracing, and AIOps observability stack designed for Docker hosts, containers, and LLM applications. It modernizes the classic stefanprodan/dockprom architecture by incorporating native AI agent interfaces via the Model Context Protocol (MCP), local host-level LLM inference, vector-enabled incident storage, and generative UI capabilities.
The repository is structured into two main layers:
-
Metrics & Monitoring:
-
Prometheus v3.15.1: Core metrics database and PromQL query engine.
-
Alertmanager v0.34.1: Deduplication, grouping, and notification routing for alerts.
-
Pushgateway v1.10.0: Ingestion acceptor for ephemeral and batch jobs.
-
Node Exporter v1.12.1: Linux kernel, host hardware, and system OS metrics.
-
Logporter 1.5.0: Modern, lightweight container metrics exporter and log shipper (replacing cAdvisor) with a built-in dashboard UI.
-
Logs & Distributed Tracing:
-
Loki 3.7.9: Scalable structured log aggregation utilizing TSDB schema v13
-
Tempo 3.1.1: High-throughput distributed tracing backend.
-
OpenTelemetry Collector Contrib 0.164.0: Unified pipeline receiving OTLP traces, logs, and metrics (via ports
4317gRPC and4318HTTP). -
Dashboards & Edge Ingress:
-
Grafana 13.3.0: Pre-provisioned dashboards covering Logporter, Node Exporter, PostgreSQL, OpenTelemetry pipelines, and MCP server health.
-
Caddy 2.11.7: Edge reverse proxy terminating bcrypt-hashed basic authentication for exposed services.
-
Incident Management & Collaboration:
-
Keep AIOps v0.54.3: Incident correlation, noise reduction, and automated alert remediation platform.
-
Mattermost Team Edition 12.0.1: Incident response collaboration workspace supporting webhook alert integration.
-
Database & Persistence:
-
pgvector (PostgreSQL 17): Unified persistence layer for Keep, Mattermost, and vector embeddings storage (
rag_storage). -
Postgres Exporter v0.20.1: Database engine monitoring with
pg_stat_statementsenabled. -
AI Telemetry & MCP Endpoints:
-
Ollama Gateway (Envoy v1.40.2): Reverse proxy running on port
11435that bridges containers to native host Ollama (host.docker.internal:11434), collecting latency statistics and OTLP traces. -
Grafana MCP Server 2.1.1: Official Model Context Protocol server exposing Grafana dashboard and Loki query tools via Server-Sent Events (SSE) on port
8000. -
Prometheus MCP Server v0.18.0: Official MCP server exposing PromQL tools via SSE on port
8080.
Designed to run alongside the core stack on the shared monitor-net Docker network:
- Chainlit AIOps Copilot (
:8050): A Python-based conversational SRE assistant orchestrating multi-step incident triage, retrieving runbooks via cosine similarity search inpgvector, executing queries through the MCP servers, and streaming reasoning steps. - Open-Intelligent-UI (
:3001): A standalone Next.js application implementing the Thesys OpenUI streaming framework (@openuidev/react) to dynamically render interactive diagnostic widgets, forms, and charts generated by the LLM.
- Works Out-of-the-Box: Includes automated host provisioning scripts (
install-host-prereqs.sh) for Docker Engine, Docker Compose, and native host Ollama setup. - Hardware-Accelerated LLMs: Runs Ollama directly on the host for direct GPU access without Dockerized CUDA/ROCm complications.
- RAG-Ready Incident Management: Combines
pgvectorwith OpenTelemetry and PromQL to ground AI agent recommendations directly in live cluster metrics, logs, and historical postmortems. - Generative UI: Combines Chainlit + OpenUI + OpenIntelligent-UI
Dockprom-AI provisions dashboards out-of-the-box using Grafana file provisioning
(grafana/provisioning/dashboards/dashboards.yml). The catalog combines
documented upstream dashboard references with locally generated dashboards for
AI workflows, security, agents, and platform operations.
| Category | Dashboard Name | Grafana Labs ID / Source | Datasource(s) | Description |
|---|---|---|---|---|
| Container & Host | Logporter Docker Exporter | 23848 |
Prometheus, Loki | Official Logporter dashboard: CPU, memory, volume sizes, restart counts, image update alerts, and embedded Loki container logs. |
| Container & Host | Node Exporter Full | 1860 |
Prometheus | Comprehensive host hardware/OS telemetry: CPU usage by core, memory commit, disk I/O latency, network traffic, and system load. |
| AI & MCP Observability | MCP Server Observability | 25252 |
Prometheus, Loki, Tempo | Full-stack monitoring for mcp-grafana and prometheus-mcp: SSE sessions, message rates, JSON-RPC errors, tool invocation latency, and execution logs. |
| AI & LLM Gateway | Envoy Ollama Proxy | Custom / Envoy Stats | Prometheus, Tempo | Request throughput, p50/p95/p99 inferencing latencies, HTTP error rates, and OTLP traces for host Ollama requests. |
| Telemetry Ingestion | OpenTelemetry Collector | 15983 |
Prometheus | Collector operational pipeline metrics: receiver queue sizes, batch processor latency, export drops, and data transmission throughput. |
| Stack Health | Prometheus 3.x System Overview | 3662 |
Prometheus | Scraping health, TSDB compaction rate, ingestion head chunks, sample scrape rates, and query latency percentiles. |
| Stack Health | Loki Logging Stack Health | 13407 |
Prometheus, Loki | Chunks stored, stream indexing status, query throughput, per-tenant volume rates, and flush operations. |
| Stack Health | Tempo Traces & Storage | 15984 |
Prometheus, Tempo | Ingestion block generation, WAL size, span compaction schedules, and search request performance. |
| Ollama Monitoring | Ollama LLM Inference | 25086 |
Prometheus, Ollama Exporter | Ollama dashboard for the open-source ollama-exporter. |
| AI Operations | AI Operations Command Center | Local JSON | Prometheus, Loki, Tempo | Cross-stack availability, workflow SLOs, errors, alerts, and slow traces. |
| AI Observability | Ollama Inference and Model Performance | Local JSON | Prometheus, Loki, Tempo | OpenTelemetry GenAI operation latency, TTFT, token usage, context utilization, queueing, and model load duration. |
| GenUI | GenUI End-to-End Experience | Local JSON | Prometheus, Tempo | Chainlit, RAG, Ollama, streaming, and OpenUI user-journey measurements. |
| RAG | RAG and pgvector Quality | Local JSON | Prometheus, PostgreSQL, Loki | Retrieval latency, empty results, similarity quality, runbook coverage, and vector-search failures. |
| AI Security | AI Security Detection Center | Local JSON | Prometheus, Loki, Tempo | Prompt injection, jailbreak, leakage, policy actions, and trace-linked detections. |
| MCP Security | MCP Tool Security and Audit | Local JSON | Prometheus, Loki, Tempo | Tool outcomes, risk, confirmations, JSON-RPC errors, audit events, and execution traces. |
| Database | PostgreSQL and pgvector Operations | Local JSON | Prometheus, PostgreSQL | Connections, transactions, cache, deadlocks, growth, long queries, and vector indexes. |
| Edge Security | Edge Authentication and API Security | Local JSON | Prometheus, Loki | Authentication failures, scanning, upstream errors, TLS, and unexpected public binds. |
| Incidents | Incident Response Effectiveness | Local JSON | Prometheus, Loki | Alert outcomes, MTTA, MTTR, notifications, runbook coverage, and incident timelines. |
| Telemetry Quality | OpenTelemetry Signal Quality | Local JSON | Prometheus | Accepted/refused signals, export failures, queues, identity gaps, and cardinality. |
| Capacity | GPU and Model Capacity | Local JSON | Prometheus | Accelerator utilization, memory, thermals, errors, queueing, and token throughput. |
| Agent Security | Agent Harness Security and Operations | Local JSON | Prometheus, Loki, Tempo | Agent outcomes, policy denials, approvals, delegation, budgets, and safety actions. |
| Runtime Security | Runtime and Supply-Chain Security | Local JSON | Prometheus, Loki | Falco detections, Trivy summaries, image policy, and container compliance. |
The repository contains deterministic dashboard JSON in grafana/dashboards
and file provisioning in grafana/provisioning. Regenerate and validate the
local dashboards before starting Grafana:
python3 scripts/generate_grafana_dashboards.py
find grafana/dashboards -name '*.json' -print0 | xargs -0 -n1 jq empty
GF_SECURITY_ADMIN_PASSWORD='replace-me' \
GRAFANA_POSTGRES_PASSWORD='replace-me' \
POSTGRES_EXPORTER_DSN='postgresql://user:password@mattermost-db:5432/postgres?sslmode=disable' \
docker compose config --quietSee DOCKPROM_AI_ROADMAP.md for the implementation
phases and website/docs/deployment/telemetry-contract.md
for the metrics and security-event contract.
The diagrams are stored with the Docusaurus static assets so the same files are rendered on GitHub and in the documentation site.
See README_GENUI.md for details about the Generative UI
companion stack.
DOCKPROM_AI_ROADMAP.md— dashboard, telemetry, exporter, security, and delivery roadmap.docs/AGENT_ROADMAP.md— recommended agents, secure harness architecture, skill contract, and rollout gates.docs/SECURITY_CONTROL_REFERENCES.md— OWASP and MITRE ATLAS references, control mapping, and release checklist.docs/DASHBOARD_IMPLEMENTATION_PROMPT.md— reusable prompt for future repository updates.