文章链接:https://blog.csdn.net/qq_43201350/article/details/163477763
“网关上线第二天,凌晨三点被报警电话吵醒——上游服务挂了,网关返回 502,但没人知道是因为什么。”
这是我三年前的经历。从那以后我学会了一件事: 网关可观测性不是选配,是刚需。 今天这篇就带你从零搭建一套生产级监控体系,30 分钟内搞定。
一、可观测性三要素
在网关场景下,我们关心的是:
| 维度 | 核心问题 | 对应工具 |
|---|---|---|
| 指标(Metrics) | QPS 多高?延迟多大?错误率多少? | Prometheus |
| 日志(Logs) | 哪个请求报错了?具体原因? | Loki / ELK |
| 追踪(Traces) | 一次请求穿过网关 → 后端 → DB 的完整链路 | Jaeger / SkyWalking |
| 可视化 | 把所有数据变成图表和大盘 | Grafana |
本文聚焦最核心的三件套: Prometheus(采集) + Loki(日志) + Grafana(展示) 。
为什么选这套?因为它们是 CNCF 毕业项目,社区最活跃,Docker 一键部署。
二、整体架构
┌────────────────────────────────────────────────────────┐
│ Grafana (端口 3000) │
│ ┌──────────┐ ┌──────────┐ ┌──────────────────┐ │
│ │ 网关大盘 │ │ 日志面板 │ │ 告警规则 │ │
│ └──────────┘ └──────────┘ └──────────────────┘ │
└──────────┬──────────────────┬──────────────────────────┘
│ 查指标 │ 查日志
▼ ▼
┌──────────────────┐ ┌──────────────────┐
│ Prometheus │ │ Loki │
│ (端口 9090) │ │ (端口 3100) │
│ │ │ │
│ ┌────────────┐ │ │ ┌────────────┐ │
│ │ 指标存储 │ │ │ │ 日志存储 │ │
│ │ (TSDB) │ │ │ │ (Index+Chunk)│ │
│ └────────────┘ │ │ └────────────┘ │
└────────▲─────────┘ └────────▲─────────┘
│ 抓取 /metrics │ 推送日志
│ │
┌────────┴───────────────────┴─────────────────────────┐
│ API 网关 (端口 9080) │
│ ┌─────────────────────────────────────────────────┐ │
│ │ prometheus 插件 (暴露 /apisix/prometheus/metrics) │ │
│ │ http-logger 插件 (推送日志到 Loki) │ │
│ └─────────────────────────────────────────────────┘ │
└───────────────────────────────────────────────────────┘
三、环境搭建
3.1 docker-compose.yml
在之前的基础上加入监控组件:
version: '3.8'
services:
# =========== 配置中心 ===========
etcd:
image: bitnami/etcd:3.5
environment:
- ALLOW_NONE_AUTHENTICATION=yes
- ETCD_ADVERTISE_CLIENT_URLS=http://0.0.0.0:2379
ports:
- "2379:2379"
networks:
- demo-net
# =========== API 网关 ===========
gateway:
image: apache/apisix:3.10.0-centos
depends_on:
- etcd
volumes:
- ./config/gateway-config.yaml:/usr/local/apisix/conf/config.yaml
ports:
- "9080:9080"
- "9180:9180"
environment:
- APISIX_STAND_ALONE=false
networks:
- demo-net
# =========== 示例后端服务 ===========
backend-v1:
image: ealen/echo-server:latest
environment:
- PORT=8080
networks:
- demo-net
backend-v2:
image: ealen/echo-server:latest
environment:
- PORT=8080
networks:
- demo-net
# =========== Prometheus ===========
prometheus:
image: prom/prometheus:v2.52.0
volumes:
- ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
- prometheus_data:/prometheus
ports:
- "9090:9090"
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--storage.tsdb.retention.time=15d'
- '--web.enable-lifecycle'
networks:
- demo-net
# =========== Loki ===========
loki:
image: grafana/loki:2.9.0
ports:
- "3100:3100"
command: -config.file=/etc/loki/local-config.yaml
volumes:
- loki_data:/loki
networks:
- demo-net
# =========== Grafana ===========
grafana:
image: grafana/grafana:10.4.0
ports:
- "3000:3000"
environment:
- GF_SECURITY_ADMIN_USER=demo_admin
- GF_SECURITY_ADMIN_PASSWORD=demo_admin_2026
- GF_AUTH_ANONYMOUS_ENABLED=false
- GF_INSTALL_PLUGINS=grafana-piechart-panel
volumes:
- grafana_data:/var/lib/grafana
- ./grafana/provisioning:/etc/grafana/provisioning
networks:
- demo-net
depends_on:
- prometheus
- loki
# =========== 压测工具 ===========
loadgen:
image: alpine/curl:latest
entrypoint: ["/bin/sh", "-c", "while true; do sleep 3600; done"]
networks:
- demo-net
networks:
demo-net:
driver: bridge
volumes:
prometheus_data:
loki_data:
grafana_data:3.2Prometheus配置
# prometheus/prometheus.yml
global:
scrape_interval: 15s
evaluation_interval: 15s
external_labels:
cluster: 'demo-gateway-cluster'
env: 'demo'
scrape_configs:
# === 抓取网关指标 ===
- job_name: 'api-gateway'
metrics_path: '/apisix/prometheus/metrics'
static_configs:
- targets: ['gateway:9091']
labels:
service: 'api-gateway'
instance: 'gateway-demo-01'
# === Prometheus 自监控 ===
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']3.3Grafana数据源预配置
# grafana/provisioning/datasources/datasources.yaml
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: true
editable: true
- name: Loki
type: loki
access: proxy
url: http://loki:3100
editable: true# grafana/provisioning/dashboards/dashboards.yaml
apiVersion: 1
providers:
- name: 'Gateway Dashboards'
orgId: 1
folder: 'API Gateway'
type: file
disableDeletion: false
updateIntervalSeconds: 30
allowUiUpdates: true
options:
path: /etc/grafana/provisioning/dashboards四、启用网关指标暴露
4.1 配置 prometheus 插件
# 全局启用 prometheus 插件
curl -s http://127.0.0.1:9180/admin-api/global_rules -X PUT -d '
{
"plugins": {
"prometheus": {
"prefer_name": true
}
}
}'配置后网关会在 http://gateway:9091/apisix/prometheus/metrics 暴露指标。
4.2 核心指标一览
# 拉取指标看看有哪些
curl -s http://127.0.0.1:9091/apisix/prometheus/metrics | grep -E "^apisix_" | head -30关键指标分类:
# ─── HTTP 请求量 ───
apisix_http_status{code="200"} # 按状态码统计
apisix_http_requests_total # 请求总量
# ─── 延迟分布 ───
apisix_http_latency_bucket # 延迟直方图
apisix_http_latency_sum # 总延迟
# ─── 带宽 ───
apisix_bandwidth_total # 总带宽
apisix_bandwidth{type="ingress"} # 入向
apisix_bandwidth{type="egress"} # 出向
# ─── 连接数 ───
apisix_nginx_http_current_connections # 当前连接数
# ─── etcd 健康 ───
apisix_etcd_modify_indexes # etcd 索引版本
apisix_etcd_reachable # etcd 可达性 (1=正常)4.3 模拟流量——让图表有数据
# 先在网关配一条测试路由
curl -s http://127.0.0.1:9180/admin-api/upstreams/demo-echo-upstream -X PUT -d '
{
"type": "roundrobin",
"nodes": {
"backend-v1:8080": 1,
"backend-v2:8080": 1
}
}'
curl -s http://127.0.0.1:9180/admin-api/routes/demo-echo-route -X PUT -d '
{
"uri": "/demo/echo/*",
"upstream_id": "demo-echo-upstream",
"plugins": {
"prometheus": {"prefer_name": false}
}
}'# 模拟持续流量——在 docker 容器中跑
docker exec -d $(docker ps -qf "name=loadgen") sh -c '
while true; do
# 正常请求 - 80%
for i in $(seq 1 8); do
curl -s -o /dev/null -w "%{http_code}\n" \
http://gateway:9080/demo/echo/hello &
done
# 慢请求 - 10%
curl -s -o /dev/null -w "%{http_code}\n" \
http://gateway:9080/demo/echo/slow &
# 错误请求 - 5%
curl -s -o /dev/null -w "%{http_code}\n" \
http://gateway:9080/demo/echo/not-found &
# 超时请求 - 5%
curl -s -o /dev/null --max-time 1 \
http://gateway:9080/demo/echo/timeout &
sleep 2
done
'五、搭建 Grafana 大盘
5.1 第一个面板:QPS & 错误率
登录 Grafana( http://localhost:3000 ,账号 demo_admin ),新建 Dashboard。
面板 1:每秒请求量(QPS)
类型: Graph (Time Series)
PromQL: sum(rate(apisix_http_requests_total[1m])) by (route)
Legend: {{route}}
面板 2:HTTP 状态码分布
类型: Pie Chart
PromQL: sum(rate(apisix_http_status[1m])) by (code)
Legend: {{code}}
面板 3:错误率趋势
类型: Stat
PromQL:
sum(rate(apisix_http_status{code=~"5.."}[5m]))
/
sum(rate(apisix_http_status[5m])) * 100
Unit: percent (0-100)
Thresholds: 绿色 < 1%, 黄色 < 5%, 红色 >= 5%
面板 4:P99 延迟
类型: Graph
PromQL: histogram_quantile(0.99, sum(rate(apisix_http_latency_bucket[5m])) by (le, route))
Legend: P99 - {{route}}
5.2 第二个面板:上游健康度
# 上游请求成功率
sum(rate(apisix_http_status{code!~"5.."}[5m])) by (node)
/
sum(rate(apisix_http_status[5m])) by (node) * 100
# 上游平均延迟
sum(rate(apisix_http_latency_sum[5m])) by (node)
/
sum(rate(apisix_http_latency_count[5m])) by (node) * 10005.3 组合成完整大盘 JSON
{
"dashboard": {
"title": "API 网关监控大盘 (Demo)",
"uid": "demo-gateway-dashboard",
"tags": ["gateway", "api", "demo"],
"timezone": "browser",
"panels": [
{
"title": "QPS (每秒请求)",
"type": "timeseries",
"targets": [{
"expr": "sum(rate(apisix_http_requests_total[1m])) by (route)",
"legendFormat": "{{route}}"
}],
"gridPos": {"x": 0, "y": 0, "w": 12, "h": 8}
},
{
"title": "状态码分布率",
"type": "stat",
"targets": [
{"expr": "sum(rate(apisix_http_status{code=\"200\"}[1m]))", "legendFormat": "2xx"},
{"expr": "sum(rate(apisix_http_status{code=~\"4..\"}[1m]))", "legendFormat": "4xx"},
{"expr": "sum(rate(apisix_http_status{code=~\"5..\"}[1m]))", "legendFormat": "5xx"}
],
"gridPos": {"x": 12, "y": 0, "w": 6, "h": 8},
"fieldConfig": {
"defaults": {
"color": {"mode": "thresholds"},
"thresholds": {
"steps": [
{"value": null, "color": "green"},
{"value": 10, "color": "orange"},
{"value": 50, "color": "red"}
]
}
}
}
},
{
"title": "P50 / P99 延迟 (ms)",
"type": "timeseries",
"targets": [
{
"expr": "histogram_quantile(0.50, sum(rate(apisix_http_latency_bucket[5m])) by (le)) * 1000",
"legendFormat": "P50"
},
{
"expr": "histogram_quantile(0.99, sum(rate(apisix_http_latency_bucket[5m])) by (le)) * 1000",
"legendFormat": "P99"
}
],
"gridPos": {"x": 0, "y": 8, "w": 12, "h": 8}
},
{
"title": "上游服务健康度",
"type": "timeseries",
"targets": [{
"expr": "sum(rate(apisix_http_status{code!~\"5..\"}[5m])) by (node) / sum(rate(apisix_http_status[5m])) by (node) * 100",
"legendFormat": "{{node}} 成功率"
}],
"gridPos": {"x": 12, "y": 8, "w": 6, "h": 8},
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"max": 100
}
}
},
{
"title": "当前活跃连接数",
"type": "gauge",
"targets": [{
"expr": "apisix_nginx_http_current_connections",
"legendFormat": "活跃连接"
}],
"gridPos": {"x": 0, "y": 16, "w": 6, "h": 8},
"fieldConfig": {
"defaults": {
"thresholds": {
"steps": [
{"value": null, "color": "green"},
{"value": 1000, "color": "orange"},
{"value": 5000, "color": "red"}
]
}
}
}
},
{
"title": "etcd 健康检查",
"type": "stat",
"targets": [{
"expr": "apisix_etcd_reachable",
"legendFormat": "etcd 可达"
}],
"gridPos": {"x": 6, "y": 16, "w": 6, "h": 8},
"fieldConfig": {
"defaults": {
"thresholds": {
"steps": [
{"value": null, "color": "red"},
{"value": 1, "color": "green"}
]
}
}
}
},
{
"title": "带宽趋势 (入/出)",
"type": "timeseries",
"targets": [
{
"expr": "rate(apisix_bandwidth{type=\"ingress\"}[5m])",
"legendFormat": "入向"
},
{
"expr": "rate(apisix_bandwidth{type=\"egress\"}[5m])",
"legendFormat": "出向"
}
],
"gridPos": {"x": 12, "y": 16, "w": 6, "h": 8}
}
]
}
}5.4 导入大盘
将上面的 JSON 导入 Grafana:
1. 登录 Grafana → 左侧 "+" → Import
2. 粘贴 JSON → Load
3. 选择 Prometheus 数据源 → Import
效果概览:
| 区域 | 面板 | 说明 |
|---|---|---|
| 左上 | QPS 曲线 | 实时请求速率 |
| 右上 | 状态码计数 | 2xx/4xx/5xx 分类 |
| 左中 | P50/P99 延迟 | 性能瓶颈一目了然 |
| 右中 | 上游健康度 | 哪个节点出问题立刻发现 |
| 左下 | 连接数仪表盘 | 并发压力指示 |
| 中下 | etcd 健康 | 配置中心是否在线 |
| 右下 | 带宽趋势 | 入/出流量监控 |
六、网关日志推送到 Loki
6.1 为什么要单独做日志
Prometheus 指标告诉你有多少错误,但 Loki 日志告诉你具体错在哪一行 。
6.2 配置日志推送
# 在全局规则中增加 http-logger
curl -s http://127.0.0.1:9180/admin-api/global_rules -X PATCH -d '
{
"plugins": {
"prometheus": {
"prefer_name": true
},
"http-logger": {
"uri": "http://loki:3100/loki/api/v1/push",
"batch_max_size": 100,
"max_retry_count": 3,
"retry_delay": 1,
"inactive_timeout": 5,
"concat_method": "new_line",
"include_req_body": false,
"include_resp_body": false
}
}
}'6.3 Loki 日志查询
在 Grafana 中切换到 Explore 页面,选择 Loki 数据源:
# 查询所有 5xx 错误的日志
{job="api-gateway"} | json | response_status >= 500
# 查询慢请求(延迟 > 1s)
{job="api-gateway"} | json | latency > 1
# 查询特定路由的错误
{job="api-gateway"} | json | route = "demo-echo-route" | response_status >= 500
# 查询近 5 分钟的 502 错误
{job="api-gateway"}
| json
| response_status = 502
| line_format "时间: {{.timestamp}}, 上游: {{.upstream}}, 原因: {{.error_msg}}"6.4 排障实战:模拟故障
# 步骤一:干掉一个上游节点
docker stop $(docker ps -qf "name=backend-v2")
# 步骤二:观察 Grafana
# → "上游服务健康度" 面板中 backend-v2:8080 成功率会从 ~100% 掉到 0%
# → "状态码分布" 面板中 502 计数会上升
# → Loki 中会看到具体的 502 日志
# 步骤三:在 Grafana → Explore → Loki 中查询
# {job="api-gateway"} | json | response_status = 502
# 会看到类似:
# upstream: "backend-v2:8080", error: "connection refused"这就是 指标先报警、日志后定位 的经典排障流程。
七、告警规则配置
光看大盘不够,还要配告警——没人会 24 小时盯着屏幕。
7.1 关键告警规则
# prometheus/rules/gateway-alerts.yml
groups:
- name: api_gateway_alerts
interval: 30s
rules:
# ===== 可用性告警 =====
- alert: HighErrorRate
expr: |
sum(rate(apisix_http_status{code=~"5.."}[5m]))
/
sum(rate(apisix_http_status[5m])) > 0.05
for: 2m
labels:
severity: critical
annotations:
summary: "网关 5xx 错误率超过 5%"
description: "当前值: {{ $value | humanizePercentage }},路由: {{ $labels.route }}"
- alert: UpstreamDown
expr: |
sum(rate(apisix_http_status{code=~"5.."}[5m])) by (node)
/
sum(rate(apisix_http_status[5m])) by (node) > 0.5
for: 1m
labels:
severity: critical
annotations:
summary: "上游节点 {{ $labels.node }} 不可用"
description: "5xx 比例超过 50%,请检查上游服务"
# ===== 性能告警 =====
- alert: HighLatency
expr: |
histogram_quantile(0.99,
sum(rate(apisix_http_latency_bucket[5m])) by (le, route)
) > 3
for: 5m
labels:
severity: warning
annotations:
summary: "P99 延迟超过 3 秒"
description: "路由 {{ $labels.route }},当前值: {{ $value }}s"
- alert: HighConnectionCount
expr: apisix_nginx_http_current_connections > 5000
for: 3m
labels:
severity: warning
annotations:
summary: "网关活跃连接数超过 5000"
description: "当前连接数: {{ $value }}"
# ===== 基础设施告警 =====
- alert: EtcdUnreachable
expr: apisix_etcd_reachable != 1
for: 1m
labels:
severity: critical
annotations:
summary: "网关无法连接 etcd"
description: "网关配置可能无法热更新,请立即检查"
- alert: NoTraffic
expr: sum(rate(apisix_http_requests_total[5m])) == 0
for: 10m
labels:
severity: warning
annotations:
summary: "网关流量为零"
description: "近 10 分钟无请求,可能配置异常或上游全部故障"7.2 注册告警规则
# 在 prometheus.yml 中追加
rule_files:
- '/etc/prometheus/rules/gateway-alerts.yml'# 重新加载 Prometheus 配置
curl -X POST http://127.0.0.1:9090/-/reload7.3 告警通知渠道
Prometheus 的Alertmanager负责把告警推送到即时通讯/邮件:
# alertmanager/config.yml
route:
group_by: ['alertname', 'severity']
group_wait: 10s
group_interval: 10s
repeat_interval: 1h
# 严重告警走即时通讯
routes:
- match:
severity: critical
receiver: 'dingtalk-critical'
- match:
severity: warning
receiver: 'dingtalk-warning'
receivers:
- name: 'dingtalk-critical'
webhook_configs:
- url: 'https://oapi.dingtalk.com/robot/send?access_token=YOUR_TOKEN'
send_resolved: true
message: |
### 🔴 严重告警
- 告警: {{ .CommonAnnotations.summary }}
- 详情: {{ .CommonAnnotations.description }}
- name: 'dingtalk-warning'
webhook_configs:
- url: 'https://oapi.dingtalk.com/robot/send?access_token=YOUR_TOKEN'
send_resolved: true八、一键部署脚本
把以上所有内容总结成一个脚本:
#!/bin/bash
# scripts/deploy-observability.sh
# 一键部署:网关 + Prometheus + Loki + Grafana
set -e
echo "========================================="
echo " 网关可观测性三件套 - 一键部署"
echo "========================================="
# 1. 创建目录结构
mkdir -p prometheus/rules grafana/provisioning/datasources grafana/provisioning/dashboards config
# 2. 检查 Docker
if ! command -v docker &> /dev/null; then
echo "❌ 请先安装 Docker"
exit 1
fi
# 3. 启动全部服务
docker compose up -d
# 4. 等待网关就绪
echo "⏳ 等待网关就绪..."
for i in $(seq 1 30); do
if curl -s http://127.0.0.1:9180/admin-api/routes > /dev/null 2>&1; then
echo "✅ 网关已就绪"
break
fi
sleep 2
done
# 5. 配置网关路由
echo "📝 配置网关路由和监控插件..."
bash scripts/setup-gateway-routes.sh
# 6. 检查组件状态
echo ""
echo "========================================="
echo " 部署完成!访问地址:"
echo "========================================="
echo " Grafana: http://localhost:3000"
echo " 账号: demo_admin"
echo " 密码: demo_admin_2026"
echo " Prometheus: http://localhost:9090"
echo " 网关 Admin: http://localhost:9180"
echo " 网关 Proxy: http://localhost:9080"
echo ""
echo " 下一步:"
echo " 1. 浏览器打开 http://localhost:3000"
echo " 2. 左侧 → Dashboards → API 网关监控大盘"
echo " 3. 运行 scripts/simulate-traffic.sh 产生测试数据"
echo "========================================="九、总结
9.1 我们搭建了什么
| 组件 | 作用 | 端口 |
|---|---|---|
| Prometheus | 采集指标、存储时序数据 | 9090 |
| Loki | 采集日志、支持类似 LogQL 查询 | 3100 |
| Grafana | 统一可视化大盘 + 告警展示 | 3000 |
| 网关 prometheus 插件 | 暴露请求量/延迟/状态码/带宽 | 自动 |
| 网关 http-logger 插件 | 推送请求日志到 Loki | 自动 |
9.2 监控检查清单
上线前确认以下内容全部就绪:
-
Prometheus 能抓到网关指标(
up= 1) -
Grafana 大盘 7 个面板都有数据
-
Loki 能查到网关日志
-
告警规则已注册(在 Prometheus → Alerts 中可见)
-
手动停一个上游节点,确认大盘能反映
-
确认 Alertmanager 能发送测试告警
9.3 进阶方向
| 方向 | 工具 | 说明 |
|---|---|---|
| 链路追踪 | Jaeger / Zipkin | 看一次请求从网关到后端的完整路径 |
| SLO 监控 | Pyrra / Sloth | 基于 SLI 指标,计算 burn rate |
| 成本分析 | OpenCost | 网关带宽成本归属 |
| 异常检测 | 基于 Z-score 的 PromQL | 自动发现异常流量模式 |
写在最后:监控不是终点,可观测性才是
恭喜你跟着这篇教程把 Prometheus + Loki + Grafana 跑起来了。
但说实话,"能装上"和"能对客户承诺 SLA"之间,差了十万八千里。
你可能很快会遇到这些问题:
-
🔸 告警太多,每天收到 200 条,根本分不清哪个重要——告警风暴怎么治?
-
🔸 阈值设多少合适?QPS 报警设 1000 还是 2000?拍脑袋设的阈值要么误报要么漏报
-
🔸 凌晨出故障,谁响应?多久响应?怎么升级?——没有 On-Call SOP 就是裸奔
-
🔸 Prometheus 自己挂了谁监控 Prometheus?——监控系统的监控盲区
-
🔸 怎么证明你的系统达到了 99.9% 可用?——SLA 指标体系怎么设计
这些问题,我在付费专栏里用一整篇文章做了系统解答,包含:
-
SLA 指标体系设计(USE 方法论)
-
告警分级标准(P0-P3)+ 阈值制定方法论
-
On-Call 排班 SOP 模板
-
故障演练方案(定期注入故障,验证监控是否真的能发现问题)
-
监控自身的监控方案
🎯 专栏《云原生API网关从入门到生产》完整包含:
-
✅ 10 篇实战教程 + 8 项增值内容(代码仓库 / Checklist / 事故手册 / 面试题……)
-
✅ Grafana Dashboard JSON(导入即用,含 SLA / 延迟 / 错误率 / 流量四块大盘)
-
✅ 告警规则完整 YAML 合集
-
✅ 付费读者优先答疑
👉 点击订阅《云原生 API 网关从入门到生产》19.9 元
这篇免费教程如果帮到了你,点赞收藏是对我最好的鼓励 🙏 我最好的鼓励 🙏