将Nginx Vhost Traffic Status的数据导入到ELK中

2 年 ago
清, 宇
9 minutes

将Nginx Vhost Traffic Status的数据导入到ELK中。

准备导入Nginx VTS（Nginx虚拟主机流量状态）的数据。

安装nginx-vts-exporter

将Nginx VTS的JSON数据转化为适用于Prometheus的输出工具。

$ git clone https://github.com/hnlq715/nginx-vts-exporter.git

$ wget https://dl.google.com/go/go1.13.7.linux-amd64.tar.gz
$ sudo tar go1.13.7.linux-amd64.tar.gz -C /usr/local/go
$ sudo cat <<EOF > /etc/profile.d/goenv.sh 
export GOROOT=/usr/local/go
export GOPATH=$HOME/go
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin
EOF
$ source /etc/profile.d/goenv.sh

$ cd nginx-vts-exporter
$ make
$ sudo cp nginx-vts-exporter /usr/bin
$ sudo cp nginx_vts_exporter.default /etc/sysconfig/nginx_vts_exporter
$ sudo cat <<EOF > /etc/sysconfig/nginx_vts_exporter
NGINX_VTS_EXPORTER_OPTS='-telemetry.address="localhost:9913" -nginx.scrape_uri=http://localhost/status/json'
EOF
$ sudo cat <<EOF > /usr/lib/systemd/system/nginx_vts_exporter.service
[Unit]
Description=Prometheus exporter thar exports Nginx VTS stats.
Documentation=https://github.com/hnlq715/nginx-vts-exporter
After=network.target

[Service]
EnvironmentFile=-/etc/sysconfig/nginx_vts_exporter
User=root
ExecStart=/usr/bin/nginx-vts-exporter $NGINX_VTS_EXPORTER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF
$ sudo systemctl enable nginx_vts_exporter
$ sudo systemctl start nginx_vts_exporter

尝试获取数据

尝试连接到本地主机的端口9913，exporter的监听器。

$ curl http://localhost:9913
<html>
            <head><title>Nginx Exporter</title></head>
            <body>
            <h1>Nginx Exporter</h1>
            <p><a href="/metrics">Metrics</a></p>
            </body>
            </html>

哎？怎么变了呢……

$ nginx-vts-exporter -h
Usage of nginx-vts-exporter:
  -insecure
        Ignore server certificate if using https (default true)
  -metrics.namespace string
        Prometheus metrics namespace. (default "nginx")
  -nginx.scrape_timeout int
        The number of seconds to wait for an HTTP response from the nginx.scrape_uri (default 2)
  -nginx.scrape_uri string
        URI to nginx stub status page (default "http://localhost/status")
  -telemetry.address string
        Address on which to expose metrics. (default ":9913")
  -telemetry.endpoint string
        Path under which to expose metrics. (default "/metrics")
  -version
        Print version information.

哦，看起来 endpoint 是 /metrics，所以需要再次测试。

$ curl http://localhost:9913/metrics
# HELP nginx_cache_bytes cache request/response bytes
# TYPE nginx_cache_bytes counter
nginx_cache_bytes{direction="in",zone="cache-zone"} 33366
nginx_cache_bytes{direction="out",zone="cache-zone"} 551016
# HELP nginx_cache_requests cache requests counter
# TYPE nginx_cache_requests counter
nginx_cache_requests{status="bypass",zone="cache-zone"} 65
nginx_cache_requests{status="expired",zone="cache-zone"} 0
nginx_cache_requests{status="hit",zone="cache-zone"} 4
nginx_cache_requests{status="miss",zone="cache-zone"} 7
nginx_cache_requests{status="revalidated",zone="cache-zone"} 0
nginx_cache_requests{status="scarce",zone="cache-zone"} 0
nginx_cache_requests{status="stale",zone="cache-zone"} 0
nginx_cache_requests{status="updating",zone="cache-zone"} 0
<snip>

我想要根据每个虚拟服务器来观察缓存命中等情况，所以尝试按照以下方式获取。

$ /usr/bin/curl -s http://localhost:9913/metrics | grep -v ^# | grep -e "_server_requests" -e "_server_bytes"
nginx_server_bytes{direction="in",host="*"} 1.1809658e+07
nginx_server_bytes{direction="in",host="elk.nakacya.jp"} 146149
nginx_server_bytes{direction="in",host="localhost"} 1.1582905e+07
nginx_server_bytes{direction="in",host="test1.nakacya.jp"} 46807
nginx_server_bytes{direction="in",host="www.nakacya.jp"} 33797
nginx_server_bytes{direction="out",host="*"} 2.554692267e+09
nginx_server_bytes{direction="out",host="elk.nakacya.jp"} 332947
nginx_server_bytes{direction="out",host="localhost"} 2.553677271e+09
nginx_server_bytes{direction="out",host="test1.nakacya.jp"} 597605
nginx_server_bytes{direction="out",host="www.nakacya.jp"} 84444
nginx_server_requests{code="1xx",host="*"} 0
nginx_server_requests{code="1xx",host="elk.nakacya.jp"} 0
nginx_server_requests{code="1xx",host="localhost"} 0
nginx_server_requests{code="1xx",host="test1.nakacya.jp"} 0
nginx_server_requests{code="1xx",host="www.nakacya.jp"} 0
nginx_server_requests{code="2xx",host="*"} 33502
nginx_server_requests{code="2xx",host="elk.nakacya.jp"} 73
nginx_server_requests{code="2xx",host="localhost"} 33309
nginx_server_requests{code="2xx",host="test1.nakacya.jp"} 83
nginx_server_requests{code="2xx",host="www.nakacya.jp"} 37
nginx_server_requests{code="3xx",host="*"} 14
nginx_server_requests{code="3xx",host="elk.nakacya.jp"} 14
nginx_server_requests{code="3xx",host="localhost"} 0
nginx_server_requests{code="3xx",host="test1.nakacya.jp"} 0
nginx_server_requests{code="3xx",host="www.nakacya.jp"} 0
nginx_server_requests{code="4xx",host="*"} 219
nginx_server_requests{code="4xx",host="elk.nakacya.jp"} 19
nginx_server_requests{code="4xx",host="localhost"} 133
nginx_server_requests{code="4xx",host="test1.nakacya.jp"} 33
nginx_server_requests{code="4xx",host="www.nakacya.jp"} 34
nginx_server_requests{code="5xx",host="*"} 2
nginx_server_requests{code="5xx",host="elk.nakacya.jp"} 0
nginx_server_requests{code="5xx",host="localhost"} 0
nginx_server_requests{code="5xx",host="test1.nakacya.jp"} 0
nginx_server_requests{code="5xx",host="www.nakacya.jp"} 2
nginx_server_requests{code="total",host="*"} 33737
nginx_server_requests{code="total",host="elk.nakacya.jp"} 106
nginx_server_requests{code="total",host="localhost"} 33442
nginx_server_requests{code="total",host="test1.nakacya.jp"} 116
nginx_server_requests{code="total",host="www.nakacya.jp"} 73

数据处理

因为方向的 “in/out” 在 “指数表达” 中，这让人感到不方便，所以我会进行修改。

这里的一行perl代码非常出色

#Server Metric GET
/usr/bin/curl -s http://localhost:9913/metrics | grep -v ^# | grep -e "_server_requests" -e "_server_bytes" > /var/log/nginx/vts_status.log

# an exponential -> real number
cat /var/log/nginx/vts_status.log | perl -anle 'printf "%s %d\n",$F[0],$F[1]' > /var/log/nginx/vts_status_number.log
mv -f /var/log/nginx/vts_status_number.log /var/log/nginx/vts_status.log

处理后

nginx_server_bytes{direction="in",host="*"} 11809961
nginx_server_bytes{direction="in",host="elk.nakacya.jp"} 146149
nginx_server_bytes{direction="in",host="localhost"} 11583208
nginx_server_bytes{direction="in",host="test1.nakacya.jp"} 46807
nginx_server_bytes{direction="in",host="www.nakacya.jp"} 33797
nginx_server_bytes{direction="out",host="*"} 2555083406
nginx_server_bytes{direction="out",host="elk.nakacya.jp"} 332947
nginx_server_bytes{direction="out",host="localhost"} 2554068410
nginx_server_bytes{direction="out",host="test1.nakacya.jp"} 597605
nginx_server_bytes{direction="out",host="www.nakacya.jp"} 84444
nginx_server_requests{code="1xx",host="*"} 0
nginx_server_requests{code="1xx",host="elk.nakacya.jp"} 0
nginx_server_requests{code="1xx",host="localhost"} 0
nginx_server_requests{code="1xx",host="test1.nakacya.jp"} 0
nginx_server_requests{code="1xx",host="www.nakacya.jp"} 0
nginx_server_requests{code="2xx",host="*"} 33505
nginx_server_requests{code="2xx",host="elk.nakacya.jp"} 73
nginx_server_requests{code="2xx",host="localhost"} 33312
nginx_server_requests{code="2xx",host="test1.nakacya.jp"} 83
nginx_server_requests{code="2xx",host="www.nakacya.jp"} 37
nginx_server_requests{code="3xx",host="*"} 14
nginx_server_requests{code="3xx",host="elk.nakacya.jp"} 14
nginx_server_requests{code="3xx",host="localhost"} 0
nginx_server_requests{code="3xx",host="test1.nakacya.jp"} 0
nginx_server_requests{code="3xx",host="www.nakacya.jp"} 0
nginx_server_requests{code="4xx",host="*"} 219
nginx_server_requests{code="4xx",host="elk.nakacya.jp"} 19
nginx_server_requests{code="4xx",host="localhost"} 133
nginx_server_requests{code="4xx",host="test1.nakacya.jp"} 33
nginx_server_requests{code="4xx",host="www.nakacya.jp"} 34
nginx_server_requests{code="5xx",host="*"} 2
nginx_server_requests{code="5xx",host="elk.nakacya.jp"} 0
nginx_server_requests{code="5xx",host="localhost"} 0
nginx_server_requests{code="5xx",host="test1.nakacya.jp"} 0
nginx_server_requests{code="5xx",host="www.nakacya.jp"} 2
nginx_server_requests{code="total",host="*"} 33740
nginx_server_requests{code="total",host="elk.nakacya.jp"} 106
nginx_server_requests{code="total",host="localhost"} 33445
nginx_server_requests{code="total",host="test1.nakacya.jp"} 116
nginx_server_requests{code="total",host="www.nakacya.jp"} 73

有了有了

$ cat /var/log/nginx/vts_status.log | sed -e s/\{/./g -e s/\}/" :"/g -e s/\"//g -e s/,/" "/g -e s/" host="/" "/g | sort -k2,2 > /tmp/temp.log
$ cat /tmp/temp.log
nginx_server_bytes.direction=in * : 11810163
nginx_server_bytes.direction=out * : 2555344161
nginx_server_requests.code=1xx * : 0
nginx_server_requests.code=2xx * : 33507
nginx_server_requests.code=3xx * : 14
nginx_server_requests.code=4xx * : 219
nginx_server_requests.code=5xx * : 2
nginx_server_requests.code=total * : 33742
nginx_server_bytes.direction=in elk.nakacya.jp : 146149
nginx_server_bytes.direction=out elk.nakacya.jp : 332947
nginx_server_requests.code=1xx elk.nakacya.jp : 0
nginx_server_requests.code=2xx elk.nakacya.jp : 73
nginx_server_requests.code=3xx elk.nakacya.jp : 14
nginx_server_requests.code=4xx elk.nakacya.jp : 19
nginx_server_requests.code=5xx elk.nakacya.jp : 0
nginx_server_requests.code=total elk.nakacya.jp : 106
nginx_server_bytes.direction=in localhost : 11583410
nginx_server_bytes.direction=out localhost : 2554329165
nginx_server_requests.code=1xx localhost : 0
nginx_server_requests.code=2xx localhost : 33314
nginx_server_requests.code=3xx localhost : 0
nginx_server_requests.code=4xx localhost : 133
nginx_server_requests.code=5xx localhost : 0
nginx_server_requests.code=total localhost : 33447
nginx_server_bytes.direction=in test1.nakacya.jp : 46807
nginx_server_bytes.direction=out test1.nakacya.jp : 597605
nginx_server_requests.code=1xx test1.nakacya.jp : 0
nginx_server_requests.code=2xx test1.nakacya.jp : 83
nginx_server_requests.code=3xx test1.nakacya.jp : 0
nginx_server_requests.code=4xx test1.nakacya.jp : 33
nginx_server_requests.code=5xx test1.nakacya.jp : 0
nginx_server_requests.code=total test1.nakacya.jp : 116
nginx_server_bytes.direction=in www.nakacya.jp : 33797
nginx_server_bytes.direction=out www.nakacya.jp : 84444
nginx_server_requests.code=1xx www.nakacya.jp : 0
nginx_server_requests.code=2xx www.nakacya.jp : 37
nginx_server_requests.code=3xx www.nakacya.jp : 0
nginx_server_requests.code=4xx www.nakacya.jp : 34
nginx_server_requests.code=5xx www.nakacya.jp : 2
nginx_server_requests.code=total www.nakacya.jp : 73

对于Filebeat进行微调

因为换行会造成困扰，而且没有日期和时间也很麻烦，所以需要：
– 添加日期和时间
– 删除换行符

$ date "+%Y/%m/%dT%H:%M:%S" |sed -z 's/\n/ /g' && cat /tmp/temp.log | sed -z 's/\n/ /g' |sed -e "s/\$/\n/"
2020/02/05T04:04:33 nginx_server_bytes.direction=in * : 11810163 nginx_server_bytes.direction=out * : 2555344161 nginx_server_requests.code=1xx * : 0 nginx_server_requests.code=2xx * : 33507 nginx_server_requests.code=3xx * : 14 nginx_server_requests.code=4xx * : 219 nginx_server_requests.code=5xx * : 2 nginx_server_requests.code=total * : 33742

最后

如果将这个数据传给 Filebeat，并在 ELK 的 Logstash 中传给 Grok 进行处理，那么数据就完成了。Grok 大致是这样的。

^%{DATA:Date} nginx_server_bytes.direction=in %{DATA:Host} : %{DATA:In-Byte} nginx_server_bytes.direction=out .*: %{DATA:Out_Byte} nginx_server_cache.status=bypass .*: %{NUMBER:Cache_Bypass} nginx_server_cache.status=expired .*: %{NUMBER:Cache_Expired} nginx_server_cache.status=hit .*: %{NUMBER:Cache_Hit} nginx_server_cache.status=miss .*: %{NUMBER:Cache_Miss} nginx_server_cache.status=revalidated .*: %{NUMBER:Cache_Revalidated} nginx_server_cache.status=scarce .*: %{NUMBER:Cache_Scarce} nginx_server_cache.status=stale .*: %{NUMBER:Cache_Stale} nginx_server_cache.status=updating .*: %{NUMBER:Cache_Updating} nginx_server_requests.code=1xx .*: %{NUMBER:Requests_Code_1xx} nginx_server_requests.code=2xx .*: %{NUMBER:Requests_Code_2xx} nginx_server_requests.code=3xx .*: %{NUMBER:Requests_Code_3xx} nginx_server_requests.code=4xx .*: %{NUMBER:Requests_Code_4xx} nginx_server_requests.code=5xx .*: %{NUMBER:Requests_Code_5xx} nginx_server_requests.code=total .*: %{NUMBER:Requests_Code_Total}