基于Prometheus指标的自动扩展在Kubernetes中
首先
在使用Kubernetes进行容器编排时，其中一个主要优点是在应用程序进行水平扩展时，非常容易应对增加的负载。通常，水平Pod自动缩放器可以根据CPU和内存使用量来扩展部署，但在更复杂的情况下，我们希望在做出扩展决策之前考虑其他指标。
首先要输入Prometheus适配器。Prometheus是一种用于监控部署的工作负载和Kubernetes集群本身的标准工具。Prometheus适配器能够利用由Prometheus收集的指标，并使用它们来进行扩展决策。这些指标可以由API服务公开，并且可以立即用于Horizontal Pod Autoscaling对象。
2. 引入
2.1架构概述
使用Prometheus适配器，从Prometheus安装中获取自定义指标，并使水平Pod自动缩放器能够使用该指标来扩大或缩小Pod。 Prometheus适配器会作为一个以公开的服务形式运行在集群内部。通常，中小规模的集群只需要一个适配器的副本就足够了。然而，如果有非常大规模的集群，可以使用节点亲和性属性和Pod-AntiAffinity属性来运行多个分布在整个节点上的Prometheus适配器的副本。
2.2 前提条件的说法
-
- 少なくとも3つのノードでKubernetesクラスタ環境で実行して、このチュートリアルではGKEクラスタを使用します。
-
- 水平ポッドの自動スケーリングに関する基本的な知識。
- クラスタ内にデプロイされた、またはエンドポイントを使用してアクセス可能なPrometheus。
在这里，我们将使用Prometheus-Thanos高可用性部署。有关详细信息，请点击这里。
2.3 部署示例应用程序
首先，我们部署一个用于测试Prometheus指标自动缩放的示例应用程序。您可以使用以下清单进行操作。
apiVersion: v1
kind: Namespace
metadata:
name: nginx
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
namespace: nginx
name: nginx-deployment
spec:
replicas: 1
template:
metadata:
annotations:
prometheus.io/path: "/status/format/prometheus"
prometheus.io/scrape: "true"
prometheus.io/port: "80"
labels:
app: nginx-server
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- nginx-server
topologyKey: kubernetes.io/hostname
containers:
- name: nginx-demo
image: vaibhavthakur/nginx-vts:1.0
imagePullPolicy: Always
resources:
limits:
cpu: 2500m
requests:
cpu: 2000m
ports:
- containerPort: 80
name: http
---
apiVersion: v1
kind: Service
metadata:
namespace: nginx
name: nginx-service
spec:
ports:
- port: 80
targetPort: 80
name: http
selector:
app: nginx-server
type: LoadBalancer
通过这个过程，将会创建一个名为nginx的命名空间，并在其中部署一个示例的nginx应用程序。该应用程序可以通过服务进行访问，并通过端口80公开nginx vts指标，端点为/status/format/prometheus。为了设置，我已经创建了一个映射到ExternalIP的dns条目，将其命名为nginx.gotham.com。
root$ kubectl get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-deployment 1/1 1 1 43d
root$ kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-deployment-65d8df7488-c578v 1/1 Running 0 9h
root$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-service ClusterIP 10.63.253.154 35.232.67.34 80/TCP 43d
root$ kubectl describe deploy nginx-deployment
Name: nginx-deployment
Namespace: nginx
CreationTimestamp: Tue, 08 Oct 2019 11:47:36 -0700
Labels: app=nginx-server
Annotations: deployment.kubernetes.io/revision: 1
kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"extensions/v1beta1","kind":"Deployment","metadata":{"annotations":{},"name":"nginx-deployment","namespace":"nginx"},"spec":...
Selector: app=nginx-server
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 1 max unavailable, 1 max surge
Pod Template:
Labels: app=nginx-server
Annotations: prometheus.io/path: /status/format/prometheus
prometheus.io/port: 80
prometheus.io/scrape: true
Containers:
nginx-demo:
Image: vaibhavthakur/nginx-vts:v1.0
Port: 80/TCP
Host Port: 0/TCP
Limits:
cpu: 250m
Requests:
cpu: 200m
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
OldReplicaSets: <none>
NewReplicaSet: nginx-deployment-65d8df7488 (1/1 replicas created)
Events: <none>
root$ curl nginx.gotham.com
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
这些是当前由应用程序公开的所有度量指标。
$ curl nginx.gotham.com/status/format/prometheus
# HELP nginx_vts_info Nginx info
# TYPE nginx_vts_info gauge
nginx_vts_info{hostname="nginx-deployment-65d8df7488-c578v",version="1.13.12"} 1
# HELP nginx_vts_start_time_seconds Nginx start time
# TYPE nginx_vts_start_time_seconds gauge
nginx_vts_start_time_seconds 1574283147.043
# HELP nginx_vts_main_connections Nginx connections
# TYPE nginx_vts_main_connections gauge
nginx_vts_main_connections{status="accepted"} 215
nginx_vts_main_connections{status="active"} 4
nginx_vts_main_connections{status="handled"} 215
nginx_vts_main_connections{status="reading"} 0
nginx_vts_main_connections{status="requests"} 15577
nginx_vts_main_connections{status="waiting"} 3
nginx_vts_main_connections{status="writing"} 1
# HELP nginx_vts_main_shm_usage_bytes Shared memory [ngx_http_vhost_traffic_status] info
# TYPE nginx_vts_main_shm_usage_bytes gauge
nginx_vts_main_shm_usage_bytes{shared="max_size"} 1048575
nginx_vts_main_shm_usage_bytes{shared="used_size"} 3510
nginx_vts_main_shm_usage_bytes{shared="used_node"} 1
# HELP nginx_vts_server_bytes_total The request/response bytes
# TYPE nginx_vts_server_bytes_total counter
# HELP nginx_vts_server_requests_total The requests counter
# TYPE nginx_vts_server_requests_total counter
# HELP nginx_vts_server_request_seconds_total The request processing time in seconds
# TYPE nginx_vts_server_request_seconds_total counter
# HELP nginx_vts_server_request_seconds The average of request processing times in seconds
# TYPE nginx_vts_server_request_seconds gauge
# HELP nginx_vts_server_request_duration_seconds The histogram of request processing time
# TYPE nginx_vts_server_request_duration_seconds histogram
# HELP nginx_vts_server_cache_total The requests cache counter
# TYPE nginx_vts_server_cache_total counter
nginx_vts_server_bytes_total{host="_",direction="in"} 3303449
nginx_vts_server_bytes_total{host="_",direction="out"} 61641572
nginx_vts_server_requests_total{host="_",code="1xx"} 0
nginx_vts_server_requests_total{host="_",code="2xx"} 15574
nginx_vts_server_requests_total{host="_",code="3xx"} 0
nginx_vts_server_requests_total{host="_",code="4xx"} 2
nginx_vts_server_requests_total{host="_",code="5xx"} 0
nginx_vts_server_requests_total{host="_",code="total"} 15576
nginx_vts_server_request_seconds_total{host="_"} 0.000
nginx_vts_server_request_seconds{host="_"} 0.000
nginx_vts_server_cache_total{host="_",status="miss"} 0
nginx_vts_server_cache_total{host="_",status="bypass"} 0
nginx_vts_server_cache_total{host="_",status="expired"} 0
nginx_vts_server_cache_total{host="_",status="stale"} 0
nginx_vts_server_cache_total{host="_",status="updating"} 0
nginx_vts_server_cache_total{host="_",status="revalidated"} 0
nginx_vts_server_cache_total{host="_",status="hit"} 0
nginx_vts_server_cache_total{host="_",status="scarce"} 0
nginx_vts_server_bytes_total{host="*",direction="in"} 3303449
nginx_vts_server_bytes_total{host="*",direction="out"} 61641572
nginx_vts_server_requests_total{host="*",code="1xx"} 0
nginx_vts_server_requests_total{host="*",code="2xx"} 15574
nginx_vts_server_requests_total{host="*",code="3xx"} 0
nginx_vts_server_requests_total{host="*",code="4xx"} 2
nginx_vts_server_requests_total{host="*",code="5xx"} 0
nginx_vts_server_requests_total{host="*",code="total"} 15576
nginx_vts_server_request_seconds_total{host="*"} 0.000
nginx_vts_server_request_seconds{host="*"} 0.000
nginx_vts_server_cache_total{host="*",status="miss"} 0
nginx_vts_server_cache_total{host="*",status="bypass"} 0
nginx_vts_server_cache_total{host="*",status="expired"} 0
nginx_vts_server_cache_total{host="*",status="stale"} 0
nginx_vts_server_cache_total{host="*",status="updating"} 0
nginx_vts_server_cache_total{host="*",status="revalidated"} 0
nginx_vts_server_cache_total{host="*",status="hit"} 0
nginx_vts_server_cache_total{host="*",status="scarce"} 0
在这些中，特别是nginx_vts_server_requests_total是一个非常有趣的指标。通过使用这个指标的值，决定是否要扩展nginx部署。
创建2.4 Prometheus适配器的SSL证书和Kubernetes机密
您可以使用以下的Makefile生成相应的Kubernetes Secret和openssl证书。
# Makefile for generating TLS certs for the Prometheus custom metrics API adapter
SHELL=bash
UNAME := $(shell uname)
PURPOSE:=metrics
SERVICE_NAME:=custom-metrics-apiserver
ALT_NAMES:="custom-metrics-apiserver.monitoring","custom-metrics-apiserver.monitoring.svc"
SECRET_FILE:=custom-metrics-api/cm-adapter-serving-certs.yaml
certs: gensecret rmcerts
.PHONY: gencerts
gencerts:
@echo Generating TLS certs
@docker pull cfssl/cfssl
@mkdir -p output
@touch output/apiserver.pem
@touch output/apiserver-key.pem
@openssl req -x509 -sha256 -new -nodes -days 365 -newkey rsa:2048 -keyout $(PURPOSE)-ca.key -out $(PURPOSE)-ca.crt -subj "/CN=ca"
@echo '{"signing":{"default":{"expiry":"43800h","usages":["signing","key encipherment","'$(PURPOSE)'"]}}}' > "$(PURPOSE)-ca-config.json"
@echo '{"CN":"'$(SERVICE_NAME)'","hosts":[$(ALT_NAMES)],"key":{"algo":"rsa","size":2048}}' | docker run -v ${HOME}:${HOME} -v ${PWD}/metrics-ca.key:/go/src/github.com/cloudflare/cfssl/metrics-ca.key -v ${PWD}/metrics-ca.crt:/go/src/github.com/cloudflare/cfssl/metrics-ca.crt -v ${PWD}/metrics-ca-config.json:/go/src/github.com/cloudflare/cfssl/metrics-ca-config.json -i cfssl/cfssl gencert -ca=metrics-ca.crt -ca-key=metrics-ca.key -config=metrics-ca-config.json - | docker run --entrypoint=cfssljson -v ${HOME}:${HOME} -v ${PWD}/output:/go/src/github.com/cloudflare/cfssl/output -i cfssl/cfssl -bare output/apiserver
.PHONY: gensecret
gensecret: gencerts
@echo Generating $(SECRET_FILE)
@echo "apiVersion: v1" > $(SECRET_FILE)
@echo "kind: Secret" >> $(SECRET_FILE)
@echo "metadata:" >> $(SECRET_FILE)
@echo " name: cm-adapter-serving-certs" >> $(SECRET_FILE)
@echo " namespace: monitoring" >> $(SECRET_FILE)
@echo "data:" >> $(SECRET_FILE)
ifeq ($(UNAME), Darwin)
@echo " serving.crt: $$(cat output/apiserver.pem | base64)" >> $(SECRET_FILE)
@echo " serving.key: $$(cat output/apiserver-key.pem | base64)" >> $(SECRET_FILE)
endif
ifeq ($(UNAME), Linux)
@echo " serving.crt: $$(cat output/apiserver.pem | base64 -w 0)" >> $(SECRET_FILE)
@echo " serving.key: $$(cat output/apiserver-key.pem | base64 -w 0)" >> $(SECRET_FILE)
endif
.PHONY: rmcerts
rmcerts:
@rm -f apiserver-key.pem apiserver.csr apiserver.pem
@rm -f metrics-ca-config.json metrics-ca.crt metrics-ca.key
.PHONY: deploy
deploy:
kubectl create -f ./custom-metrics-api
一旦创建了make文件，请执行以下命令。
make certs
另外，将创建与SSL证书对应的Kubernetes机密。在创建机密之前，请确保监视命名空间存在。这个机密将被用于后续部署的Prometheus适配器。
创建2.5 Prometheus适配器ConfigMap。
使用以下的宣言，创建Prometheus适配器配置映射。
apiVersion: v1
kind: ConfigMap
metadata:
name: adapter-config
namespace: monitoring
data:
config.yaml: |
rules:
- seriesQuery: 'nginx_vts_server_requests_total'
resources:
overrides:
kubernetes_namespace:
resource: namespace
kubernetes_pod_name:
resource: pod
name:
matches: "^(.*)_total"
as: "${1}_per_second"
metricsQuery: (sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>))
这个配置图只指定了一个度量标准。然而，您可以随时添加其他度量标准。您可以参考此链接来进一步添加度量标准。我们强烈建议仅获取水平Pod自动缩放器所需的度量标准，这将有助于调试。此外，这些插件会生成非常详细的日志，并被日志后端所捕获。如果获取不必要的指标，将会增加服务的负载，同时不必要的日志也会被发送到日志后端。有关配置图的详细信息，请查看这里。
这个构成地图只指定一个测量标准。尽管如此，您随时可以添加测量标准。您可以参考此链接来进一步添加测量标准。强烈推荐仅提取水平 Pod 自动缩放器所需的测量标准。对于调试会有帮助。此外，这些附加组件会生成非常详细的日志，并且由日志后端收集。如果获取不需要的指标，则服务将变得负载较重，同时还会发送不必要的日志到日志后端。请参阅设置地图的详细信息。
创建2.6 Prometheus适配器部署。
请使用以下清单部署Prometheus适配器。
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: custom-metrics-apiserver
name: custom-metrics-apiserver
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app: custom-metrics-apiserver
template:
metadata:
labels:
app: custom-metrics-apiserver
name: custom-metrics-apiserver
spec:
serviceAccountName: monitoring
containers:
- name: custom-metrics-apiserver
image: quay.io/coreos/k8s-prometheus-adapter-amd64:v0.4.1
args:
- /adapter
- --secure-port=6443
- --tls-cert-file=/var/run/serving-cert/serving.crt
- --tls-private-key-file=/var/run/serving-cert/serving.key
- --logtostderr=true
- --prometheus-url=http://thanos-querier.monitoring:9090/
- --metrics-relist-interval=30s
- --v=10
- --config=/etc/adapter/config.yaml
ports:
- containerPort: 6443
volumeMounts:
- mountPath: /var/run/serving-cert
name: volume-serving-cert
readOnly: true
- mountPath: /etc/adapter/
name: config
readOnly: true
volumes:
- name: volume-serving-cert
secret:
secretName: cm-adapter-serving-certs
- name: config
configMap:
name: adapter-config
通过这样做，将创建一个部署，生成Prometheus适配器pod并从Prometheus中拉取度量。请注意设置参数 –prometheus-url = http://thanos-querier.monitoring:9090/。这是因为在与Prometheus适配器相同的Kubernetes集群中的监控命名空间内部署了由Thanos支持的Prometheus集群。您可以通过更改此参数来指向Prometheus部署。
如果您注意到这个容器的日志，就会发现它正在获取在设置文件中定义的指标。
I1122 00:26:53.228394 1 api.go:74] GET http://thanos-querier.monitoring:9090/api/v1/series?match%5B%5D=nginx_vts_server_requests_total&start=1574381213.217 200 OK
I1122 00:26:53.234234 1 api.go:93] Response Body: {"status":"success","data":[{"__name__":"nginx_vts_server_requests_
total","app":"nginx-server","cluster":"prometheus-ha","code":"1xx","host":"*","instance":"10.60.64.39:80","job":"kubernetes
-pods","kubernetes_namespace":"nginx","kubernetes_pod_name":"nginx-deployment-65d8df7488-sbp95","pod_template_hash":"65d8df
7488"},{"__name__":"nginx_vts_server_requests_total","app":"nginx-server","cluster":"prometheus-ha","code":"1xx","host":"*"
,"instance":"10.60.64.8:80","job":"kubernetes-pods","kubernetes_namespace":"nginx","kubernetes_pod_name":"nginx-deployment-
65d8df7488-mwzxg","pod_template_hash":"65d8df7488"},{"__name__":"nginx_vts_server_requests_total","app":"nginx-server",
"cluster":"prometheus-ha","code":"1xx","host":"_","instance":"10.60.64.39:80","job":"kubernetes-pods","kubernetes_namespace
":"nginx","kubernetes_pod_name":"nginx-deployment-65d8df7488-sbp95","pod_template_hash":"65d8df7488"},{"__name__":"nginx_
vts_server_requests_total","app":"nginx-server","cluster":"prometheus-ha","code":"1xx","host":"_","instance":"10.60.64.8:80
","job":"kubernetes-pods","kubernetes_namespace":"nginx","kubernetes_pod_name":"nginx-deployment-65d8df7488-mwzxg",
"pod_template_hash":"65d8df7488"},{"__name__":"nginx_vts_server_requests_total","app":"nginx-server","cluster":"prometheus-
ha","code":"2xx","host":"*","instance":"10.60.64.39:80","job":"kubernetes-pods","kubernetes_namespace":"nginx","kubernetes_
pod_name":"nginx-deployment-65d8df7488-sbp95","pod_template_hash":"65d8df7488"},{"__name__":"nginx_vts_server_requests_
total","app":"nginx-server","cluster":"prometheus-ha","code":"2xx","host":"*","instance":"10.60.64.8:80","job":"kubernetes-
pods","kubernetes_namespace":"nginx","kubernetes_pod_name":"nginx-deployment-65d8df7488-mwzxg","pod_template_hash":
"65d8df7488"},{"__name__":"nginx_vts_server_requests_total","app":"nginx-server","cluster":"prometheus-ha","code":"2xx",
"host":"_","instance":"10.60.64.39:80","job":"kubernetes-pods","kubernetes_namespace":"nginx","kubernetes_pod_name":
"nginx-deployment-65d8df7488-sbp95","pod_template_hash":"65d8df7488"},{"__name__":"nginx_vts_server_requests_total",
"app":"nginx-server","cluster":"prometheus-ha","code":"2xx","host":"_","instance":"10.60.64.8:80","job":"kubernetes-pods"
,"kubernetes_namespace":"nginx","kubernetes_pod_name":"nginx-deployment-65d8df7488-mwzxg","pod_template_hash":
"65d8df7488"},{"__name__":"nginx_vts_server_requests_total","app":"nginx-server","cluster":"prometheus-ha","code":"3xx",
"host":"*","instance":"10.60.64.39:80","job":"kubernetes-pods","kubernetes_namespace":"nginx","kubernetes_pod_name":
"nginx-deployment-65d8df7488-sbp95","pod_template_hash":"65d8df7488"},{"__name__":"nginx_vts_server_requests_total",
"app":"nginx-server","cluster":"prometheus-ha","code":"3xx","host":"*","instance":"10.60.64.8:80","job":"kubernetes-pods"
,"kubernetes_namespace":"nginx","kubernetes_pod_name":"nginx-deployment-65d8df7488-mwzxg","pod_template_hash":
"65d8df7488"},{"__name__":"nginx_vts_server_requests_total","app":"nginx-server","cluster":"prometheus-ha","code":"3xx",
"host":"_","instance":"10.60.64.39:80","job":"kubernetes-pods","kubernetes_namespace":"nginx","kubernetes_pod_name":
"nginx-deployment-65d8df7488-sbp95","pod_template_hash":"65d8df7488"},{"__name__":"nginx_vts_server_requests_total",
"app":"nginx-server","cluster":"prometheus-ha","code":"3xx","host":"_","instance":"10.60.64.8:80","job":"kubernetes-pods"
,"kubernetes_namespace":"nginx","kubernetes_pod_name":"nginx-deployment-65d8df7488-mwzxg","pod_template_hash":
"65d8df7488"},{"__name__":"nginx_vts_server_requests_total","app":"nginx-server","cluster":"prometheus-ha","code":"4xx",
"host":"*","instance":"10.60.64.39:80","job":"kubernetes-pods","kubernetes_namespace":"nginx","kubernetes_pod_name":
"nginx-deployment-65d8df7488-sbp95","pod_template_hash":"65d8df7488"},{"__name__":"nginx_vts_server_requests_total",
"app":"nginx-server","cluster":"prometheus-ha","code":"4xx","host":"*","instance":"10.60.64.8:80","job":"kubernetes-pods"
,"kubernetes_namespace":"nginx","kubernetes_pod_name":"nginx-deployment-65d8df7488-mwzxg","pod_template_hash":
"65d8df7488"},{"__name__":"nginx_vts_server_requests_total","app":"nginx-server","cluster":"prometheus-ha","code":"4xx",
"host":"_","instance":"10.60.64.39:80","job":"kubernetes-pods","kubernetes_namespace":"nginx","kubernetes_pod_name":
"nginx-deployment-65d8df7488-sbp95","pod_template_hash":"65d8df7488"},{"__name__":"nginx_vts_server_requests_total",
"app":"nginx-server","cluster":"prometheus-ha","code":"4xx","host":"_","instance":"10.60.64.8:80","job":"kubernetes-pods"
,"kubernetes_namespace":"nginx","kubernetes_pod_name":"nginx-deployment-65d8df7488-mwzxg","pod_template_hash":
"65d8df7488"},{"__name__":"nginx_vts_server_requests_total","app":"nginx-server","cluster":"prometheus-ha","code":"5xx",
"host":"*","instance":"10.60.64.39:80","job":"kubernetes-pods","kubernetes_namespace":"nginx","kubernetes_pod_name":
"nginx-deployment-65d8df7488-sbp95","pod_template_hash":"65d8df7488"},{"__name__":"nginx_vts_server_requests_total",
"app":"nginx-server","cluster":"prometheus-ha","code":"5xx","host":"*","instance":"10.60.64.8:80","job":"kubernetes-pods"
,"kubernetes_namespace":"nginx","kubernetes_pod_name":"nginx-deployment-65d8df7488-mwzxg","pod_template_hash":
"65d8df7488"},{"__name__":"nginx_vts_server_requests_total","app":"nginx-server","cluster":"prometheus-ha","code":"5xx",
"host":"_","instance":"10.60.64.39:80","job":"kubernetes-pods","kubernetes_namespace":"nginx","kubernetes_pod_name":
"nginx-deployment-65d8df7488-sbp95","pod_template_hash":"65d8df7488"},{"__name__":"nginx_vts_server_requests_total",
"app":"nginx-server","cluster":"prometheus-ha","code":"5xx","host":"_","instance":"10.60.64.8:80","job":"kubernetes-pods"
,"kubernetes_namespace":"nginx","kubernetes_pod_name":"nginx-deployment-65d8df7488-mwzxg","pod_template_hash":
"65d8df7488"},{"__name__":"nginx_vts_server_requests_total","app":"nginx-server","cluster":"prometheus-ha","code":
"total","host":"*","instance":"10.60.64.39:80","job":"kubernetes-pods","kubernetes_namespace":"nginx","kubernetes_pod_
name":"nginx-deployment-65d8df7488-sbp95","pod_template_hash":"65d8df7488"},{"__name__":"nginx_vts_server_requests_total"
,"app":"nginx-server","cluster":"prometheus-ha","code":"total","host":"*","instance":"10.60.64.8:80","job":
"kubernetes-pods","kubernetes_namespace":"nginx","kubernetes_pod_name":"nginx-deployment-65d8df7488-mwzxg",
"pod_template_hash":"65d8df7488"},{"__name__":"nginx_vts_server_requests_total","app":"nginx-server","cluster":
"prometheus-ha","code":"total","host":"_","instance":"10.60.64.39:80","job":"kubernetes-pods","kubernetes_namespace":
"nginx","kubernetes_pod_name":"nginx-deployment-65d8df7488-sbp95","pod_template_hash":"65d8df7488"},{"__name__":
"nginx_vts_server_requests_total","app":"nginx-server","cluster":"prometheus-ha","code":"total","host":"_","instance":
"10.60.64.8:80","job":"kubernetes-pods","kubernetes_namespace":"nginx","kubernetes_pod_name":"nginx-deployment-
65d8df7488-mwzxg","pod_template_hash":"65d8df7488"}]}
用于创建API服务的以下清单将使得可以从Kubernetes API访问Prometheus适配器，并能够通过水平Pod自动缩放器获取指标。
apiVersion: v1
kind: Service
metadata:
name: custom-metrics-apiserver
namespace: monitoring
spec:
ports:
- port: 443
targetPort: 6443
selector:
app: custom-metrics-apiserver
---
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
name: v1beta1.custom.metrics.k8s.io
spec:
service:
name: custom-metrics-apiserver
namespace: monitoring
group: custom.metrics.k8s.io
version: v1beta1
insecureSkipTLSVerify: true
groupPriorityMinimum: 100
versionPriority: 100
3. 安装测试
让我们查看所有可用的自定义指标。
root$ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .
{
"kind": "APIResourceList",
"apiVersion": "v1",
"groupVersion": "custom.metrics.k8s.io/v1beta1",
"resources": [
{
"name": "pods/nginx_vts_server_requests_per_second",
"singularName": "",
"namespaced": true,
"kind": "MetricValueList",
"verbs": [
"get"
]
},
{
"name": "namespaces/nginx_vts_server_requests_per_second",
"singularName": "",
"namespaced": false,
"kind": "MetricValueList",
"verbs": [
"get"
]
}
]
}
可以看到nginx_vts_server_requests_per_second指标是可用的。
接下来，我们来查看一下这个指标的当前数值。
root$ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/nginx/pods/*/nginx_vts_server_requests_per_second" | jq .
{
"kind": "MetricValueList",
"apiVersion": "custom.metrics.k8s.io/v1beta1",
"metadata": {
"selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/nginx/pods/%2A/nginx_vts_server_requests_per_second"
},
"items": [
{
"describedObject": {
"kind": "Pod",
"namespace": "nginx",
"name": "nginx-deployment-65d8df7488-v575j",
"apiVersion": "/v1"
},
"metricName": "nginx_vts_server_requests_per_second",
"timestamp": "2019-11-19T18:38:21Z",
"value": "1236m"
}
]
}
使用以下清单可以创建一个使用这些指标的HPA。
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: nginx-custom-hpa
namespace: nginx
spec:
scaleTargetRef:
apiVersion: extensions/v1beta1
kind: Deployment
name: nginx-deployment
minReplicas: 2
maxReplicas: 10
metrics:
- type: Pods
pods:
metricName: nginx_vts_server_requests_per_second
targetAverageValue: 4000m
当您应用此清单时，您可以确认HPA的当前状态如下。
root$ kubectl describe hpa
Name: nginx-custom-hpa
Namespace: nginx
Labels: <none>
Annotations: autoscaling.alpha.kubernetes.io/metrics:
[{"type":"Pods","pods":{"metricName":"nginx_vts_server_requests_per_second","targetAverageValue":"4"}}]
kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"autoscaling/v2beta1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"nginx-custom-hpa","namespace":"n...
CreationTimestamp: Thu, 21 Nov 2019 11:11:05 -0800
Reference: Deployment/nginx-deployment
Min replicas: 2
Max replicas: 10
Deployment pods: 0 current / 0 desired
Events: <none>
那么，让我们给服务增加一些负载吧。我们可以使用一个叫做vegeta的工具来实现这个目的。
在另一个终端上，执行以下命令。
echo "GET http://nginx.gotham.com/" | vegeta attack -rate=5 -duration=0 | vegeta report
同时监控nginx的Pod和水平Pod自动缩放器，会得到以下结果。
root$ kubectl get -w pods
NAME READY STATUS RESTARTS AGE
nginx-deployment-65d8df7488-mwzxg 1/1 Running 0 9h
nginx-deployment-65d8df7488-sbp95 1/1 Running 0 4m9s
NAME AGE
nginx-deployment-65d8df7488-pwjzm 0s
nginx-deployment-65d8df7488-pwjzm 0s
nginx-deployment-65d8df7488-pwjzm 0s
nginx-deployment-65d8df7488-pwjzm 2s
nginx-deployment-65d8df7488-pwjzm 4s
nginx-deployment-65d8df7488-jvbvp 0s
nginx-deployment-65d8df7488-jvbvp 0s
nginx-deployment-65d8df7488-jvbvp 1s
nginx-deployment-65d8df7488-jvbvp 4s
nginx-deployment-65d8df7488-jvbvp 7s
nginx-deployment-65d8df7488-skjkm 0s
nginx-deployment-65d8df7488-skjkm 0s
nginx-deployment-65d8df7488-jh5vw 0s
nginx-deployment-65d8df7488-skjkm 0s
nginx-deployment-65d8df7488-jh5vw 0s
nginx-deployment-65d8df7488-jh5vw 1s
nginx-deployment-65d8df7488-skjkm 2s
nginx-deployment-65d8df7488-jh5vw 2s
nginx-deployment-65d8df7488-skjkm 3s
nginx-deployment-65d8df7488-jh5vw 4s
root$ kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
nginx-custom-hpa Deployment/nginx-deployment 5223m/4 2 10 3 5m5s
我们清楚地看到水平Pod自动缩放器扩展了Pod以满足需求。中断vegeta命令会显示vegeta报告。这清楚地表明所有请求都已被应用程序处理完毕。
root$ echo "GET http://nginx.gotham.com/" | vegeta attack -rate=5 -duration=0 | vegeta report
^CRequests [total, rate, throughput] 224, 5.02, 5.02
Duration [total, attack, wait] 44.663806863s, 44.601823883s, 61.98298ms
Latencies [mean, 50, 95, 99, max] 63.3879ms, 60.867241ms, 79.414139ms, 111.981619ms, 229.310088ms
Bytes In [total, mean] 137088, 612.00
Bytes Out [total, mean] 0, 0.00
Success [ratio] 100.00%
Status Codes [code:count] 200:224
Error Set:
4. 总结
这个设置使用 Prometheus 适配器，解释了根据一些自定义指标来自动扩展部署的方法。为了简单起见，从 Prometheus 服务器获取的指标只有一个。然而，通过扩展适配器配置映射，您可以获取一部分或全部可用的指标，并将它们用于自动扩展。
如果Prometheus安装在Kubernetes集群外部，只需确保集群可以访问查询端点，然后在适配器部署清单中进行更新。可以想象出复杂的场景，可以获取多个指标并组合进行扩展决策。
如果你对使用Prometheus感兴趣，但担心配置和管理会消耗太多资源，你可以预约一次演示，并咨询Hosted Prometheus如何适配监控环境。另外，你也可以直接访问免费试用，立即进行确认，务必进行查看！
这篇文章是由客座博客Vaibhav Thakur撰写的。如果你喜欢这篇文章，请在LinkedIn上查看更多关于他的详细信息。
那么，我们下一篇文章再见！
