[K8s解决方案] 在 v1.11 版本中已修复的 Liveness Probe 的 failureThreshold 的意外行为
增加记录
我在Slack的Kubernetes的jp-users频道发了帖子，收到了nekop先生下面的建议。非常感谢！
https://github.com/kubernetes/kubernetes/pull/46371/files
https://github.com/sjenning/kubernetes/blob/2c866a7aaa45dd552df9ea53cc1ca20b0d502a52/pkg/kubelet/prober/worker.go#L227
https://github.com/sjenning/kubernetes/blob/master/pkg/kubelet/prober/worker.go#L243
https://github.com/sjenning/kubernetes/commit/b49e07846f6779fdb35efb33eab5456bb85ec9f8
https://github.com/kubernetes/kubernetes/commit/b49e07846f6779fdb35efb33eab5456bb85ec9f8
以下是一些选项：
1. https://github.com/kubernetes/kubernetes/pull/46371/files
https://github.com/sjenning/kubernetes/blob/2c866a7aaa45dd552df9ea53cc1ca20b0d502a52/pkg/kubelet/prober/worker.go#L227
https://github.com/sjenning/kubernetes/blob/master/pkg/kubelet/prober/worker.go#L243
https://github.com/sjenning/kubernetes/commit/b49e07846f6779fdb35efb33eab5456bb85ec9f8
https://github.com/kubernetes/kubernetes/commit/b49e07846f6779fdb35efb33eab5456bb85ec9f8
「既然说『1.11及以后是否修好了呢？』，那我就真的试着在1.11版本上操作了一下。」
环境
minikube v0.28.2
然而，这次我们将使用–kubernetes-version v1.11.3来更改版本。
（之前使用的是1.10系列）
kubectl version
#Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-09T18:02:47Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
#Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-09T17:53:03Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
看起来正在运行的是1.11.3版本。
我们将使用与上次相同的 Deployment.yml 文件。
确认连通
0s 0s 1 nginx-deployment-54d7b449d9-xwmsf.15588f8e14305452 Pod spec.containers{nginx} Normal Pulled kubelet, minikube Container image "nginx:1.7.9" already present on machine
0s 0s 1 nginx-deployment-54d7b449d9-xwmsf.15588f8e24353f4d Pod spec.containers{nginx} Normal Created kubelet, minikube Created container
0s 0s 1 nginx-deployment-54d7b449d9-xwmsf.15588f8e2f30bd16 Pod spec.containers{nginx} Normal Started kubelet, minikube Started container
0s 0s 1 nginx-deployment-54d7b449d9-xwmsf.15588f91418ce085 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 5s 2 nginx-deployment-54d7b449d9-xwmsf.15588f91418ce085 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 10s 3 nginx-deployment-54d7b449d9-xwmsf.15588f91418ce085 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 15s 4 nginx-deployment-54d7b449d9-xwmsf.15588f91418ce085 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 20s 5 nginx-deployment-54d7b449d9-xwmsf.15588f91418ce085 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 0s 1 nginx-deployment-54d7b449d9-xwmsf.15588f9607af13d7 Pod spec.containers{nginx} Normal Killing kubelet, minikube Killing container with id docker://nginx:Container failed liveness probe.. Container will be killed and recreated.
0s 34s 2 nginx-deployment-54d7b449d9-xwmsf.15588f8e14305452 Pod spec.containers{nginx} Normal Pulled kubelet, minikube Container image "nginx:1.7.9" already present on machine
0s 34s 2 nginx-deployment-54d7b449d9-xwmsf.15588f8e24353f4d Pod spec.containers{nginx} Normal Created kubelet, minikube Created container
0s 34s 2 nginx-deployment-54d7b449d9-xwmsf.15588f8e2f30bd16 Pod spec.containers{nginx} Normal Started kubelet, minikube Started container
0s 35s 6 nginx-deployment-54d7b449d9-xwmsf.15588f91418ce085 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 40s 7 nginx-deployment-54d7b449d9-xwmsf.15588f91418ce085 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 45s 8 nginx-deployment-54d7b449d9-xwmsf.15588f91418ce085 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 50s 9 nginx-deployment-54d7b449d9-xwmsf.15588f91418ce085 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 55s 10 nginx-deployment-54d7b449d9-xwmsf.15588f91418ce085 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 35s 2 nginx-deployment-54d7b449d9-xwmsf.15588f9607af13d7 Pod spec.containers{nginx} Normal Killing kubelet, minikube Killing container with id docker://nginx:Container failed liveness probe.. Container will be killed and recreated.
0s 1m 3 nginx-deployment-54d7b449d9-xwmsf.15588f8e14305452 Pod spec.containers{nginx} Normal Pulled kubelet, minikube Container image "nginx:1.7.9" already present on machine
第二个活动探针的检查次数如预期一样出现了5次。
正如nekop先生所指出的结果一样！
总结
-
- 1.10系での2回目移行のfailureThresholdが1回分少ないのは不具合
- v1.11系で修正済み
「這應該不是故障，應該就是這樣的吧。」這樣在我心中一直這麼想著，但我自己也得好好調查一下才行呢……。
我感覺必須能夠大致知道哪個組件受到了影響。
（即使只是稍微去觸碰一下源代碼……。）
过去[正在调查中] Kubernetes 的 Liveness Probe 的 failureThreshold
首先
当进行Kubernetes健康检查故障阈值(failureThreshold)的验证时，与我预期的不同，所以我做了笔记。
第一次的故障阈值是根据指定值进行容器重新启动的，但重新启动后，似乎是按照故障阈值减1的次数进行重新启动的行为。
顺畅交流的环境
minikube版本为v0.28.2。
部署
对不存在的路径进行每5秒一次的健康检查，并故意让其失败。
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80
livenessProbe:
httpGet:
path: /pathnotfound # 存在しないパスのため404
port: 80
initialDelaySeconds: 10
periodSeconds: 5
failureThreshold: 5
预期行为与实际行为相符
预期的行动
[创建容器]
[启动容器]
[不健康（第一次）]
[不健康（第二次）]
[不健康（第三次）]
[不健康（第四次）]
[不健康（第五次）]（失败阈值）
[杀死容器]
[创建容器]
[启动容器]
[不健康（第六次）]
[不健康（第七次）]
[不健康（第八次）]
[不健康（第九次）]
[不健康（第十次）]（失败阈值）
[杀死容器]
实际行动
创建容器
启动容器
不健康（第一次）
不健康（第二次）
不健康（第三次）
不健康（第四次）
不健康（第五次）（失败阈值）正常
终止容器
创建容器
启动容器
不健康（第六次）
不健康（第七次）
不健康（第八次）
不健康（第九次）（失败阈值）与预期不符
终止容器
kubectl查看事件的结果
0s 0s 1 nginx-deployment-5cdfc956db-n56dj.155845c2e184ea50 Pod spec.containers{nginx} Normal Pulled kubelet, minikube Container image "nginx:1.7.9" already present on machine
0s 0s 1 nginx-deployment-5cdfc956db-n56dj.155845c2edc3eb4f Pod spec.containers{nginx} Normal Created kubelet, minikube Created container
0s 0s 1 nginx-deployment-5cdfc956db-n56dj.155845c2fa234bdc Pod spec.containers{nginx} Normal Started kubelet, minikube Started container
0s 0s 1 nginx-deployment-5cdfc956db-n56dj.155845c618efd412 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 5s 2 nginx-deployment-5cdfc956db-n56dj.155845c618efd412 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 10s 3 nginx-deployment-5cdfc956db-n56dj.155845c618efd412 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 15s 4 nginx-deployment-5cdfc956db-n56dj.155845c618efd412 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 20s 5 nginx-deployment-5cdfc956db-n56dj.155845c618efd412 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 0s 1 nginx-deployment-5cdfc956db-n56dj.155845cadd2a06c2 Pod spec.containers{nginx} Normal Killing kubelet, minikube Killing container with id docker://nginx:Container failed liveness probe.. Container will be killed and recreated.
0s 34s 2 nginx-deployment-5cdfc956db-n56dj.155845c2e184ea50 Pod spec.containers{nginx} Normal Pulled kubelet, minikube Container image "nginx:1.7.9" already present on machine
0s 34s 2 nginx-deployment-5cdfc956db-n56dj.155845c2edc3eb4f Pod spec.containers{nginx} Normal Created kubelet, minikube Created container
0s 34s 2 nginx-deployment-5cdfc956db-n56dj.155845c2fa234bdc Pod spec.containers{nginx} Normal Started kubelet, minikube Started container
0s 35s 6 nginx-deployment-5cdfc956db-n56dj.155845c618efd412 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 40s 7 nginx-deployment-5cdfc956db-n56dj.155845c618efd412 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 45s 8 nginx-deployment-5cdfc956db-n56dj.155845c618efd412 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 50s 9 nginx-deployment-5cdfc956db-n56dj.155845c618efd412 Pod spec.containers{nginx} Warning Unhealthy kubelet, minikube Liveness probe failed: HTTP probe failed with statuscode: 404
0s 30s 2 nginx-deployment-5cdfc956db-n56dj.155845cadd2a06c2 Pod spec.containers{nginx} Normal Killing kubelet, minikube Killing container with id docker://nginx:Container failed liveness probe.. Container will be killed and recreated.
0s 1m 3 nginx-deployment-5cdfc956db-n56dj.155845c2e184ea50 Pod spec.containers{nginx} Normal Pulled kubelet, minikube Container image "nginx:1.7.9" already present on machine
