在节点上查看OpenShift Pod的磁盘使用量
首先
在考虑ROSAN（Red Hat OpenShift Service on AWS）的配置时，我们注意到默认情况下配置了未使用的PV的Prometheus的Pod正在消耗Infra/Worker节点上的磁盘空间，因此我们进行了确认并记录了相关内容。
本文的内容并不是专门针对Prometheus的，而是关于如何确认OpenShift的Pod在Worker节点上使用了多少磁盘空间的方法。
环境
-
- ROSA ver.4.10
-
- STS＆PrivateLinkモード
-
- Workerノード数：3台
- 複数台のテスト用アプリPodが稼働する環境
确认步骤
手順概述如下。
-
- 确认目标 Pod 在运行的 Worker 节点上的特定位置
-
- 通过 oc debug node 命令访问 Infra/Worker 节点
-
- 使用 crictl 命令确认 Pod 的信息
- 使用 du 命令确认磁盘使用量
虽然可以直接访问Pod来确认，但在这里故意选择了在节点上进行确认的方法。
特定运行工作节点的Pod确认目标。
使用oc get pod -o wide命令来确认OpenShift-Monitoring项目中的Pod。
$ oc get pod -n openshift-monitoring -o wide
NAME READY STATUS RESTARTS AGE IP NODE
NOMINATED NODE READINESS GATES
alertmanager-main-0 6/6 Running 0 62d 10.128.4.7 ip-10-0-135-107.ec2.internal <none> <none>
alertmanager-main-1 6/6 Running 0 62d 10.131.2.8 ip-10-0-182-241.ec2.internal <none> <none>
cluster-monitoring-operator-99fb67bf4-784lp 2/2 Running 0 62d 10.128.0.39 ip-10-0-221-180.ec2.internal <none> <none>
configure-alertmanager-operator-56ff578d77-sktsq 1/1 Running 0 9d 10.131.2.88 ip-10-0-182-241.ec2.internal <none> <none>
configure-alertmanager-operator-registry-6knhb 1/1 Running 0 56d 10.131.1.67 ip-10-0-215-48.ec2.internal <none> <none>
grafana-65dc6dd678-r89dp 3/3 Running 0 62d 10.128.4.9 ip-10-0-135-107.ec2.internal <none> <none>
kube-state-metrics-66bf46c69d-lfxkb 3/3 Running 0 62d 10.128.4.20 ip-10-0-135-107.ec2.internal <none> <none>
node-exporter-45hsx 2/2 Running 2 62d 10.0.182.241 ip-10-0-182-241.ec2.internal <none> <none>
node-exporter-5hz6v 2/2 Running 2 62d 10.0.190.186 ip-10-0-190-186.ec2.internal <none> <none>
node-exporter-864s2 2/2 Running 2 62d 10.0.153.196 ip-10-0-153-196.ec2.internal <none> <none>
node-exporter-bmxkn 2/2 Running 2 62d 10.0.221.180 ip-10-0-221-180.ec2.internal <none> <none>
node-exporter-bq7n5 2/2 Running 2 62d 10.0.135.140 ip-10-0-135-140.ec2.internal <none> <none>
node-exporter-jr47c 2/2 Running 2 62d 10.0.135.107 ip-10-0-135-107.ec2.internal <none> <none>
node-exporter-p5xwt 2/2 Running 2 62d 10.0.199.127 ip-10-0-199-127.ec2.internal <none> <none>
node-exporter-szrfv 2/2 Running 0 49d 10.0.171.183 ip-10-0-171-183.ec2.internal <none> <none>
node-exporter-x6g8w 2/2 Running 2 62d 10.0.215.48 ip-10-0-215-48.ec2.internal <none> <none>
openshift-state-metrics-5d5454f44d-mbk8w 3/3 Running 0 62d 10.128.4.10 ip-10-0-135-107.ec2.internal <none> <none>
osd-rebalance-infra-nodes-27828240-rnjjn 0/1 Completed 0 40m 10.128.4.136 ip-10-0-135-107.ec2.internal <none> <none>
osd-rebalance-infra-nodes-27828255-hrxd9 0/1 Completed 0 25m 10.128.4.140 ip-10-0-135-107.ec2.internal <none> <none>
osd-rebalance-infra-nodes-27828270-wfw5s 0/1 Completed 0 10m 10.128.4.143 ip-10-0-135-107.ec2.internal <none> <none>
prometheus-adapter-7bd5fcb654-h6x2h 1/1 Running 0 2d14h 10.128.4.205 ip-10-0-135-107.ec2.internal <none> <none>
prometheus-adapter-7bd5fcb654-j5pqs 1/1 Running 0 2d14h 10.130.2.38 ip-10-0-199-127.ec2.internal <none> <none>
prometheus-k8s-0 6/6 Running 0 62d 10.131.2.7 ip-10-0-182-241.ec2.internal <none> <none>
prometheus-k8s-1 6/6 Running 0 62d 10.130.2.7 ip-10-0-199-127.ec2.internal <none> <none>
prometheus-operator-68d866488f-pf7dw 2/2 Running 0 62d 10.128.4.11 ip-10-0-135-107.ec2.internal <none> <none>
sre-dns-latency-exporter-6qr68 1/1 Running 3 139d 10.131.2.5 ip-10-0-182-241.ec2.internal <none> <none>
sre-dns-latency-exporter-762sp 1/1 Running 0 49d 10.129.4.5 ip-10-0-171-183.ec2.internal <none> <none>
sre-dns-latency-exporter-8qvvs 1/1 Running 2 139d 10.129.0.9 ip-10-0-153-196.ec2.internal <none> <none>
sre-dns-latency-exporter-cmmr5 1/1 Running 2 139d 10.128.4.5 ip-10-0-135-107.ec2.internal <none> <none>
sre-dns-latency-exporter-f7n8c 1/1 Running 1 139d 10.130.2.4 ip-10-0-199-127.ec2.internal <none> <none>
sre-dns-latency-exporter-fqg68 1/1 Running 2 139d 10.128.2.2 ip-10-0-135-140.ec2.internal <none> <none>
sre-dns-latency-exporter-htjnf 1/1 Running 2 139d 10.131.0.5 ip-10-0-215-48.ec2.internal <none> <none>
sre-dns-latency-exporter-kflb7 1/1 Running 2 139d 10.128.0.2 ip-10-0-221-180.ec2.internal <none> <none>
sre-dns-latency-exporter-phqxg 1/1 Running 2 139d 10.130.0.12 ip-10-0-190-186.ec2.internal <none> <none>
telemeter-client-774cd546df-q6j4b 3/3 Running 0 62d 10.128.4.19 ip-10-0-135-107.ec2.internal <none> <none>
thanos-querier-55dd66b46-fvclz 6/6 Running 0 62d 10.128.4.17 ip-10-0-135-107.ec2.internal <none> <none>
thanos-querier-55dd66b46-gkpsr 6/6 Running 0 62d 10.130.2.17 ip-10-0-199-127.ec2.internal <none> <none>
token-refresher-7f8c567555-qwwn2 1/1 Running 0 62d 10.130.2.18 ip-10-0-199-127.ec2.internal <none> <none>
由于Pod数众多，因此只需要grep Prometheus。
$ oc get pod -n openshift-monitoring -o wide | grep prome
prometheus-adapter-7bd5fcb654-h6x2h 1/1 Running 0 2d14h 10.128.4.205 ip-10-0-135-107.ec2.internal <none> <none>
prometheus-adapter-7bd5fcb654-j5pqs 1/1 Running 0 2d14h 10.130.2.38 ip-10-0-199-127.ec2.internal <none> <none>
prometheus-k8s-0 6/6 Running 0 62d 10.131.2.7 ip-10-0-182-241.ec2.internal <none> <none>
prometheus-k8s-1 6/6 Running 0 62d 10.130.2.7 ip-10-0-199-127.ec2.internal <none> <none>
prometheus-operator-68d866488f-pf7dw 2/2 Running 0 62d 10.128.4.11 ip-10-0-135-107.ec2.internal <none> <none>
→ Prometheus 在以下节点上运行。
-
- prometheus-k8s-0: ip-10-0-182-241.ec2.internal
-
- prometheus-k8s-1: ip-10-0-199-127.ec2.internal
- ※今回はprometheusのpodを確認対象とした。
上述的节点作为角色被标记为Infra/Worker。
$ oc get no -o wide
（infra,workerロールのノードのみ抜粋）
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ip-10-0-135-107.ec2.internal Ready infra,worker 141d v1.23.5+012e945 10.0.135.107 <none> Red Hat Enterprise Linux CoreOS 410.84.202208161501-0 (Ootpa) 4.18.0-305.57.1.el8_4.x86_64 cri-o://1.23.3-15.rhaos4.10.git6af791c.el8
ip-10-0-182-241.ec2.internal Ready infra,worker 141d v1.23.5+012e945 10.0.182.241 <none> Red Hat Enterprise Linux CoreOS 410.84.202208161501-0 (Ootpa) 4.18.0-305.57.1.el8_4.x86_64 cri-o://1.23.3-15.rhaos4.10.git6af791c.el8ip-
ip-10-0-199-127.ec2.internal Ready infra,worker 141d v1.23.5+012e945 10.0.199.127 <none> Red Hat Enterprise Linux CoreOS 410.84.202208161501-0 (Ootpa) 4.18.0-305.57.1.el8_4.x86_64 cri-o://1.23.3-15.rhaos4.10.git6af791c.el8
使用oc debug node命令访问Infra/Worker节点。
使用 oc debug node 命令访问 Prometheus Pod 运行的 Worker 节点。
$ oc debug node/ip-10-0-182-241.ec2.internal
Starting pod/ip-10-0-182-241ec2internal-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.182.241
If you don't see a command prompt, try pressing enter.
sh-4.4#
sh-4.4# chroot /host
sh-4.4#
为了执行命令，请提前执行chroot /host。
使用crictl命令来确认Pod的信息。
为了确认稼动中的pod的信息，执行crictl命令。
sh-4.4# crictl ps
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID
250070a9b431a a1ce65e6ae985f352a5e339a1392420ac4279eef182d2e20c191b3ba84d04fc7 3 minutes ago Running container-00 0 88e93156d908a
13a02acb6bb30 quay.io/app-sre/splunk-forwarder@sha256:afc413cd2504b586b6f28c16355db243c8bfdef536cc80e44f61e03c01e12235 21 hours ago Running splunk-uf 0 9fe134fcbf6d6
531afe579fa6a dcfb77e8999b71b39fa062d4f274a0784943a017991cf5d5a4058c08ed9f7554 8 days ago Running webhook 0 a4c9e48f9316c
535cb9dadc7ae quay.io/app-sre/addon-operator-manager@sha256:41a7cf65835e4229e4468593a64ab0cdc4dac5a265aab88a379216c5eef688d6 8 days ago Running manager 0 0b4637b4ee99f
439339a0a779c 19fe8540f8737eec13b6ef65a0acfe4135415299551fed824afaefcb3067f7ad 8 days ago Running metrics-relay-server 0 0b4637b4ee99f
626378edf135d quay.io/app-sre/configure-alertmanager-operator@sha256:6c86f39326fb76657d4ed64a5bc0470374574be30507cb67d3ffb1ac78e6f851 9 days ago Running configure-alertmanager-operator 0 13fcca973bbe5
02c392154d4fb quay.io/app-sre/ocm-agent@sha256:51bafd25d611662abd8937ff3914b9a4ae5ef9361719e38947a56fd38199014d 12 days ago Running ocm-agent 0 78d512939824c
370947ae91b06 quay.io/app-sre/ocm-agent-operator@sha256:d697996b5051ee1fdf174a7d23fdd0f0fe612876e8b0d8e94a0c92fee72ea011 13 days ago Running ocm-agent-operator 0 d4e6016222c39
fbd22d001eef5 quay.io/app-sre/custom-domains-operator@sha256:59158acb6a90927f89d85ffaf448e4831b8e8b352d4539b597b97e5f5ff92f6b 4 weeks ago Running custom-domains-operator 0 c4f5c43d5d5eb
9208b357e7965 quay.io/app-sre/must-gather-operator@sha256:2e9c61f5bb3d7f95805130843b0368853d38934756e892954cdf3e251941ac43 6 weeks ago Running must-gather-operator 0 27b5ebcdd77e4
54c912fdc02c5 quay.io/app-sre/managed-node-metadata-operator@sha256:81cb5d1fcc8261858727876036c86fcf45b5b3cf9f5129450019c6f107cd11da 7 weeks ago Running managed-node-metadata-operator 0 aa02b2b6f8f3e
4d971e953ad86 410e15456e199925ac531d55aa982db94829abd83b825a1f3c8ef8ad367ec3a8 2 months ago Running prom-label-proxy 0 32e6aba11b9b3
340cf915b365c b0eb8d3033872241913cc97cb4ef9e9b8e52e3584b8c3a3dfbc38332f17501aa 2 months ago Running kube-rbac-proxy-metric 0 32e6aba11b9b3
6fb5f39097c94 b0eb8d3033872241913cc97cb4ef9e9b8e52e3584b8c3a3dfbc38332f17501aa 2 months ago Running kube-rbac-proxy 0 32e6aba11b9b3
c4ab5a439405d 4a35485847278231eaefeeba227df3b848fcd88251bf387cc75814683c6e1cc9 2 months ago Running alertmanager-proxy 0 32e6aba11b9b3
44aba6706a156 d7e69e3e6c77da1eb69a016accb5755bd259cdf6b08bbe2505482bdaf191c1a7 2 months ago Running config-reloader 0 32e6aba11b9b3
43a4c4e9f1aa1 5b7a0f5af01de4e8ccd7d80557c92f25452b8ae54423ebbd93cd6b8cc11bf086 2 months ago Running alertmanager 0 32e6aba11b9b3
0a26c550f6b59 b0eb8d3033872241913cc97cb4ef9e9b8e52e3584b8c3a3dfbc38332f17501aa 2 months ago Running kube-rbac-proxy-thanos 0 fde2304412307
bc28291be733a b0eb8d3033872241913cc97cb4ef9e9b8e52e3584b8c3a3dfbc38332f17501aa 2 months ago Running kube-rbac-proxy 0 fde2304412307
9d94467cb5722 4a35485847278231eaefeeba227df3b848fcd88251bf387cc75814683c6e1cc9 2 months ago Running prometheus-proxy 0 fde2304412307
4359fb8b597b6 1cff5886ffcf498a37eb3868762403fb2b0ca2feb4a14c98471fc256b680619e 2 months ago Running thanos-sidecar 0 fde2304412307
01b339ba5ea1d d7e69e3e6c77da1eb69a016accb5755bd259cdf6b08bbe2505482bdaf191c1a7 2 months ago Running config-reloader 0 fde2304412307
27cccaa137e39 d3017e38d93fb39bcad5ae9ee8521f5829c774c389ae7e65e44a00388d67cf1e 2 months ago Running prometheus 0 fde2304412307
9789aad219d40 b08d2cd9618b330d905aeafe0bb8187a768fbe94c17fe9916d088dbf59acefc3 2 months ago Running kube-multus-additional-cni-plugins 1 923c608753588
d05f57994cae1 46bea5c4f6e46d4112dbbf837186ba68c231c9edd3bb22ccfbf76ad32a329ffd 2 months ago Running main 3 6beefa759e932
86c189c7529f6 53ba5113526775eab777eed2bfdd9d6148ae563a3330f609394dcca565283f7b 2 months ago Running serve-healthcheck-canary 1 989fc22ba34bb
fd0a17b4d8ecb 77a2af086aeb1c31914686118f3f20e44f836a251d89f256c5620cda914445a9 2 months ago Running network-check-target-container 1 0139053cefa0d
5ed1426014c35 b0eb8d3033872241913cc97cb4ef9e9b8e52e3584b8c3a3dfbc38332f17501aa 2 months ago Running kube-rbac-proxy 1 f57ba7ed09785
1032c983cbbe0 3e0c06861c106b17273bd4c553ca4be6f34ca3cc226f89b2092aa9d7062a735d 2 months ago Running network-metrics-daemon 1 f57ba7ed09785
c8dc4f438ce2e b0eb8d3033872241913cc97cb4ef9e9b8e52e3584b8c3a3dfbc38332f17501aa 2 months ago Running kube-rbac-proxy 1 f74f3801db28d
f6453301898b4 fa34f535a918d4d6d07e8b105b9209dfd8b6c9152cee340e10f55934d6b07690 2 months ago Running node-exporter 1 f74f3801db28d
56010ff9bb875 75ae6c537332567afdc034809cf030f0cfea3cb7ecbc2557be821406abf5adb4 2 months ago Running csi-liveness-probe 1 fc7868d48269e
360e48e827df6 b0eb8d3033872241913cc97cb4ef9e9b8e52e3584b8c3a3dfbc38332f17501aa 2 months ago Running kube-rbac-proxy 1 aa3699225a6dd
61b3fe4869316 4a35485847278231eaefeeba227df3b848fcd88251bf387cc75814683c6e1cc9 2 months ago Running oauth-proxy 1 6c0bdff805442
5ca4224aa30f6 97448add286660fbbf4e14abb55c4a728c07162727cf7f7753c014c6540ba9b7 2 months ago Running csi-node-driver-registrar 1 fc7868d48269e
4cc36ee09e831 f44d6c1a2dad7deef542a030cce029d3904da2d69a2a7203094e8e7a15d30883 2 months ago Running dns-node-resolver 1 3a86f4d95389b
7d70ca3301053 bded42f93595dfe68ce77cac9a401479c0e60180b82947576276e95319fd185c 2 months ago Running sdn 1 aa3699225a6dd
8a8fb4dd0b34f 9e085bf5fd76546272ae0de97e127cceb9fa19518128928f4ea1ff19794a420d 2 months ago Running csi-driver 1 fc7868d48269e
b46475b65fe1b 3e34a83ad92bf15167ec7b450520a61bb5028985ac19ce0bf16aa89d84b58c0f 2 months ago Running machine-config-daemon 1 6c0bdff805442
ab100fa37c49f 8cc9e7e28492bd39a5a0efde92b8a11cbbeb7066e907cb25dab5d7d1ad346477 2 months ago Running node-ca 1 dfb3974a993f5
8a7b1ea93c0c3 b08d2cd9618b330d905aeafe0bb8187a768fbe94c17fe9916d088dbf59acefc3 2 months ago Running kube-multus 1 628ede44e698b
3918dd5445192 57d6bfa9aa0faa2b850635e158e56230035e1910361d5412bb939ec7fe409527 2 months ago Running tuned 1 04b08f03e5865
sh-4.4#
因为pod数目较多，所以使用grep命令。
sh-4.4# crictl ps | grep prometheus
9d94467cb5722 4a35485847278231eaefeeba227df3b848fcd88251bf387cc75814683c6e1cc9 2 months ago Running prometheus-proxy 0 fde2304412307
27cccaa137e39 d3017e38d93fb39bcad5ae9ee8521f5829c774c389ae7e65e44a00388d67cf1e 2 months ago Running prometheus 0 fde2304412307
sh-4.4#
为了查看Prometheus Pod的详细信息，执行crictl inspect命令。(虽然很长，但是我会全部写出来)
sh-4.4# crictl inspect 27cccaa137e39
{
"status": {
"id": "27cccaa137e395427c3ddaef728528ddfd3e1d83cd2710a9d15fa1202e93e084",
"metadata": {
"attempt": 0,
"name": "prometheus"
},
"state": "CONTAINER_RUNNING",
"createdAt": "2022-09-27T08:46:55.100751755Z",
"startedAt": "2022-09-27T08:46:55.126901515Z",
"finishedAt": "0001-01-01T00:00:00Z",
"exitCode": 0,
"image": {
"annotations": {},
"image": "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00d156818e2e4264c3268c9ae41de9b561e2811363b64515ca64865c992ec8ea"
},
"imageRef": "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00d156818e2e4264c3268c9ae41de9b561e2811363b64515ca64865c992ec8ea",
"reason": "",
"message": "",
"labels": {
"io.kubernetes.container.name": "prometheus",
"io.kubernetes.pod.name": "prometheus-k8s-0",
"io.kubernetes.pod.namespace": "openshift-monitoring",
"io.kubernetes.pod.uid": "a4477163-737d-457c-8698-8688fdd00003"
},
"annotations": {
"io.kubernetes.container.hash": "7ba9b53",
"io.kubernetes.container.restartCount": "0",
"io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
"io.kubernetes.container.terminationMessagePolicy": "FallbackToLogsOnError",
"io.kubernetes.pod.terminationGracePeriod": "600"
},
"mounts": [
{
"containerPath": "/prometheus",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volume-subpaths/pvc-b17ae945-02d2-43e2-a2d2-23891203b1f3/prometheus/3",
"propagation": "PROPAGATION_PRIVATE",
"readonly": false,
"selinuxRelabel": true
},
{
"containerPath": "/dev/termination-log",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/containers/prometheus/5ab56fc4",
"propagation": "PROPAGATION_PRIVATE",
"readonly": false,
"selinuxRelabel": true
},
{
"containerPath": "/etc/hosts",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/etc-hosts",
"propagation": "PROPAGATION_PRIVATE",
"readonly": false,
"selinuxRelabel": true
},
{
"containerPath": "/etc/prometheus/config_out",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~empty-dir/config-out",
"propagation": "PROPAGATION_PRIVATE",
"readonly": true,
"selinuxRelabel": true
},
{
"containerPath": "/etc/prometheus/certs",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~projected/tls-assets",
"propagation": "PROPAGATION_PRIVATE",
"readonly": true,
"selinuxRelabel": true
},
{
"containerPath": "/etc/prometheus/secrets/prometheus-k8s-thanos-sidecar-tls",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls",
"propagation": "PROPAGATION_PRIVATE",
"readonly": true,
"selinuxRelabel": true
},
{
"containerPath": "/etc/prometheus/secrets/kube-etcd-client-certs",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-kube-etcd-client-certs",
"propagation": "PROPAGATION_PRIVATE",
"readonly": true,
"selinuxRelabel": true
},
{
"containerPath": "/etc/prometheus/secrets/prometheus-k8s-tls",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls",
"propagation": "PROPAGATION_PRIVATE",
"readonly": true,
"selinuxRelabel": true
},
{
"containerPath": "/etc/prometheus/secrets/prometheus-k8s-proxy",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-proxy",
"propagation": "PROPAGATION_PRIVATE",
"readonly": true,
"selinuxRelabel": true
},
{
"containerPath": "/etc/prometheus/web_config/web-config.yaml",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volume-subpaths/web-config/prometheus/5",
"propagation": "PROPAGATION_PRIVATE",
"readonly": true,
"selinuxRelabel": true
},
{
"containerPath": "/etc/prometheus/secrets/kube-rbac-proxy",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-kube-rbac-proxy",
"propagation": "PROPAGATION_PRIVATE",
"readonly": true,
"selinuxRelabel": true
},
{
"containerPath": "/etc/prometheus/secrets/metrics-client-certs",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-metrics-client-certs",
"propagation": "PROPAGATION_PRIVATE",
"readonly": true,
"selinuxRelabel": true
},
{
"containerPath": "/etc/prometheus/configmaps/serving-certs-ca-bundle",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-serving-certs-ca-bundle",
"propagation": "PROPAGATION_PRIVATE",
"readonly": true,
"selinuxRelabel": true
},
{
"containerPath": "/etc/prometheus/configmaps/kubelet-serving-ca-bundle",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-kubelet-serving-ca-bundle",
"propagation": "PROPAGATION_PRIVATE",
"readonly": true,
"selinuxRelabel": true
},
{
"containerPath": "/etc/prometheus/configmaps/metrics-client-ca",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-metrics-client-ca",
"propagation": "PROPAGATION_PRIVATE",
"readonly": true,
"selinuxRelabel": true
},
{
"containerPath": "/etc/prometheus/rules/prometheus-k8s-rulefiles-0",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/prometheus-k8s-rulefiles-0",
"propagation": "PROPAGATION_PRIVATE",
"readonly": true,
"selinuxRelabel": true
},
{
"containerPath": "/var/run/secrets/kubernetes.io/serviceaccount",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~projected/kube-api-access-jfzqj",
"propagation": "PROPAGATION_PRIVATE",
"readonly": true,
"selinuxRelabel": true
},
{
"containerPath": "/etc/pki/ca-trust/extracted/pem/",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/prometheus-trusted-ca-bundle",
"propagation": "PROPAGATION_PRIVATE",
"readonly": true,
"selinuxRelabel": true
}
],
"logPath": "/var/log/pods/openshift-monitoring_prometheus-k8s-0_a4477163-737d-457c-8698-8688fdd00003/prometheus/0.log"
},
"info": {
"sandboxID": "fde2304412307a44d51118bcf4d3ef4089fd71e5aa12418ca6716a6395194d7d",
"pid": 4693,
"runtimeSpec": {
"ociVersion": "1.0.2-dev",
"process": {
"user": {
"uid": 65534,
"gid": 65534,
"additionalGids": [
65534
]
},
"args": [
"/bin/prometheus",
"--web.console.templates=/etc/prometheus/consoles",
"--web.console.libraries=/etc/prometheus/console_libraries",
"--config.file=/etc/prometheus/config_out/prometheus.env.yaml",
"--storage.tsdb.path=/prometheus",
"--storage.tsdb.retention.time=11d",
"--web.enable-lifecycle",
"--web.external-url=https://prometheus-k8s-openshift-monitoring.apps.aws-rosa-xxx-xxx.xxx.xxx.openshiftapps.com/",
"--web.route-prefix=/",
"--web.listen-address=127.0.0.1:9090",
"--web.config.file=/etc/prometheus/web_config/web-config.yaml"
],
"env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"TERM=xterm",
"HOSTNAME=prometheus-k8s-0",
"NSS_SDB_USE_CACHE=no",
"GRAFANA_PORT_3000_TCP_ADDR=172.30.251.5",
"GRAFANA_PORT_3002_TCP_ADDR=172.30.251.5",
"THANOS_QUERIER_PORT_9093_TCP_PROTO=tcp",
"TOKEN_REFRESHER_SERVICE_PORT_HTTP=80",
"KUBERNETES_PORT_443_TCP_ADDR=172.30.0.1",
"CONFIGURE_ALERTMANAGER_OPERATOR_SERVICE_PORT=8080",
"CONFIGURE_ALERTMANAGER_OPERATOR_PORT_8080_TCP_PROTO=tcp",
"ALERTMANAGER_MAIN_SERVICE_HOST=172.30.199.91",
"TOKEN_REFRESHER_SERVICE_PORT=80",
"KUBERNETES_SERVICE_PORT_HTTPS=443",
"PROMETHEUS_K8S_SERVICE_PORT_WEB=9091",
"THANOS_QUERIER_PORT_9093_TCP_PORT=9093",
"PROMETHEUS_K8S_PORT_9091_TCP=tcp://172.30.214.226:9091",
"ALERTMANAGER_MAIN_PORT_9092_TCP_PROTO=tcp",
"GRAFANA_PORT_3000_TCP_PROTO=tcp",
"GRAFANA_PORT_3002_TCP_PORT=3002",
"PROMETHEUS_K8S_PORT=tcp://172.30.214.226:9091",
"THANOS_QUERIER_PORT_9093_TCP_ADDR=172.30.64.108",
"TOKEN_REFRESHER_PORT_80_TCP=tcp://172.30.227.107:80",
"ALERTMANAGER_MAIN_PORT_9097_TCP_ADDR=172.30.199.91",
"THANOS_QUERIER_PORT_9094_TCP_PROTO=tcp",
"CONFIGURE_ALERTMANAGER_OPERATOR_PORT_8080_TCP=tcp://172.30.32.44:8080",
"GRAFANA_PORT=tcp://172.30.251.5:3000",
"SRE_DNS_LATENCY_EXPORTER_PORT_80_TCP=tcp://172.30.68.79:80",
"PROMETHEUS_K8S_PORT_9092_TCP_ADDR=172.30.214.226",
"CONFIGURE_ALERTMANAGER_OPERATOR_REGISTRY_PORT_50051_TCP_PORT=50051",
"ALERTMANAGER_MAIN_PORT_9097_TCP_PORT=9097",
"KUBERNETES_PORT_443_TCP_PORT=443",
"PROMETHEUS_K8S_SERVICE_PORT_METRICS=9092",
"THANOS_QUERIER_PORT_9091_TCP_PORT=9091",
"THANOS_QUERIER_PORT_9092_TCP=tcp://172.30.64.108:9092",
"THANOS_QUERIER_PORT_9094_TCP_PORT=9094",
"ALERTMANAGER_MAIN_PORT_9094_TCP_PROTO=tcp",
"THANOS_QUERIER_SERVICE_PORT_WEB=9091",
"CONFIGURE_ALERTMANAGER_OPERATOR_PORT=tcp://172.30.32.44:8080",
"PROMETHEUS_K8S_PORT_9091_TCP_ADDR=172.30.214.226",
"ALERTMANAGER_MAIN_PORT_9094_TCP=tcp://172.30.199.91:9094",
"ALERTMANAGER_MAIN_PORT_9092_TCP_ADDR=172.30.199.91",
"PROMETHEUS_ADAPTER_PORT_443_TCP=tcp://172.30.48.128:443",
"GRAFANA_PORT_3000_TCP_PORT=3000",
"GRAFANA_PORT_3000_TCP=tcp://172.30.251.5:3000",
"CONFIGURE_ALERTMANAGER_OPERATOR_REGISTRY_SERVICE_PORT_GRPC=50051",
"CONFIGURE_ALERTMANAGER_OPERATOR_REGISTRY_PORT_50051_TCP_PROTO=tcp",
"ALERTMANAGER_MAIN_SERVICE_PORT=9094",
"PROMETHEUS_ADAPTER_SERVICE_PORT=443",
"SRE_DNS_LATENCY_EXPORTER_PORT_80_TCP_ADDR=172.30.68.79",
"THANOS_QUERIER_SERVICE_PORT_TENANCY=9092",
"THANOS_QUERIER_PORT=tcp://172.30.64.108:9091",
"THANOS_QUERIER_PORT_9091_TCP=tcp://172.30.64.108:9091",
"PROMETHEUS_ADAPTER_PORT_443_TCP_PORT=443",
"THANOS_QUERIER_PORT_9091_TCP_ADDR=172.30.64.108",
"THANOS_QUERIER_PORT_9092_TCP_PROTO=tcp",
"CONFIGURE_ALERTMANAGER_OPERATOR_PORT_8080_TCP_PORT=8080",
"ALERTMANAGER_MAIN_PORT_9094_TCP_ADDR=172.30.199.91",
"TOKEN_REFRESHER_PORT=tcp://172.30.227.107:80",
"GRAFANA_SERVICE_PORT_HTTPS=3000",
"SRE_DNS_LATENCY_EXPORTER_SERVICE_HOST=172.30.68.79",
"PROMETHEUS_K8S_PORT_9091_TCP_PORT=9091",
"THANOS_QUERIER_PORT_9094_TCP_ADDR=172.30.64.108",
"PROMETHEUS_ADAPTER_SERVICE_HOST=172.30.48.128",
"THANOS_QUERIER_SERVICE_PORT=9091",
"THANOS_QUERIER_PORT_9092_TCP_ADDR=172.30.64.108",
"THANOS_QUERIER_PORT_9093_TCP=tcp://172.30.64.108:9093",
"TOKEN_REFRESHER_PORT_80_TCP_PORT=80",
"GRAFANA_PORT_3002_TCP_PROTO=tcp",
"PROMETHEUS_K8S_PORT_9092_TCP_PROTO=tcp",
"CONFIGURE_ALERTMANAGER_OPERATOR_REGISTRY_PORT_50051_TCP_ADDR=172.30.187.125",
"ALERTMANAGER_MAIN_SERVICE_PORT_WEB=9094",
"CONFIGURE_ALERTMANAGER_OPERATOR_REGISTRY_PORT=tcp://172.30.187.125:50051",
"ALERTMANAGER_MAIN_PORT_9097_TCP=tcp://172.30.199.91:9097",
"KUBERNETES_SERVICE_PORT=443",
"GRAFANA_SERVICE_PORT=3000",
"SRE_DNS_LATENCY_EXPORTER_SERVICE_PORT_HTTP_MAIN=80",
"PROMETHEUS_K8S_PORT_9092_TCP=tcp://172.30.214.226:9092",
"PROMETHEUS_K8S_PORT_9091_TCP_PROTO=tcp",
"ALERTMANAGER_MAIN_SERVICE_PORT_METRICS=9097",
"ALERTMANAGER_MAIN_PORT_9097_TCP_PROTO=tcp",
"THANOS_QUERIER_SERVICE_PORT_METRICS=9094",
"CONFIGURE_ALERTMANAGER_OPERATOR_SERVICE_PORT_METRICS=8080",
"GRAFANA_SERVICE_HOST=172.30.251.5",
"GRAFANA_SERVICE_PORT_METRICS=3002",
"PROMETHEUS_K8S_SERVICE_PORT=9091",
"CONFIGURE_ALERTMANAGER_OPERATOR_REGISTRY_SERVICE_PORT=50051",
"THANOS_QUERIER_SERVICE_HOST=172.30.64.108",
"CONFIGURE_ALERTMANAGER_OPERATOR_SERVICE_HOST=172.30.32.44",
"GRAFANA_PORT_3002_TCP=tcp://172.30.251.5:3002",
"SRE_DNS_LATENCY_EXPORTER_SERVICE_PORT=80",
"ALERTMANAGER_MAIN_PORT_9092_TCP=tcp://172.30.199.91:9092",
"ALERTMANAGER_MAIN_PORT_9092_TCP_PORT=9092",
"THANOS_QUERIER_PORT_9092_TCP_PORT=9092",
"CONFIGURE_ALERTMANAGER_OPERATOR_PORT_8080_TCP_ADDR=172.30.32.44",
"THANOS_QUERIER_SERVICE_PORT_TENANCY_RULES=9093",
"TOKEN_REFRESHER_SERVICE_HOST=172.30.227.107",
"SRE_DNS_LATENCY_EXPORTER_PORT_80_TCP_PORT=80",
"PROMETHEUS_ADAPTER_SERVICE_PORT_HTTPS=443",
"THANOS_QUERIER_PORT_9094_TCP=tcp://172.30.64.108:9094",
"TOKEN_REFRESHER_PORT_80_TCP_PROTO=tcp",
"SRE_DNS_LATENCY_EXPORTER_PORT_80_TCP_PROTO=tcp",
"ALERTMANAGER_MAIN_SERVICE_PORT_TENANCY=9092",
"ALERTMANAGER_MAIN_PORT_9094_TCP_PORT=9094",
"PROMETHEUS_ADAPTER_PORT_443_TCP_ADDR=172.30.48.128",
"THANOS_QUERIER_PORT_9091_TCP_PROTO=tcp",
"KUBERNETES_PORT_443_TCP=tcp://172.30.0.1:443",
"KUBERNETES_PORT_443_TCP_PROTO=tcp",
"ALERTMANAGER_MAIN_PORT=tcp://172.30.199.91:9094",
"KUBERNETES_SERVICE_HOST=172.30.0.1",
"KUBERNETES_PORT=tcp://172.30.0.1:443",
"CONFIGURE_ALERTMANAGER_OPERATOR_REGISTRY_PORT_50051_TCP=tcp://172.30.187.125:50051",
"PROMETHEUS_ADAPTER_PORT=tcp://172.30.48.128:443",
"PROMETHEUS_ADAPTER_PORT_443_TCP_PROTO=tcp",
"CONFIGURE_ALERTMANAGER_OPERATOR_REGISTRY_SERVICE_HOST=172.30.187.125",
"TOKEN_REFRESHER_PORT_80_TCP_ADDR=172.30.227.107",
"SRE_DNS_LATENCY_EXPORTER_PORT=tcp://172.30.68.79:80",
"PROMETHEUS_K8S_SERVICE_HOST=172.30.214.226",
"PROMETHEUS_K8S_PORT_9092_TCP_PORT=9092",
"__doozer=merge",
"BUILD_RELEASE=202208150436.p0.g3c6cd55.assembly.stream",
"BUILD_VERSION=v4.10.0",
"OS_GIT_MAJOR=4",
"OS_GIT_MINOR=10",
"OS_GIT_PATCH=0",
"OS_GIT_TREE_STATE=clean",
"OS_GIT_VERSION=4.10.0-202208150436.p0.g3c6cd55.assembly.stream-3c6cd55",
"SOURCE_GIT_TREE_STATE=clean",
"OS_GIT_COMMIT=3c6cd55",
"SOURCE_DATE_EPOCH=1640014589",
"SOURCE_GIT_COMMIT=3c6cd55e3a5e5332e810274ad27fddf662d4f506",
"SOURCE_GIT_TAG=golang-github-prometheus-prometheus-4.0.0-0.102.0-4305-g3c6cd55e3",
"SOURCE_GIT_URL=https://github.com/openshift/prometheus",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"container=oci"
],
"cwd": "/prometheus",
"capabilities": {
"bounding": [
"CAP_CHOWN",
"CAP_DAC_OVERRIDE",
"CAP_FSETID",
"CAP_FOWNER",
"CAP_NET_BIND_SERVICE",
"CAP_SETPCAP"
],
"effective": [
"CAP_CHOWN",
"CAP_DAC_OVERRIDE",
"CAP_FSETID",
"CAP_FOWNER",
"CAP_NET_BIND_SERVICE",
"CAP_SETPCAP"
],
"permitted": [
"CAP_CHOWN",
"CAP_DAC_OVERRIDE",
"CAP_FSETID",
"CAP_FOWNER",
"CAP_NET_BIND_SERVICE",
"CAP_SETPCAP"
]
},
"oomScoreAdj": 968,
"selinuxLabel": "system_u:system_r:container_t:s0:c21,c0"
},
"root": {
"path": "/var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged"
},
"hostname": "prometheus-k8s-0",
"mounts": [
{
"destination": "/proc",
"type": "proc",
"source": "proc",
"options": [
"nosuid",
"noexec",
"nodev"
]
},
{
"destination": "/dev",
"type": "tmpfs",
"source": "tmpfs",
"options": [
"nosuid",
"strictatime",
"mode=755",
"size=65536k"
]
},
{
"destination": "/dev/pts",
"type": "devpts",
"source": "devpts",
"options": [
"nosuid",
"noexec",
"newinstance",
"ptmxmode=0666",
"mode=0620",
"gid=5"
]
},
{
"destination": "/dev/mqueue",
"type": "mqueue",
"source": "mqueue",
"options": [
"nosuid",
"noexec",
"nodev"
]
},
{
"destination": "/sys",
"type": "sysfs",
"source": "sysfs",
"options": [
"nosuid",
"noexec",
"nodev",
"ro"
]
},
{
"destination": "/sys/fs/cgroup",
"type": "cgroup",
"source": "cgroup",
"options": [
"nosuid",
"noexec",
"nodev",
"relatime",
"ro"
]
},
{
"destination": "/dev/shm",
"type": "bind",
"source": "/run/containers/storage/overlay-containers/fde2304412307a44d51118bcf4d3ef4089fd71e5aa12418ca6716a6395194d7d/userdata/shm",
"options": [
"rw",
"bind"
]
},
{
"destination": "/etc/resolv.conf",
"type": "bind",
"source": "/run/containers/storage/overlay-containers/fde2304412307a44d51118bcf4d3ef4089fd71e5aa12418ca6716a6395194d7d/userdata/resolv.conf",
"options": [
"rw",
"bind",
"nodev",
"nosuid",
"noexec"
]
},
{
"destination": "/etc/hostname",
"type": "bind",
"source": "/run/containers/storage/overlay-containers/fde2304412307a44d51118bcf4d3ef4089fd71e5aa12418ca6716a6395194d7d/userdata/hostname",
"options": [
"rw",
"bind"
]
},
{
"destination": "/run/.containerenv",
"type": "bind",
"source": "/run/containers/storage/overlay-containers/fde2304412307a44d51118bcf4d3ef4089fd71e5aa12418ca6716a6395194d7d/userdata/.containerenv",
"options": [
"rw",
"bind"
]
},
{
"destination": "/prometheus",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volume-subpaths/pvc-b17ae945-02d2-43e2-a2d2-23891203b1f3/prometheus/3",
"options": [
"rw",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/dev/termination-log",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/containers/prometheus/5ab56fc4",
"options": [
"rw",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/etc/hosts",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/etc-hosts",
"options": [
"rw",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/run/secrets",
"type": "bind",
"source": "/run/containers/storage/overlay-containers/27cccaa137e395427c3ddaef728528ddfd3e1d83cd2710a9d15fa1202e93e084/userdata/run/secrets",
"options": [
"bind",
"rprivate",
"bind"
]
},
{
"destination": "/etc/prometheus/config_out",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~empty-dir/config-out",
"options": [
"ro",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/etc/prometheus/certs",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~projected/tls-assets",
"options": [
"ro",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/etc/prometheus/secrets/metrics-client-certs",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-metrics-client-certs",
"options": [
"ro",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/etc/prometheus/secrets/prometheus-k8s-tls",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls",
"options": [
"ro",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/etc/prometheus/secrets/prometheus-k8s-proxy",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-proxy",
"options": [
"ro",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/etc/prometheus/secrets/kube-etcd-client-certs",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-kube-etcd-client-certs",
"options": [
"ro",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/etc/prometheus/secrets/kube-rbac-proxy",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-kube-rbac-proxy",
"options": [
"ro",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/etc/prometheus/secrets/prometheus-k8s-thanos-sidecar-tls",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls",
"options": [
"ro",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/etc/prometheus/configmaps/serving-certs-ca-bundle",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-serving-certs-ca-bundle",
"options": [
"ro",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/etc/prometheus/configmaps/kubelet-serving-ca-bundle",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-kubelet-serving-ca-bundle",
"options": [
"ro",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/etc/prometheus/configmaps/metrics-client-ca",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-metrics-client-ca",
"options": [
"ro",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/etc/prometheus/rules/prometheus-k8s-rulefiles-0",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/prometheus-k8s-rulefiles-0",
"options": [
"ro",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/etc/prometheus/web_config/web-config.yaml",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volume-subpaths/web-config/prometheus/5",
"options": [
"ro",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/var/run/secrets/kubernetes.io/serviceaccount",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~projected/kube-api-access-jfzqj",
"options": [
"ro",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/etc/pki/ca-trust/extracted/pem/",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/prometheus-trusted-ca-bundle",
"options": [
"ro",
"rbind",
"rprivate",
"bind"
]
}
],
"annotations": {
"openshift.io/scc": "nonroot",
"io.kubernetes.cri-o.Name": "k8s_prometheus_prometheus-k8s-0_openshift-monitoring_a4477163-737d-457c-8698-8688fdd00003_0",
"io.kubernetes.cri-o.SandboxName": "k8s_prometheus-k8s-0_openshift-monitoring_a4477163-737d-457c-8698-8688fdd00003_0",
"kubectl.kubernetes.io/default-container": "prometheus",
"io.kubernetes.cri-o.Stdin": "false",
"io.kubernetes.cri-o.ResolvPath": "/run/containers/storage/overlay-containers/fde2304412307a44d51118bcf4d3ef4089fd71e5aa12418ca6716a6395194d7d/userdata/resolv.conf",
"io.kubernetes.cri-o.Volumes": "[{\"container_path\":\"/prometheus\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volume-subpaths/pvc-b17ae945-02d2-43e2-a2d2-23891203b1f3/prometheus/3\",\"readonly\":false,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/dev/termination-log\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/containers/prometheus/5ab56fc4\",\"readonly\":false,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/hosts\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/etc-hosts\",\"readonly\":false,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/config_out\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~empty-dir/config-out\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/certs\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~projected/tls-assets\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/secrets/prometheus-k8s-thanos-sidecar-tls\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/secrets/kube-etcd-client-certs\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-kube-etcd-client-certs\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/secrets/prometheus-k8s-tls\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/secrets/prometheus-k8s-proxy\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-proxy\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/web_config/web-config.yaml\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volume-subpaths/web-config/prometheus/5\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/secrets/kube-rbac-proxy\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-kube-rbac-proxy\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/secrets/metrics-client-certs\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-metrics-client-certs\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/configmaps/serving-certs-ca-bundle\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-serving-certs-ca-bundle\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/configmaps/kubelet-serving-ca-bundle\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-kubelet-serving-ca-bundle\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/configmaps/metrics-client-ca\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-metrics-client-ca\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/prometheus-k8s-rulefiles-0\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/var/run/secrets/kubernetes.io/serviceaccount\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~projected/kube-api-access-jfzqj\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/pki/ca-trust/extracted/pem/\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/prometheus-trusted-ca-bundle\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true}]",
"io.kubernetes.container.hash": "7ba9b53",
"io.kubernetes.pod.name": "prometheus-k8s-0",
"io.kubernetes.cri-o.IP.0": "10.131.2.7",
"io.kubernetes.cri-o.SeccompProfilePath": "",
"io.kubernetes.cri-o.Metadata": "{\"name\":\"prometheus\"}",
"io.kubernetes.cri-o.Labels": "{\"io.kubernetes.container.name\":\"prometheus\",\"io.kubernetes.pod.name\":\"prometheus-k8s-0\",\"io.kubernetes.pod.namespace\":\"openshift-monitoring\",\"io.kubernetes.pod.uid\":\"a4477163-737d-457c-8698-8688fdd00003\"}",
"io.kubernetes.pod.uid": "a4477163-737d-457c-8698-8688fdd00003",
"kubernetes.io/config.seen": "2022-09-27T08:46:47.929904784Z",
"io.kubernetes.cri-o.ImageName": "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00d156818e2e4264c3268c9ae41de9b561e2811363b64515ca64865c992ec8ea",
"io.kubernetes.cri-o.ContainerType": "container",
"io.kubernetes.cri-o.StdinOnce": "false",
"io.kubernetes.cri-o.Created": "2022-09-27T08:46:55.0148206Z",
"io.kubernetes.container.terminationMessagePolicy": "FallbackToLogsOnError",
"kubernetes.io/config.source": "api",
"io.kubernetes.cri-o.ImageRef": "d3017e38d93fb39bcad5ae9ee8521f5829c774c389ae7e65e44a00388d67cf1e",
"io.kubernetes.container.restartCount": "0",
"org.systemd.property.DefaultDependencies": "true",
"io.kubernetes.container.name": "prometheus",
"io.kubernetes.cri-o.LogPath": "/var/log/pods/openshift-monitoring_prometheus-k8s-0_a4477163-737d-457c-8698-8688fdd00003/prometheus/0.log",
"io.kubernetes.cri-o.TTY": "false",
"io.container.manager": "cri-o",
"io.kubernetes.cri-o.MountPoint": "/var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged",
"io.kubernetes.cri-o.Annotations": "{\"io.kubernetes.container.hash\":\"7ba9b53\",\"io.kubernetes.container.restartCount\":\"0\",\"io.kubernetes.container.terminationMessagePath\":\"/dev/termination-log\",\"io.kubernetes.container.terminationMessagePolicy\":\"FallbackToLogsOnError\",\"io.kubernetes.pod.terminationGracePeriod\":\"600\"}",
"io.kubernetes.pod.terminationGracePeriod": "600",
"io.kubernetes.pod.namespace": "openshift-monitoring",
"io.kubernetes.cri-o.SandboxID": "fde2304412307a44d51118bcf4d3ef4089fd71e5aa12418ca6716a6395194d7d",
"io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
"io.kubernetes.cri-o.Image": "d3017e38d93fb39bcad5ae9ee8521f5829c774c389ae7e65e44a00388d67cf1e",
"io.kubernetes.cri-o.ContainerID": "27cccaa137e395427c3ddaef728528ddfd3e1d83cd2710a9d15fa1202e93e084",
"org.systemd.property.TimeoutStopUSec": "uint64 600000000",
"org.systemd.property.CollectMode": "'inactive-or-failed'",
"org.systemd.property.After": "['crio.service']"
},
"linux": {
"resources": {
"devices": [
{
"allow": false,
"access": "rwm"
}
],
"cpu": {
"shares": 71,
"quota": 0,
"period": 100000
},
"pids": {
"limit": 4096
},
"hugepageLimits": [
{
"pageSize": "2MB",
"limit": 0
},
{
"pageSize": "1GB",
"limit": 0
}
]
},
"cgroupsPath": "kubepods-burstable-poda4477163_737d_457c_8698_8688fdd00003.slice:crio:27cccaa137e395427c3ddaef728528ddfd3e1d83cd2710a9d15fa1202e93e084",
"namespaces": [
{
"type": "pid"
},
{
"type": "network",
"path": "/var/run/netns/5b03e11b-6a0a-4027-b037-cb8aec0048c7"
},
{
"type": "ipc",
"path": "/var/run/ipcns/5b03e11b-6a0a-4027-b037-cb8aec0048c7"
},
{
"type": "uts",
"path": "/var/run/utsns/5b03e11b-6a0a-4027-b037-cb8aec0048c7"
},
{
"type": "mount"
}
],
"maskedPaths": [
"/proc/acpi",
"/proc/kcore",
"/proc/keys",
"/proc/latency_stats",
"/proc/timer_list",
"/proc/timer_stats",
"/proc/sched_debug",
"/proc/scsi",
"/sys/firmware"
],
"readonlyPaths": [
"/proc/asound",
"/proc/bus",
"/proc/fs",
"/proc/irq",
"/proc/sys",
"/proc/sysrq-trigger"
],
"mountLabel": "system_u:object_r:container_file_t:s0:c21,c0"
}
},
"privileged": false
}
}
sh-4.4#
想要确认Worker节点上的目录路径，所以用Path来grep。
sh-4.4# crictl inspect 27cccaa137e39 | grep Path
"io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
"containerPath": "/prometheus",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volume-subpaths/pvc-b17ae945-02d2-43e2-a2d2-23891203b1f3/prometheus/3",
"containerPath": "/dev/termination-log",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/containers/prometheus/5ab56fc4",
"containerPath": "/etc/hosts",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/etc-hosts",
"containerPath": "/etc/prometheus/config_out",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~empty-dir/config-out",
"containerPath": "/etc/prometheus/certs",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~projected/tls-assets",
"containerPath": "/etc/prometheus/secrets/prometheus-k8s-thanos-sidecar-tls",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls",
"containerPath": "/etc/prometheus/secrets/kube-etcd-client-certs",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-kube-etcd-client-certs",
"containerPath": "/etc/prometheus/secrets/prometheus-k8s-tls",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls",
"containerPath": "/etc/prometheus/secrets/prometheus-k8s-proxy",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-proxy",
"containerPath": "/etc/prometheus/web_config/web-config.yaml",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volume-subpaths/web-config/prometheus/5",
"containerPath": "/etc/prometheus/secrets/kube-rbac-proxy",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-kube-rbac-proxy",
"containerPath": "/etc/prometheus/secrets/metrics-client-certs",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-metrics-client-certs",
"containerPath": "/etc/prometheus/configmaps/serving-certs-ca-bundle",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-serving-certs-ca-bundle",
"containerPath": "/etc/prometheus/configmaps/kubelet-serving-ca-bundle",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-kubelet-serving-ca-bundle",
"containerPath": "/etc/prometheus/configmaps/metrics-client-ca",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-metrics-client-ca",
"containerPath": "/etc/prometheus/rules/prometheus-k8s-rulefiles-0",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/prometheus-k8s-rulefiles-0",
"containerPath": "/var/run/secrets/kubernetes.io/serviceaccount",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~projected/kube-api-access-jfzqj",
"containerPath": "/etc/pki/ca-trust/extracted/pem/",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/prometheus-trusted-ca-bundle",
"logPath": "/var/log/pods/openshift-monitoring_prometheus-k8s-0_a4477163-737d-457c-8698-8688fdd00003/prometheus/0.log"
"io.kubernetes.cri-o.Annotations": "{\"io.kubernetes.container.hash\":\"7ba9b53\",\"io.kubernetes.container.restartCount\":\"0\",\"io.kubernetes.container.terminationMessagePath\":\"/dev/termination-log\",\"io.kubernetes.container.terminationMessagePolicy\":\"FallbackToLogsOnError\",\"io.kubernetes.pod.terminationGracePeriod\":\"600\"}",
"io.kubernetes.cri-o.LogPath": "/var/log/pods/openshift-monitoring_prometheus-k8s-0_a4477163-737d-457c-8698-8688fdd00003/prometheus/0.log",
"io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
"io.kubernetes.cri-o.ResolvPath": "/run/containers/storage/overlay-containers/fde2304412307a44d51118bcf4d3ef4089fd71e5aa12418ca6716a6395194d7d/userdata/resolv.conf",
"io.kubernetes.cri-o.SeccompProfilePath": "",
"cgroupsPath": "kubepods-burstable-poda4477163_737d_457c_8698_8688fdd00003.slice:crio:27cccaa137e395427c3ddaef728528ddfd3e1d83cd2710a9d15fa1202e93e084",
"maskedPaths": [
"readonlyPaths": [
sh-4.4#
可以看出，主机（Worker节点）正在使用/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/目录。
"root": {
"path": "/var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged"
},
在确认其他部分后，我们可以得知他们正在使用/var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged目录。
确认磁盘使用量
确认Infra/Worker节点上该目录的大小。
sh-4.4# du -sh /var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/*
0 /var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/containers
4.0K /var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/etc-hosts
4.0K /var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/plugins
14G /var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volume-subpaths
1.1M /var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes
sh-4.4#
→ 使用約14GB的空間。
sh-4.4# du -sh /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/*
0 /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/bin
0 /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/boot
0 /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/dev
2.3M /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/etc
0 /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/home
0 /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/lib
0 /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/lib64
0 /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/lost+found
0 /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/media
0 /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/mnt
0 /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/opt
0 /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/proc
0 /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/prometheus
996K /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/root
0 /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/run
0 /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/sbin
0 /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/srv
0 /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/sys
8.0K /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/tmp
370M /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/usr
24M /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/var
sh-4.4#
→ 需要大约370MB的空间。
整体而言，可以确认Prometheus 1pod使用了大约15GB左右的存储空间。
* 参考：这个ROSA环境只是一个简单的验证环境，只有几个简单的应用在运行。Prometheus的保留设置等配置保持默认状态。
总结
为了确认Pod的磁盘使用量，需要获取Pod使用Infra/Worker节点的目录信息，因此使用crictl命令进行确认。
虽然可能有其他更智能的方法来确认Pod的磁盘使用量，但是未能找到，所以就暂时使用本文所述的方法进行确认。
请参照以下内容，需中文为母语提供一种选择：
参考内容。
在本文中，我们省略了关于存储使用的解释，但是有关OpenShift的Pod使用的存储的详细信息，请参考以下手册。理解暂时性存储。
请问有附赠品吗？
ROSA的基础设施/工作节点默认使用300GB的EBS。
目前已经使用了以下数量的磁盘，并且如果将来其他Pod大量消耗磁盘，则应考虑使用永久卷（PV）。
sh-4.4# df -h /var/lib
Filesystem Size Used Avail Use% Mounted on
/dev/nvme0n1p4 300G 19G 281G 7% /var
