http_request_rateおよび/またはpackets_per_secondのポッド統計を監視する目的でPrometheusを使用しようとしています。そうするために、私はPrometheus Adapterを使用することを計画していました。私が読んだことから、Prometheus Operatorを使用する必要があります。
Helm 安定チャートから Prometheus Operator をインストールする際に問題が発生しました。インストール コマンド「helm install prom stable/prometheus-operator」を実行すると、次の警告メッセージが 6 回表示されます
$ manifest_sorter.go:192 info: skipping unknown hook: "crd-install".
インストールが続行され、ポッドがデプロイされますが、prometheus-node-exporter ポッドのステータスは CrashLoopBackOff になります。
ポッドを説明するときのメッセージは「Back-off restarting failed container」であるため、詳細な理由はわかりません。
バージョン 1.7.2 で Minikube を実行しています。
バージョン 3.1.1 で Helm を実行しています。
>>>更新<<<
問題のある Pod の説明の出力
> $ kubectl describe pod prom-oper-prometheus-node-exporter-2m6vm -n default
>
> Name: prom-oper-prometheus-node-exporter-2m6vm Namespace:
> default Priority: 0 Node: max-ubuntu/10.2.40.198 Start
> Time: Wed, 04 Mar 2020 18:06:44 +0000 Labels:
> app=prometheus-node-exporter
> chart=prometheus-node-exporter-1.8.2
> controller-revision-hash=68695df4c5
> heritage=Helm
> jobLabel=node-exporter
> pod-template-generation=1
> release=prom-oper Annotations: <none> Status: Running IP: 10.2.40.198 IPs: IP: 10.2.40.198
> Controlled By: DaemonSet/prom-oper-prometheus-node-exporter
> Containers: node-exporter:
> Container ID: docker://50b2398f72a0269672c4ac73bbd1b67f49732362b4838e16cd10e3a5247fdbfe
> Image: quay.io/prometheus/node-exporter:v0.18.1
> Image ID: docker-pullable://quay.io/prometheus/node-exporter@sha256:a2f29256e53cc3e0b64d7a472512600b2e9410347d53cdc85b49f659c17e02ee
> Port: 9100/TCP
> Host Port: 9100/TCP
> Args:
> --path.procfs=/host/proc
> --path.sysfs=/host/sys
> --web.listen-address=0.0.0.0:9100
> --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/)
> --collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$
> State: Waiting
> Reason: CrashLoopBackOff
> Last State: Terminated
> Reason: Error
> Exit Code: 1
> Started: Wed, 04 Mar 2020 18:10:10 +0000
> Finished: Wed, 04 Mar 2020 18:10:10 +0000
> Ready: False
> Restart Count: 5
> Liveness: http-get http://:9100/ delay=0s timeout=1s period=10s #success=1 #failure=3
> Readiness: http-get http://:9100/ delay=0s timeout=1s period=10s #success=1 #failure=3
> Environment: <none>
> Mounts:
> /host/proc from proc (ro)
> /host/sys from sys (ro)
> /var/run/secrets/kubernetes.io/serviceaccount from prom-oper-prometheus-node-exporter-token-n9dj9 (ro) Conditions: Type
> Status Initialized True Ready False
> ContainersReady False PodScheduled True Volumes: proc:
> Type: HostPath (bare host directory volume)
> Path: /proc
> HostPathType: sys:
> Type: HostPath (bare host directory volume)
> Path: /sys
> HostPathType: prom-oper-prometheus-node-exporter-token-n9dj9:
> Type: Secret (a volume populated by a Secret)
> SecretName: prom-oper-prometheus-node-exporter-token-n9dj9
> Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: :NoSchedule
> node.kubernetes.io/disk-pressure:NoSchedule
> node.kubernetes.io/memory-pressure:NoSchedule
> node.kubernetes.io/network-unavailable:NoSchedule
> node.kubernetes.io/not-ready:NoExecute
> node.kubernetes.io/pid-pressure:NoSchedule
> node.kubernetes.io/unreachable:NoExecute
> node.kubernetes.io/unschedulable:NoSchedule Events: Type Reason Age From
> Message ---- ------ ---- ----
> ------- Normal Scheduled 5m26s default-scheduler Successfully assigned default/prom-oper-prometheus-node-exporter-2m6vm
> to max-ubuntu Normal Started 4m28s (x4 over 5m22s) kubelet,
> max-ubuntu Started container node-exporter Normal Pulled
> 3m35s (x5 over 5m24s) kubelet, max-ubuntu Container image
> "quay.io/prometheus/node-exporter:v0.18.1" already present on machine
> Normal Created 3m35s (x5 over 5m24s) kubelet, max-ubuntu
> Created container node-exporter Warning BackOff 13s (x30 over
> 5m18s) kubelet, max-ubuntu Back-off restarting failed container
問題のある Pod ログの出力
> $ kubectl logs prom-oper-prometheus-node-exporter-2m6vm -n default
> time="2020-03-04T18:18:02Z" level=info msg="Starting node_exporter
> (version=0.18.1, branch=HEAD,
> revision=3db77732e925c08f675d7404a8c46466b2ece83e)"
> source="node_exporter.go:156" time="2020-03-04T18:18:02Z" level=info
> msg="Build context (go=go1.12.5, user=root@b50852a1acba,
> date=20190604-16:41:18)" source="node_exporter.go:157"
> time="2020-03-04T18:18:02Z" level=info msg="Enabled collectors:"
> source="node_exporter.go:97" time="2020-03-04T18:18:02Z" level=info
> msg=" - arp" source="node_exporter.go:104" time="2020-03-04T18:18:02Z"
> level=info msg=" - bcache" source="node_exporter.go:104"
> time="2020-03-04T18:18:02Z" level=info msg=" - bonding"
> source="node_exporter.go:104" time="2020-03-04T18:18:02Z" level=info
> msg=" - conntrack" source="node_exporter.go:104"
> time="2020-03-04T18:18:02Z" level=info msg=" - cpu"
> source="node_exporter.go:104" time="2020-03-04T18:18:02Z" level=info
> msg=" - cpufreq" source="node_exporter.go:104"
> time="2020-03-04T18:18:02Z" level=info msg=" - diskstats"
> source="node_exporter.go:104" time="2020-03-04T18:18:02Z" level=info
> msg=" - edac" source="node_exporter.go:104"
> time="2020-03-04T18:18:02Z" level=info msg=" - entropy"
> source="node_exporter.go:104" time="2020-03-04T18:18:02Z" level=info
> msg=" - filefd" source="node_exporter.go:104"
> time="2020-03-04T18:18:02Z" level=info msg=" - filesystem"
> source="node_exporter.go:104" time="2020-03-04T18:18:02Z" level=info
> msg=" - hwmon" source="node_exporter.go:104"
> time="2020-03-04T18:18:02Z" level=info msg=" - infiniband"
> source="node_exporter.go:104" time="2020-03-04T18:18:02Z" level=info
> msg=" - ipvs" source="node_exporter.go:104"
> time="2020-03-04T18:18:02Z" level=info msg=" - loadavg"
> source="node_exporter.go:104" time="2020-03-04T18:18:02Z" level=info
> msg=" - mdadm" source="node_exporter.go:104"
> time="2020-03-04T18:18:02Z" level=info msg=" - meminfo"
> source="node_exporter.go:104" time="2020-03-04T18:18:02Z" level=info
> msg=" - netclass" source="node_exporter.go:104"
> time="2020-03-04T18:18:02Z" level=info msg=" - netdev"
> source="node_exporter.go:104" time="2020-03-04T18:18:02Z" level=info
> msg=" - netstat" source="node_exporter.go:104"
> time="2020-03-04T18:18:02Z" level=info msg=" - nfs"
> source="node_exporter.go:104" time="2020-03-04T18:18:02Z" level=info
> msg=" - nfsd" source="node_exporter.go:104"
> time="2020-03-04T18:18:02Z" level=info msg=" - pressure"
> source="node_exporter.go:104" time="2020-03-04T18:18:02Z" level=info
> msg=" - sockstat" source="node_exporter.go:104"
> time="2020-03-04T18:18:02Z" level=info msg=" - stat"
> source="node_exporter.go:104" time="2020-03-04T18:18:02Z" level=info
> msg=" - textfile" source="node_exporter.go:104"
> time="2020-03-04T18:18:02Z" level=info msg=" - time"
> source="node_exporter.go:104" time="2020-03-04T18:18:02Z" level=info
> msg=" - timex" source="node_exporter.go:104"
> time="2020-03-04T18:18:02Z" level=info msg=" - uname"
> source="node_exporter.go:104" time="2020-03-04T18:18:02Z" level=info
> msg=" - vmstat" source="node_exporter.go:104"
> time="2020-03-04T18:18:02Z" level=info msg=" - xfs"
> source="node_exporter.go:104" time="2020-03-04T18:18:02Z" level=info
> msg=" - zfs" source="node_exporter.go:104" time="2020-03-04T18:18:02Z"
> level=info msg="Listening on 0.0.0.0:9100"
> source="node_exporter.go:170" time="2020-03-04T18:18:02Z" level=fatal
> msg="listen tcp 0.0.0.0:9100: bind: address already in use"
> source="node_exporter.go:172"