Skip to content

[kube-prometheus-stack] Warning messages in linux system log in k8s versions 1.34, 1.35 and 1.36 #7115

Description

@KostadiE

Describe the bug a clear and concise description of what the bug is.

Warning messages in ubuntu system log on each 10-20 seconds.

What's your helm version?

version.BuildInfo{Version:"v3.20.2", GitCommit:"8fb76d6ab555577e98e23b7500009537a471feee", GitTreeState:"clean", GoVersion:"go1.25.0"}

What's your kubectl version?

Client Version: v1.36.2 Kustomize Version: v5.8.1 Server Version: v1.36.2

Which chart?

kube-prometheus-stack

What's the chart version?

86.2.3

What happened?

After update of microk8s from 1.33 to 1.36 we noticed warning messages in system log approximately on 10 seconds.
On servers without prometheus these messages do not appear. All applications on cluster are Running. Cluster is healthy, Tests that we did:

  1. stop all 4 prometheus pods: we have prome-operator; state-metrics; node-exporter; prome-prometheus
    In result - no warnings in the log
  2. start node-exporter: no messages
  3. start state-metrics: no messages
  4. start prome-prometheus: the messages appear again.
    The warning messages are like these:

Jul 15 10:33:31 viecon001 microk8s.daemon-kubelite[26583]: W0715 10:33:31.319093 26583 logging.go:55] [core] [Channel #3683 SubChannel #3684] grpc: addrConn.createTransport failed to connect to {Addr: "unix:///var/snap/microk8s/9092/var/kubernetes/backend/kine.sock:12379", ServerName: "kine.sock:12379", }. Err: connection error: desc = "transport: failed to write client preface: write unix @->/var/snap/microk8s/9092/var/kubernetes/backend/kine.sock:12379: use of closed network connection"
Jul 15 10:34:01 viecon001 microk8s.daemon-kubelite[26583]: W0715 10:34:01.319138 26583 logging.go:55] [core] [Channel #3694 SubChannel #3695] grpc: addrConn.createTransport failed to connect to {Addr: "unix:///var/snap/microk8s/9092/var/kubernetes/backend/kine.sock:12379", ServerName: "kine.sock:12379", }. Err: connection error: desc = "transport: Error while dialing: dial unix /var/snap/microk8s/9092/var/kubernetes/backend/kine.sock:12379: operation was canceled"
Jul 15 10:34:04 viecon001 microk8s.daemon-kubelite[26583]: W0715 10:34:04.268519 26583 logging.go:55] [core] [Channel #3701 SubChannel #3702] grpc: addrConn.createTransport failed to connect to {Addr: "unix:///var/snap/microk8s/9092/var/kubernetes/backend/kine.sock:12379", ServerName: "kine.sock:12379", }. Err: connection error: desc = "error reading server preface: read unix @->/var/snap/microk8s/9092/var/kubernetes/backend/kine.sock:12379: use of closed network connection"
Jul 15 10:34:31 viecon001 microk8s.daemon-kubelite[26583]: W0715 10:34:31.333771 26583 logging.go:55] [core] [Channel #3712 SubChannel #3713] grpc: addrConn.createTransport failed to connect to {Addr: "unix:///var/snap/microk8s/9092/var/kubernetes/backend/kine.sock:12379", ServerName: "kine.sock:12379", }. Err: connection error: desc = "transport: Error while dialing: dial unix /var/snap/microk8s/9092/var/kubernetes/backend/kine.sock:12379: operation was canceled"

What you expected to happen?

To know the root cause of the problem and to have a fix or advice how to stop the messages.

How to reproduce it?

Install microk8s v.1.36/stable, install prometheus. Use this repo: https://prometheus-community.github.io/helm-charts and this chart: prometheus-community/kube-prometheus-stack

Enter the changed values of values.yaml?

No response

Enter the command that you execute and failing/misfunctioning.

microk8s helm3 upgrade --install kube-prom-stack prometheus-community/kube-prometheus-stack
--create-namespace --namespace monitoring -f /path/to/values/prometheus.yaml
--version 86.2.3

Anything else we need to know?

It happens on ubuntu 22.04 and ubuntu 24.04 LTS.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingchart/kube-prometheus-stackIssues related to the kube-prometheus-stack Helm Chart

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions