How to get kubelet healthz status?

# curl -k https://127.0.0.1:10250/healthz -H "Authorization: Bearer OX8__bQr1nDu-ebqQJJGjIuh"
[+]ping ok
[+]log ok
[-]syncloop failed: reason withheld
healthz check failed

How to get the Bearer token?

# cat /etc/kubernetes/admin.conf |grep token
    token: "ZALlL_BOTtZeZRfsQSwkRI8E"

How to get apiserver metrics?

curl -k  -H "Authorization: Bearer ZALlL_BOTtZeZRfsQSwkRI8E" https://localhost:8443/metrics

How to get debug or goroutine info?

curl -H "Authorization: Bearer OX8__bQr1nDu-ebqQJJGjIuh" -k https://127.0.0.1:10250/debug


curl -H "Authorization: Bearer OX8__bQr1nDu-ebqQJJGjIuh" -k https://127.0.0.1:10250/debug/pprof


curl -H "Authorization: Bearer OX8__bQr1nDu-ebqQJJGjIuh" -k https://127.0.0.1:10250/debug/pprof/goroutine -o goroutine


curl -H "Authorization: Bearer OX8__bQr1nDu-ebqQJJGjIuh" -k https://127.0.0.1:10250/debug/pprof/goroutine?debug=2 -o goroutine-2


### stack trace sample:


goroutine 251 [syscall, 939 minutes]:
syscall.Syscall6(0x106, 0xffffffffffffff9c, 0xc001357440, 0xc0015a2378, 0x0, 0x0, 0x0, 0xc0017d3320, 0xc0017d3338, 0x411b98)
        /usr/local/go/src/syscall/asm_linux_amd64.s:41 +0x5
syscall.fstatat(0xffffffffffffff9c, 0xc001357320, 0x5e, 0xc0015a2378, 0x0, 0xc001357129, 0x8)
        /usr/local/go/src/syscall/zsyscall_linux_amd64.go:1480 +0xc8
syscall.Stat(...)
        /usr/local/go/src/syscall/syscall_linux_amd64.go:65
os.statNolog(0xc001357320, 0x5e, 0x3cf2e80, 0x0, 0x6, 0x7f154449dec8)
        /usr/local/go/src/os/stat_unix.go:31 +0x6e
os.Stat(0xc001357320, 0x5e, 0xc000d56b68, 0x3, 0xc000facb40, 0x48)
        /usr/local/go/src/os/stat.go:13 +0x4d
k8s.io/kubernetes/vendor/k8s.io/mount-utils.(*Mounter).IsLikelyNotMountPoint(0xc000becfc0, 0xc001357320, 0x5e, 0xc001302700, 0x1, 0x1)
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/mount-utils/mount_linux.go:284 +0x3c
k8s.io/kubernetes/pkg/kubelet.(*Kubelet).getMountedVolumePathListFromDisk(0xc0000c5500, 0xc00097b798, 0x24, 0xc001449e90, 0xc0017d35b0, 0xe73afb5b, 0xd6f7addb2f85b69d, 0x3929dc9)
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_getters.go:360 +0x110
k8s.io/kubernetes/pkg/kubelet.(*Kubelet).podVolumesExist(0xc0000c5500, 0xc00097b798, 0x24, 0x0)
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_volumes.go:64 +0xb4
k8s.io/kubernetes/pkg/kubelet.(*Kubelet).cleanupOrphanedPodCgroups(0xc0000c5500, 0x4fe5880, 0xc0017b8e00, 0xc001603920, 0xc00117f600, 0x5, 0x8)
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_pods.go:1955 +0x3cd
k8s.io/kubernetes/pkg/kubelet.(*Kubelet).HandlePodCleanups(0xc0000c5500, 0x0, 0x0)
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_pods.go:1130 +0x659
k8s.io/kubernetes/pkg/kubelet.(*Kubelet).syncLoopIteration(0xc0000c5500, 0xc000c1e900, 0x4fd5fe0, 0xc0000c5500, 0xc000291260, 0xc0002912c0, 0xc001006a20, 0x1)
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:2000 +0x18e2
k8s.io/kubernetes/pkg/kubelet.(*Kubelet).syncLoop(0xc0000c5500, 0xc000c1e900, 0x4fd5fe0, 0xc0000c5500)
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1867 +0x3f2
k8s.io/kubernetes/pkg/kubelet.(*Kubelet).Run(0xc0000c5500, 0xc000c1e900)
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1447 +0x2a5
created by k8s.io/kubernetes/cmd/kubelet/app.startKubelet
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:1183 +0x5f

How to check system info?

# dmesg |grep "not responding" | tail
[109934.556157] nfs: server 10.150.200.91 not responding, timed out
[109950.940236] nfs: server 10.150.200.91 not responding, timed out

original refer