K3s pods stuck on Ubuntu upgrade

After upgrading to Ubuntu 22.04 LTS from 20.04 LTS, I saw that many of the K3s pods on my home server were stuck in various error states. Below is the sample K3s state.

$ kubectl get pods -A
NAMESPACE      NAME                                       READY   STATUS                       RESTARTS           AGE
kube-system    local-path-provisioner-6c79684f77-ll4jr    0/1     Unknown                      1                  42d
kube-system    svclb-traefik-8bflz                        0/2     Unknown                      0                  16d
default        netdata-child-8f7lm                        0/2     ContainerStatusUnknown       6 (5h19m ago)      16d
kube-system    csi-smb-node-jjkss                         0/3     Unknown                      1073 (5h19m ago)   16d
cert-manager   cert-manager-b4d6fd99b-7c8f4               0/1     ExitCode:0                   6                  16d
kube-system    coredns-96cc4f57d-6rg6w                    0/1     ExitCode:0                   4                  42d
kube-system    metrics-server-7cd5fcb6b7-tr2jx            0/1     Completed                    1                  42d
cert-manager   cert-manager-cainjector-74bfccdfdf-v2c4c   0/1     ExitCode:0                   1149               7d9h
cert-manager   cert-manager-webhook-65b766b5f8-vfrgw      0/1     ExitCode:0                   146                7d9h
kube-system    nvidia-device-plugin-daemonset-wlmw9       0/1     Unknown                      1                  42d
kube-system    csi-smb-controller-55d7dfc465-vw2z5        0/3     ExitCode:0                   19                 42d
kube-system    traefik-58b759688b-srwhc                   0/1     ExitCode:0                   4                  42d
mariadb        mariadb-deployment-5b65c54c7d-6txnn        0/1     Unknown                      1                  41d
jellyfin       jellyfin-6b779d688f-lm2wz                  0/1     CreateContainerConfigError   1 (5h20m ago)      42d

On describing the pod, I saw the below error –

Events:
  Type     Reason                  Age                     From     Message
  ----     ------                  ----                    ----     -------
  Normal   SandboxChanged          4h42m (x91 over 5h8m)   kubelet  Pod sandbox changed, it will be killed and re-created.
  Warning  FailedCreatePodSandBox  4h7m                    kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7e7634da0b833
b4686cacdfb156a554374079bb4f856fe9967df428ae951b4b0": netplugin failed with no error message: signal: killed
  Warning  FailedMount             4h3m (x2 over 4h3m)     kubelet  MountVolume.SetUp failed for volume "kube-api-access-4d58m" : failed to sync configmap cache: timed out waiting f
or the condition
  Warning  FailedCreatePodSandBox  3h58m (x142 over 5h7m)  kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: cgroups: cgroup
mountpoint does not exist: unknown
  Normal   SandboxChanged          5s (x7 over 2m59s)      kubelet  Pod sandbox changed, it will be killed and re-created.
  Warning  FailedCreatePodSandBox  3s (x7 over 112s)       kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: cgroups: cgroup
mountpoint does not exist: unknown

On searching over the web, I saw other people had also encountered this error. Below is the solution that was proposed which worked.

$ sudo mkdir /sys/fs/cgroup/systemd
$ sudo mount -t cgroup -o none,name=systemd cgroup /sys/fs/cgroup/systemd

At this moment, I am uncertain why I have started this seeing this error. The only thing that changed was the server was upgraded from Ubuntu 20.04 to 22.04.

For now, this command needs to be run at every reboot. I might add this in the startup script to fix the issue until I find a better solution or I can determine the root cause.