Community Articles

Find and share helpful community-sourced technical articles.
Cloudera Employee

This article will cover the following:

 

  • The purpose of this article
  • Problem Statement
  • Troubleshooting
    • Step-1: Use the kubectl describe pod command to view the details of the Pod of the Session.
    • Step-2: Use kubectl describe node to view the status of all the nodes.
    • Step-3: Verify the kubelet process argument.
    • Step-4: Check the nodefs and imagefs recognized by kubelet.
  • Summary

The purpose of this article

This article records the troubleshooting about CDSW unable to start the Session. The reason for the problem involved is that the remaining capacity of nodels and imagefs used by Kubelet has reached the threshold of evicting the node, resulting in the node being labeled as NoSchedule taint by Kubernetes.

Problem Statement

One of my customers reported that they found their CDSW failed to start sessions. The following is a screenshot of the failure to start a session on CDSW.

cdsw-session_failure-no_available_node.jpeg

Troubleshooting

    1. Use the Kubectl describe pod command to view the details of the Pod of the session:
      Every session created by CDSW is a Pod, and the name of the Pod is a random string. This Pod will be placed in a NameSpace named "default-user-{number}". The following figure is an example of the random Pod name generated when the session is created.cdsw-session-pod_name.pngNow, we can use the following Kubectl commands to navigate to the Pod and look into the detail of this Pod. First, you have to use root user to log in to CDSW Master node.
      kubectl get pods --all-namespaces | sed -rn '1p;/74n46954jqzve46c/Ip'

      Sample output:

      NAMESPACE        NAME               READY   STATUS      RESTARTS   AGE
      default-user-1   74n46954jqzve46c   3/3     Running     0          58m
      
      kubectl --namespace=default-user-1 describe pod 74n46954jqzve46c
      Sample output:
      Name:         74n46954jqzve46c
      Namespace:    default-user-1
      Priority:     0
      Node:         host-10-17-103-219/10.17.103.219
      Start Time:   Wed, 16 Jun 2021 16:11:58 +0000
      Labels:       ds-role=session
                    ds-runtime=74n46954jqzve46c
                    livelog-published-container=engine
      Annotations:  cluster-autoscaler.kubernetes.io/safe-to-evict: false
                    seccomp.security.alpha.kubernetes.io/allowedProfileNames: runtime/default
                    seccomp.security.alpha.kubernetes.io/defaultProfileName: runtime/default
      Status:       Failed
      Reason:       Evicted
      Message:      Pod The node was low on resource: [DiskPressure].
      ...
       Please pay attention to the status, reason, and message above.
    2. Use Kubectl describe node to view the status of all the nodes.
      At this time, if we use kubectl describe node to view the details of all nodes, we will find a situation similar to the following screenshot:cdsw-node-taint-disk-pressure(1).jpg
    3. Verify the Kubelet process argument:
      ps aux | sed -rn '1p;/kubelet/Ip' | grep -v sed

      Sample output:

      root     20348  8.2  0.5 2431868 95304 ?       Sl   01:27  47:58 /opt/cloudera/parcels/CDSW-1.9.1.p1.10118148/kubernetes/bin/kubelet --kubeconfig=/etc/kubernetes/kubelet.conf --config=/etc/cdsw/scratch/kubelet.k8s.conf --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --feature-gates=SupportPodPidsLimit=false --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt --read-only-port=0 --fail-swap-on=false --network-plugin=cni --cni-conf-dir=/opt/cloudera/parcels/CDSW-1.9.1.p1.10118148/cni/net.d --cni-bin-dir=/opt/cloudera/parcels/CDSW-1.9.1.p1.10118148/cni/bin --cluster-dns=100.77.0.10 --cluster-domain=cluster.local --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice --kube-reserved=cpu=1,memory=2Gi --system-reserved=cpu=500m,memory=1Gi --eviction-hard=memory.available<500Mi --eviction-soft=imagefs.available<20% --eviction-soft-grace-period=imagefs.available=24h --v=2 --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

      Notice these arguments:

      • --eviction-hard=memory.available<500Mi
      • --eviction-soft=imagefs.available<20%
      • --eviction-soft-grace-period=imagefs.available=24h

      Obviously, the reason why we are now seeing Pod being evicted is because all Nodes are marked with taints. Regarding this taint, the prompt we got was "Disk Pressure". This must be due to imagefs.available<20% then.

      So what is imagefs.available?

      We can find relevant explanations on the Kubernetes official website.

      Node Condition Eviction Signal Description
      DiskPressure nodefs.available, nodefs.inodesFree, imagefs.available, or imagefs.inodesFree Available disk space and inodes on either the node's root filesystem or image filesystem have satisfied an eviction threshold
    4. Check the nodefs and imagefs recognized by Kubelet.
      So, in simple terms, we finally need to confirm the remaining capacity of nodefs and imagefs used by kubelet.
      Similarly, we can find related explanations in the official Kubernetes documentation.

The kubelet supports the following filesystem partitions:

  1. nodefs: The node's main filesystem, used for local disk volumes, emptyDir, log storage, and more. For example, nodefs contains /var/lib/kubelet/.
  2. imagefs: An optional filesystem that container runtimes use to store container images and container writable layers.

In this case, the container runtime we use is Docker, so we can get the usage of imagefs through the docker info command.

docker info | grep -i "data space"
# Data Space Used: 313.3 GB
# Data Space Total: 380 GB
# Data Space Available: 66.65 GB
# Metadata Space Used: 132.6 MB
# Metadata Space Total: 3.997 GB
# Metadata Space Available: 3.865 GB

In the example above, we have a docker block device of 380 GB and because we’ve less than 67GB (18% of 380GB) Available, after a grace period of 24hrs, this node will be marked as NodeUnderDiskPressure().

Summary

The Pod that we encountered this time cannot get up because there are no nodes available. Because the nodes have been tainted, and the cause of the taint is Disk Pressure. The reason for Disk Pressure is that eviction is configured in the process arguments of kubelet. The condition of eviction detection is to detect the available space of nodefs (/var/lib/kubelet) and imagefs (provided by container runtime).

 

That's all for node.kubernetes.io/disk-pressure:NoSchedule. Thank you for reading.

524 Views
Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.
Version history
Last update:
‎06-20-2021 08:31 PM
Updated by:
Contributors
Top Kudoed Authors