• Recent blogs

    Kubernetes : Kubernetes Node Management, Maintenance, Delete

    Kubernetes : Kubernetes Node Management, Maintenance, Delete

    Kubernetes Node Management is a crucial part of a Kubernetes cluster. In terms of node maintenance, we all need to accept that the compute resources such as physical servers or cloud instances need to have a downtime for maintenance works such as hardware changes, Operating system upgrade or any other maintenance work. Same theory applies to the Kubernetes Cluster Nodes and lets see how we can perform the Kubernetes Node Management, Maintenance and editing PODs without interrupting to it’s services.


    According to the Kubernetes documentation the drain command can be used to “safely evict all of your pods from a node before you perform maintenance on the node,” and “safe evictions allow the pod’s containers to gracefully terminate and will respect the PodDisruptionBudgets you have specified”.

    kubectl drain does following two things:

    1. Cordons the node

    This part is quite simple, cordoning a node means that it will be marked unschedulable, so new pods can no longer be scheduled to the node. If we know in advance that a node will be taken from the cluster (because of maintenance, like a kernel update, or because we know that there will be scaling in the node), cordoning is a good first step. We don’t want new pods scheduled on this node and then taken away after a few seconds. For example, if we know two minutes in advance that a spot instance on AWS will be terminated, new pods shouldn’t be scheduled on that node, then we can work towards gracefully scheduling all the other pods, as well. On the API level, cordoning means patching the node with node.Spec.Unschedulable=true.

    2. Evicts or deletes the pods

    After the node is made unschedulable, the drain command will try to evict the pods that are already running on that node. If eviction is supported on the cluster (from Kubernetes version 1.7) the drain command will use the Eviction API that takes disruption budgets into account, if it’s not supported it will simply delete the pods on the node.

    Commands for this task:

    kubectl drain <n> --delete-local-data --force --ignore-daemonsets # Stop all pods on <n>
    kubectl delete node <name>                                        # Remove <node> from the cluster

    Interesting Articles on Kubernetes:

    Kubernetes : Kubernetes Node Management, Maintenance, Delete
    How to add a  New Worker Node to a existing kubernetes Cluster
    MinIO Client Installation and Quickstart
    PLEG is not healthy: Kubernetes Worker Node is in "NotReady" state
    Backup MySQL databases in Kubernetes
    How to Run Automated Tasks in Kubernetes with a cronjob
    How to Completely remove Kubernetes

    Remove an unresponsive node from your Kubernetes cluster.

    Get a list of your cluster nodes.

    kubectl get nodes

    Following is a sample output:

    NAME             STATUS    ROLES                          AGE       VERSION   Ready     etcd,management,master,proxy   39d       v1.11.1+icp-ee   NotReady  worker                         39d       v1.11.1+icp-ee             <<<<<<< unresponsive node   Ready     worker                         39d       v1.11.1+icp-ee

    Delete the unresponsive node from your cluster.

    kubectl delete node <node-IP-address>

    Following is a sample command and output:

    kubectl delete node
    node "" deleted
    Remove the IP address of the unresponsive node from the <installation_directory>/cluster/hosts file.

    Kubernetes Node Management :

    Lets see how we can perform the Cluster Maintenance work without interrupting the running Services

    Evict PODs and Safely Perform A Node Maintenance
    View the Kubernetes node status :

    kubectl get nodes

    To check the running POD status and the respective nodes :
    kubectl get pods -o wide

    Kubernetes Node Management : Node and POD Status , Disable schedule

    Executing below the command you can safely take a node out from the cluster and running PODs will be evicted from the node

    kubectl drain [NODE_HOSTNAME] --ignore-daemonsets

    Kubernetes Node Management : “uncordon” the node

    Once, the operation completed we can “uncordon” the node and schooling will be enabled again

    kubectl uncordon <NODE_NAME>

    Deleting A Node From The Cluster

    If we are planning to delete a node from the Kubernetes cluster, we have to drain, evict the running PODs and disable the POD scheduling first.

    Kubernetes Node Management : Delete a node

    kubectl delete node [NODE_NAME]

    How to add a  New Worker Node to a existing kubernetes Cluster

    No comments