• Recent Articles

    How to Debug DNS Resolution issue in Kubernetes?

    How to Debug DNS Resolution issue in Kubernetes?

    How to Debug DNS Resolution issue in Kubernetes?

    Create a file named dnsutils.yaml with the following contents:

    apiVersion: v1
    kind: Pod
      name: dnsutils
      namespace: default
      - name: dnsutils
        image: gcr.io/kubernetes-e2e-test-images/dnsutils:1.3
          - sleep
          - "3600"
        imagePullPolicy: IfNotPresent
      restartPolicy: Always

    Then create a pod using this file and verify its status:

    [root@kubernetesmaster ~]# kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml
    pod/dnsutils created

    [root@kubernetesmaster ~]# kubectl get pods dnsutils
    dnsutils   1/1     Running   0          53m
    [root@kubernetesmaster ~]#

    Once that pod is running, you can exec nslookup in that environment. If you see something like the following, DNS is working correctly.

    [root@kubernetesmaster ~]# kubectl exec -ti dnsutils -- nslookup kubernetes.default

    Name:   kubernetes.default.svc.cluster.local

    [root@kubernetesmaster ~]#

    Check the local DNS configuration 

    Take a look inside the resolv.conf file. It should look like following.

    [root@kubernetesmaster ~]# kubectl exec dnsutils cat /etc/resolv.conf
    search default.svc.cluster.local svc.cluster.local cluster.local
    options ndots:5
    [root@kubernetesmaster ~]#

    Errors such as the following indicate a problem with the coredns/kube-dns add-on or associated Services:

    kubectl exec -ti dnsutils -- nslookup kubernetes.default
    Address 1:

    nslookup: can't resolve 'kubernetes.default'

    Check if the DNS pod is running

    Use the kubectl get pods command to verify that the DNS pod is running.

    For CoreDNS:

    [root@kubernetesmaster ~]# kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
    NAME                       READY   STATUS    RESTARTS   AGE
    coredns-576cbf47c7-jhwgd   1/1     Running   0          96m
    coredns-576cbf47c7-t97b4   1/1     Running   0          95m
    [root@kubernetesmaster ~]#

    For kube-dns:

    kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
    NAME                    READY     STATUS    RESTARTS   AGE
    kube-dns-v19-ezo1y      3/3       Running   0           1h
    If you see that no pod is running or that the pod has failed/completed, the DNS add-on may not be deployed by default in your current environment and you will have to deploy it manually.

    Check for Errors in the DNS pod

    Use kubectl logs command to see logs for the DNS containers.

    For CoreDNS: Expected to see these kind of logs for a healthy pod

    [root@kubernetesmaster ~]# for p in $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name); do kubectl logs --namespace=kube-system $p; done
    2020/02/14 17:48:13 [INFO] CoreDNS-1.2.2
    2020/02/14 17:48:13 [INFO] linux/amd64, go1.11, eb51e8b
    linux/amd64, go1.11, eb51e8b
    2020/02/14 17:48:13 [INFO] plugin/reload: Running configuration MD5 = f65c4821c8a9b7b5eb30fa4fbc167769

    Is DNS service up?

    Verify that the DNS service is up by using the kubectl get service command.

    [root@kubernetesmaster ~]# kubectl get svc --namespace=kube-system
    NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
    kube-dns             ClusterIP      <none>        53/UDP,53/TCP       53d
    You can verify that DNS endpoints are exposed by using the kubectl get endpoints command.
    [root@kubernetesmaster ~]# kubectl get ep kube-dns --namespace=kube-system
    NAME       ENDPOINTS                                                   AGE
    kube-dns   xxx.xxx.xxx.xxx:53,yyy.yyy.yyy.yyy:53                       53d
    [root@kubernetesmaster ~]#

    How to delete dnsutils pod?

    [root@kubernetesmaster ~]# kubectl delete -f https://k8s.io/examples/admin/dns/dnsutils.yaml
    pod "dnsutils" deleted
    [root@kubernetesmaster ~]#

    No comments