How to troubleshoot udp port unreachable between busybox and kube-dns

elhombre

10:50

Hello?

Suresh Vishnoi

Hi ,

elhombre

gr8 it works, much easier :)

Suresh Vishnoi

from my understanding, Service resources is there to expose pod to internal and external world

yea that's true

If a pod does not have clusterIp then how other pods discover it ?

elhombre

My understanding: A service resource can primarly be used to expose a pod insight a the cluster network. This is because pods get dynamic ip's services have static ip which redirect to the pod list it saves.

Radek 'Goblin' Pieczonka

they do not need to discover it, if they dont need to initiate the connection to it

elhombre

10:55

@Radek'Goblin'Pieczonka I agree

Radek 'Goblin' Pieczonka

if it is pod that sends the initial packet, the communication gets established on IP level

it's like with your own computer, you do not need the 8.8.8.8 google dns server to "see you" on some IP

@elhombre posted a bit longer answer on how I'd approach debugging

Suresh Vishnoi

that's interesting

elhombre

So the initiator is busybox, in this case busybox doesn't need a service. But coredns pod is the receiver so it needs a service in front to be reachable hence svc/kube-dns in my setup :)

@Radek'Goblin'Pieczonka Checking out your answer....

Suresh Vishnoi

does kube-proxy make rules for pod IP address in the NAT table ?

Radek 'Goblin' Pieczonka

nope

kube-proxy makes rules for services with ClusterIP

elhombre

11:02

@Radek'Goblin'Pieczonka Ok did what you gave as an answer.

Radek 'Goblin' Pieczonka

without kube-proxy, you should be able to connect between pods by means of pods own IPs, with kube-proxy you effectively configure iptables based local loadbalancing rules translating services CluserIP to one of backing pods IP (random matching from all pods that service selector matches

elhombre

/ # host kubernetes.default.svc.cluster.local 10.200.0.30
/bin/sh: host: not found
/ # host kubernetes.default.svc.cluster.local 10.32.0.10
/bin/sh: host: not found

Radek 'Goblin' Pieczonka

you dont have host command

Suresh Vishnoi

Yea. I have been stuck with one of my deployment, I have been trying to understand IPTABLEs rules in the NAT TABLE. At the moment, I could not make a decision how all of the rules are made by Kube-proxy. https://github.com/vishnoisuresh/netbox-kubernetes
This is git-repo

elhombre

@Radek'Goblin'Pieczonka yes, right, sorry :blush:

Radek 'Goblin' Pieczonka

11:05

what iptables rules boil down to is a set of DNAT rules with a growing matching probability

Suresh Vishnoi

In My deployment, When I access the nginx pod from browser I got a response as "bad request" However When I try using "kubectl port-forward " it works like a charm.

elhombre

# nslookup kubernetes.default.svc.cluster.local 10.200.0.30
Server: 10.200.0.30
Address 1: 10.200.0.30

Name: kubernetes.default.svc.cluster.local
Address 1: 10.32.0.1 kubernetes.default.svc.cluster.local

nslookup kubernetes.default.svc.cluster.local 10.32.0.10
Server: 10.32.0.10
Address 1: 10.32.0.10

nslookup: can't resolve 'kubernetes.default.svc.cluster.local'

Radek 'Goblin' Pieczonka

10.32.0.10 being service ClusterIP I assume

elhombre

kube-system svc/kube-dns ClusterIP 10.32.0.10 <none> 53/UDP,53/TCP,9153/TCP

Radek 'Goblin' Pieczonka

well, 1st place to look then is kube-proxy or iptables

elhombre

11:09

Looking at the logs.....

Radek 'Goblin' Pieczonka

do you have kube-proxy running on the node and does it run without errors

(do you launch it on host as something like say systemd unit or is it a DaemonSet)

elhombre

kube-proxy as well as all other kubernetes components run as bare binaries on a single machine. The are run by systemd.

My setup is on my local bare metals server in a vm. It is an all in one setup (master/node on a single bare metal instance) inspired by github.com/kelseyhightower/kubernetes-the-hard-way

Radek 'Goblin' Pieczonka

SV: in your nginx service when you test it, what port do you call ?

el: anything usefull in journalctl -u kube-proxy (or however the unit is called) ?

Suresh Vishnoi

what difference does it make if it runs as DaemonSet or systemD Unit.? I understand that Daemonset is created by DockerEngine

When I try to accesc through loadbalancer then just externalIP

Radek 'Goblin' Pieczonka

SV: in your GH repo it's a NodePort, hence my question

Suresh Vishnoi

11:14

Rodek: My repo was for minikube

sorry about that

I tried to deploy the repo on Azure I had similar result. I deployed a normal nginx then compare the IPTABLES rules for both of the pods and there are 3 extra rule for normal nginx.

Radek 'Goblin' Pieczonka

elhombre: how about result of iptables -t nat -nL | grep kube-dns

oh, and also kubectl -n kube-system describe svc kube-dns

elhombre

@Radek'Goblin'Pieczonka Regarding kube-proxy, seems to not look good. I restarted the server to get some new logs.

● kube-proxy.service - Kubernetes Kube Proxy
Loaded: loaded (/etc/systemd/system/kube-proxy.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2017-11-30 12:15:27 CET; 1min 48s ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Main PID: 997 (kube-proxy)
Tasks: 0
Memory: 26.4M
CPU: 175ms
CGroup: /system.slice/kube-proxy.service
‣ 997 /usr/local/bin/kube-proxy --cluster-cidr=10.200.0.0/16 --kubeconfig=/var/lib/kube-proxy/kubeconfig --proxy-mode=iptables --v=2

Nov 30 12:15:39 cka-cert kube-proxy[997]: E1130 12:15:39.126267 997 reflector.go:205] k8s.io/k…

iptables -t nat -nL | grep kube-dns
KUBE-MARK-MASQ all -- 10.200.0.31 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */
DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */ tcp to:10.200.0.31:53
KUBE-MARK-MASQ all -- 10.200.0.31 0.0.0.0/0 /* kube-system/kube-dns:dns */
DNAT udp -- 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */ udp to:10.200.0.31:53
KUBE-MARK-MASQ all -- 10.200.0.31 0.0.0.0/0 /* kube-system/kube-dns:metrics */

kubectl -n kube-system describe svc kube-dns
Name: kube-dns
Namespace: kube-system
Labels: k8s-app=coredns
kubernetes.io/cluster-service=true
kubernetes.io/name=CoreDNS
Annotations: <none>
Selector: k8s-app=coredns
Type: ClusterIP
IP: 10.32.0.10
Port: dns 53/UDP
TargetPort: 53/UDP
Endpoints: 10.200.0.31:53
Port: dns-tcp 53/TCP
TargetPort: 53/TCP
Endpoints: 10.200.0.31:53

Radek 'Goblin' Pieczonka

10.200.0.30 vs 10.200.0.31

unless you deleted the previous coredns pod already

elhombre

@Radek'Goblin'Pieczonka From busybox

nslookup kubernetes.default.svc.cluster.local 10.200.0.31
Server: 10.200.0.31
Address 1: 10.200.0.31

Name: kubernetes.default.svc.cluster.local
Address 1: 10.32.0.1 kubernetes.default.svc.cluster.local

Radek 'Goblin' Pieczonka

ok... it gets interesting indeed.

kubectl get pod <busyboxname> -o yaml

elhombre

11:28

@Radek'Goblin'Pieczonka You know what is even more interesting :)

.... dpoints: Get 192.168.0.218:6443/api/v1/endpoints?resourceVersion=0: dial tcp 192.168.0.218:6443: getsockopt: connection refused

curl --cacert ca.pem https://192.168.0.218:6443/api/v1/endpoints?resourceVersion=0
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {

},
"status": "Failure",
"message": "endpoints is forbidden: User \"system:anonymous\" cannot list endpoints at the cluster scope",
"reason": "Forbidden",
"details": {
"kind": "endpoints"
},
"code": 403

Radek 'Goblin' Pieczonka

RBAC

elhombre

your command: kubectl get pod <busyboxname> -o yaml

apiVersion: v1
kind: Pod
metadata:
annotations:
kubernetes.io/created-by: |
{"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"busybox-56db8bd9d7","uid":"0b422fda-d5a1-11e7-8a78-080027cc30d9","apiVersion":"extensions","resourceVersion":"82753"}}
creationTimestamp: 2017-11-30T07:35:31Z
generateName: busybox-56db8bd9d7-
labels:
pod-template-hash: "1286468583"
run: busybox
name: busybox-56db8bd9d7-fv7np
namespace: default
ownerReferences:

@Radek'Goblin'Pieczonka Is my suspicion correct that the error may be with RBAC??

Radek 'Goblin' Pieczonka

don't think so... but I'm getting a bit confused here

can you restart kube-proxy and provide the log of if from the moment of restart ?

Suresh Vishnoi

If it is RBAC issue, then I do not understand with the default service account the pod should have the capability to operate this task.

Radek 'Goblin' Pieczonka

nope, default serviceaccount has no access rights to api any more since like k8s 1.5 with RBAC enabled

elhombre

11:39

@Radek'Goblin'Pieczonka That would be chat.stackexchange.com/transcript/message/41451943#41451943 But I will restart and post a full log. Give me a moment....

@Radek'Goblin'Pieczonka Here you go -- Logs begin at Thu 2017-11-23 14:34:27 CET, end at Thu 2017-11-30 12:40:37 CET. --
Nov 30 12:40:07 cka-cert systemd[1]: Started Kubernetes Kube Proxy.
Nov 30 12:40:11 cka-cert kube-proxy[973]: W1130 12:40:11.783697 973 server.go:191] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
Nov 30 12:40:12 cka-cert kube-proxy[973]: I1130 12:40:12.526452 973 server_others.go:117] Using iptables Proxier.

Radek 'Goblin' Pieczonka

Nov 30 12:40:19 cka-cert kube-proxy[973]: E1130 12:40:19.222523 973 reflector.go:205] k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:73: Failed to list *api.Endpoints: Get 192.168.0.218:6443/api/v1/…: dial tcp 192.168.0.218:6443: getsockopt: connection refused
surprises me honestly

is api server restarting at the same time ?

elhombre

Let me see....

Radek 'Goblin' Pieczonka

back in 20... lunch just arrived

elhombre

@Radek'Goblin'Pieczonka sure, will be here. Thank you very much for taking time to help. Have a good lunch :)

@SureshVishnoi You too thank you for helping out!

Suresh Vishnoi

No Worries :) I am also learning here. I have limited knowledge with IPTABLES

elhombre

11:54

@SureshVishnoi We all are on a continuous journey of learning ;) Let's see where this one is going to get us.

Suresh Vishnoi

True :)

If you happen to have time please try out this repo https://github.com/vishnoisuresh/netbox-kubernetes
I am having the issue with iptables. when I try to access the nginx pod on the browser with nodeport I get 'bad request' response, however, with 'kubectl port-forwad' it works .

Its very strange behaviour of Kube-proxy

elhombre

@SureshVishnoi looking at it....

Suresh Vishnoi

Thanks :)

elhombre

12:15

@SureshVishnoi Regarding the port-foward comment: kubectl port-forward forwards internal cluster traffic to your kubectl client. Like this you can test cluster internal communications. (ex. reaching all internal services and pods from your desktop)

Suresh Vishnoi

yea

I was testing by using it

In the beginning I thought there is a problem with resources manifests However this command help me understand that there is no problem with manifests

then I tried to debug kube-proxy , here I am stuck with it

elhombre

@SureshVishnoi It's a bit weird on my side. So I setup as you told in the instructions. but now the nodeport is open on a ipv6 port????

Suresh Vishnoi

ohhh ok

yea, I had same situation in netstat , port was assign to ipv6

elhombre

let me see what happens if I expose manually

Suresh Vishnoi

12:33

How does it give ipv6 port ? I need to research it

elhombre

@SureshVishnoi I don't get it neither, but I can remember that I found thread where this was discussed with a solution. I am just wondering why on my machine it now want's to expose ipv6 ports instead of ipv4?

ss -tulpn | grep 30469
tcp LISTEN 0 128 :::30469 :::*

should be normally like this

tcp LISTEN 0 128 127.0.0.1:10251 *:*
tcp LISTEN 0 128 127.0.0.1:2379 *:*
tcp LISTEN 0 128 127.0.0.1:10252 *:*
tcp LISTEN 0 128 127.0.0.1:2380 *:*
tcp LISTEN 0 128 192.168.0.217:2222 *:*
tcp LISTEN 0 128 192.168.0.217:80 *:*

Suresh Vishnoi

Thanks :) I will try to understand it and update you. Now I think I should go my University

elhombre

@SureshVishnoi Have a nice day pressing the bank. Should I find it out I will give you the update ;)

Back to my problem now....

@Radek'Goblin'Pieczonka You there?

elhombre

12:58

@Radek'Goblin'Pieczonka Regarding if the API server is restarting at the same time, you are right it does. I haven't declared an order of services to start.

Radek 'Goblin' Pieczonka

13:11

ok, so that would explain the initial error messages, but not the lack of clusterip connection

elhombre

@Radek'Goblin'Pieczonka What are you refering to with lack of clusterip connection?

Radek 'Goblin' Pieczonka

13:34

well, it seems like your pod is unable to connect to service clusterIP, while it can connect (and resolve) when connecting to pod IP directly.

elhombre

@Radek'Goblin'Pieczonka Thank you

@Radek'Goblin'Pieczonka Do you know of a test method like pinging service clusterIP's?

Radek 'Goblin' Pieczonka

service IP works only for the ports defined in the service, as it is implemented as a set of netfilter rules

so in your case this nslookup is kind of that

is this a clear machine, or is it possible that there was/is some preexisting firewall in place?

elhombre

Yep there was a preexisting firewalld in place (ubuntu/ufw) which I disabled after realising that I forgot about that. Can it be that ufw mixed up the iptables which after a restart cant be reset?

Radek 'Goblin' Pieczonka

hmm... maybe systectl mask kube-proxy so it is not autostarted, reboot and check if iptables afdter restart are empty

elhombre

13:49

doing that....

Seems I have a problem with masking the service (Failed to execute operation: Invalid argument). I just disabled it and rebooted.

iptables --list-rules
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N KUBE-FIREWALL
-A INPUT -j KUBE-FIREWALL
-A OUTPUT -j KUBE-FIREWALL
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP

@Radek'Goblin'Pieczonka

Radek 'Goblin' Pieczonka

hmm, wonder why they persisted

iptables -t nat -nL

elhombre

iptables -t nat -nL
Chain PREROUTING (policy ACCEPT)
target prot opt source destination

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
KUBE-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes postrouting rules */
CNI-f38165a782ddcf73d57378a6 all -- 10.200.0.0/16 0.0.0.0/0 /* name: "bridge" id: "d53327bf9d3660232cf44b0617eb36…

Radek 'Goblin' Pieczonka

14:06

well, it looks decent

launch kube-proxy again

elhombre

Done. Do you want me to iptables -t nat -nL

Radek 'Goblin' Pieczonka

yes

elhombre

Chain PREROUTING (policy ACCEPT)
target prot opt source destination
KUBE-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
KUBE-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination

Radek 'Goblin' Pieczonka

ok, from the actual node (not pod/container) can you do
host google.com 10.32.0.10

also
host google.com 10.200.0.35

elhombre

host google.com 10.32.0.10
Using domain server:
Name: 10.32.0.10
Address: 10.32.0.10#53
Aliases:

google.com has address 172.217.16.142
google.com has IPv6 address 2a00:1450:400a:805::200e
google.com mail is handled by 40 alt3.aspmx.l.google.com.
google.com mail is handled by 30 alt2.aspmx.l.google.com.
google.com mail is handled by 50 alt4.aspmx.l.google.com.
google.com mail is handled by 20 alt1.aspmx.l.google.com.
google.com mail is handled by 10 aspmx.l.google.com.
root@cka-cert:~# host google.com 10.200.0.35

Radek 'Goblin' Pieczonka

14:16

hmm

this is interesting

try the same again from your busybox pod

elhombre

Doing ...

nslookup google.com 10.32.0.10
Server: 10.32.0.10
Address 1: 10.32.0.10

nslookup: can't resolve 'google.com'
/ # host google.com 10.200.0.35

Name: google.com
Address 1: 2a00:1450:400a:805::200e XXX04s06-in-x0e.1e100.net
Address 2: 172.217.16.142 XXX04s06-in-f14.1e100.net

Radek 'Goblin' Pieczonka

surprising

elhombre

@Radek'Goblin'Pieczonka One lookup is going over the coredns service the other goes directly over the coredns pod. Where am I missing the suprise?

Radek 'Goblin' Pieczonka

well, that it works from node but not from pod

iptables -t nat -nvL | grep kube-dns

elhombre

14:32

iptables -t nat -nvL | grep kube-dns
0 0 KUBE-MARK-MASQ all -- * * 10.200.0.35 0.0.0.0/0 /* kube-system/kube-dns:dns */
0 0 DNAT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */ udp to:10.200.0.35:53
0 0 KUBE-MARK-MASQ all -- * * 10.200.0.35 0.0.0.0/0 /* kube-system/kube-dns:metrics */
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:metrics */ tcp to:10.200.0.35:9153

Radek 'Goblin' Pieczonka

ok, now make a nslookup to service ip and run the iptables again

we'll see if counters change

elhombre

from busybox?

Radek 'Goblin' Pieczonka

yes

elhombre

nslookup 10.32.0.10

waiting ....

# nslookup 10.32.0.10
Server: 10.32.0.10
Address 1: 10.32.0.10

Name: 10.32.0.10
Address 1: 10.32.0.10

Radek 'Goblin' Pieczonka

nono

nslookup google.com 10.32.0.10

elhombre

14:40

/ # nslookup google.com 10.32.0.10
Server: 10.32.0.10
Address 1: 10.32.0.10

nslookup: can't resolve 'google.com'

iptables -t nat -nvL | grep kube-dns
0 0 KUBE-MARK-MASQ all -- * * 10.200.0.35 0.0.0.0/0 /* kube-system/kube-dns:dns */
0 0 DNAT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */ udp to:10.200.0.35:53
0 0 KUBE-MARK-MASQ all -- * * 10.200.0.35 0.0.0.0/0 /* kube-system/kube-dns:metrics */
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:metrics */ tcp to:10.200.0.35:9153

Radek 'Goblin' Pieczonka

ok, so it did not reach the nat rules

I have to go home in a moment, but to point you to something, try adding a udp 53 rule on forward chain and see if it registers trafffic, plus consider debugging with tcpdump

elhombre

@Rad

@Radek'Goblin'Pieczonka Thank you very much for taking the time to go through this problem. I highly appreciate the help you have given to me today and it was a joy!

@Radek'Goblin'Pieczonka Have a nice evening!

Radek 'Goblin' Pieczonka

so do you

Transcript for

How to troubleshoot udp port unreacha…