User Radek 'Goblin' Pieczonka | chat.stackexchange.com

Nov 30, 2017 14:44

so do you

Nov 30, 2017 14:42

I have to go home in a moment, but to point you to something, try adding a udp 53 rule on forward chain and see if it registers trafffic, plus consider debugging with tcpdump

Radek 'Goblin' Pieczonka

Nov 30, 2017 14:41

ok, so it did not reach the nat rules

Radek 'Goblin' Pieczonka

Nov 30, 2017 14:38

nslookup google.com 10.32.0.10

Radek 'Goblin' Pieczonka

Nov 30, 2017 14:38

nono

Radek 'Goblin' Pieczonka

Nov 30, 2017 14:36

yes

Radek 'Goblin' Pieczonka

Nov 30, 2017 14:35

we'll see if counters change

Radek 'Goblin' Pieczonka

Nov 30, 2017 14:35

ok, now make a nslookup to service ip and run the iptables again

Radek 'Goblin' Pieczonka

Nov 30, 2017 14:32

iptables -t nat -nvL | grep kube-dns

Radek 'Goblin' Pieczonka

Nov 30, 2017 14:29

well, that it works from node but not from pod

Radek 'Goblin' Pieczonka

Nov 30, 2017 14:24

surprising

Radek 'Goblin' Pieczonka

Nov 30, 2017 14:16

try the same again from your busybox pod

Radek 'Goblin' Pieczonka

Nov 30, 2017 14:16

this is interesting

Radek 'Goblin' Pieczonka

Nov 30, 2017 14:16

hmm

Radek 'Goblin' Pieczonka

Nov 30, 2017 14:12

also
host google.com 10.200.0.35

Radek 'Goblin' Pieczonka

Nov 30, 2017 14:11

ok, from the actual node (not pod/container) can you do
host google.com 10.32.0.10

Radek 'Goblin' Pieczonka

Nov 30, 2017 14:07

yes

Radek 'Goblin' Pieczonka

Nov 30, 2017 14:06

launch kube-proxy again

Radek 'Goblin' Pieczonka

Nov 30, 2017 14:06

well, it looks decent

Radek 'Goblin' Pieczonka

Nov 30, 2017 14:02

iptables -t nat -nL

Radek 'Goblin' Pieczonka

Nov 30, 2017 14:01

hmm, wonder why they persisted

Radek 'Goblin' Pieczonka

Nov 30, 2017 13:49

hmm... maybe systectl mask kube-proxy so it is not autostarted, reboot and check if iptables afdter restart are empty

Radek 'Goblin' Pieczonka

Nov 30, 2017 13:44

is this a clear machine, or is it possible that there was/is some preexisting firewall in place?

Radek 'Goblin' Pieczonka

Nov 30, 2017 13:43

so in your case this nslookup is kind of that

Radek 'Goblin' Pieczonka

Nov 30, 2017 13:43

service IP works only for the ports defined in the service, as it is implemented as a set of netfilter rules

Radek 'Goblin' Pieczonka

Nov 30, 2017 13:34

well, it seems like your pod is unable to connect to service clusterIP, while it can connect (and resolve) when connecting to pod IP directly.

Radek 'Goblin' Pieczonka

Nov 30, 2017 13:11

ok, so that would explain the initial error messages, but not the lack of clusterip connection

Radek 'Goblin' Pieczonka

Nov 30, 2017 11:44

back in 20... lunch just arrived

Radek 'Goblin' Pieczonka

Nov 30, 2017 11:43

is api server restarting at the same time ?

Radek 'Goblin' Pieczonka

Nov 30, 2017 11:42

Nov 30 12:40:19 cka-cert kube-proxy[973]: E1130 12:40:19.222523 973 reflector.go:205] k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:73: Failed to list *api.Endpoints: Get 192.168.0.218:6443/api/v1/…: dial tcp 192.168.0.218:6443: getsockopt: connection refused
surprises me honestly

Radek 'Goblin' Pieczonka

Nov 30, 2017 11:39

nope, default serviceaccount has no access rights to api any more since like k8s 1.5 with RBAC enabled

Radek 'Goblin' Pieczonka

Nov 30, 2017 11:37

can you restart kube-proxy and provide the log of if from the moment of restart ?

Radek 'Goblin' Pieczonka

Nov 30, 2017 11:37

don't think so... but I'm getting a bit confused here

Radek 'Goblin' Pieczonka

Nov 30, 2017 11:29

RBAC

Radek 'Goblin' Pieczonka

Nov 30, 2017 11:28

kubectl get pod <busyboxname> -o yaml

Radek 'Goblin' Pieczonka

Nov 30, 2017 11:27

ok... it gets interesting indeed.

Radek 'Goblin' Pieczonka

Nov 30, 2017 11:21

unless you deleted the previous coredns pod already

Radek 'Goblin' Pieczonka

Nov 30, 2017 11:21

10.200.0.30 vs 10.200.0.31

Radek 'Goblin' Pieczonka

Nov 30, 2017 11:17

oh, and also kubectl -n kube-system describe svc kube-dns

Radek 'Goblin' Pieczonka

Nov 30, 2017 11:16

elhombre: how about result of iptables -t nat -nL | grep kube-dns

Radek 'Goblin' Pieczonka

Nov 30, 2017 11:14

SV: in your GH repo it's a NodePort, hence my question

Radek 'Goblin' Pieczonka

Nov 30, 2017 11:13

el: anything usefull in journalctl -u kube-proxy (or however the unit is called) ?

Radek 'Goblin' Pieczonka

Nov 30, 2017 11:12

SV: in your nginx service when you test it, what port do you call ?

Radek 'Goblin' Pieczonka

Nov 30, 2017 11:09

(do you launch it on host as something like say systemd unit or is it a DaemonSet)

Radek 'Goblin' Pieczonka

Nov 30, 2017 11:09

do you have kube-proxy running on the node and does it run without errors

Radek 'Goblin' Pieczonka

Nov 30, 2017 11:08

well, 1st place to look then is kube-proxy or iptables

Radek 'Goblin' Pieczonka

Nov 30, 2017 11:08

10.32.0.10 being service ClusterIP I assume

Radek 'Goblin' Pieczonka

Nov 30, 2017 11:05

what iptables rules boil down to is a set of DNAT rules with a growing matching probability

Radek 'Goblin' Pieczonka

Nov 30, 2017 11:03

you dont have host command

Radek 'Goblin' Pieczonka

Nov 30, 2017 11:02

without kube-proxy, you should be able to connect between pods by means of pods own IPs, with kube-proxy you effectively configure iptables based local loadbalancing rules translating services CluserIP to one of backing pods IP (random matching from all pods that service selector matches

Radek 'Goblin' Pieczonka

How to troubleshoot udp port unreacha…