User elhombre | chat.stackexchange.com

elhombre

Nov 30, 2017 14:44

@Radek'Goblin'Pieczonka Have a nice evening!

elhombre

Nov 30, 2017 14:43

@Radek'Goblin'Pieczonka Thank you very much for taking the time to go through this problem. I highly appreciate the help you have given to me today and it was a joy!

elhombre

Nov 30, 2017 14:42

@Rad

elhombre

Nov 30, 2017 14:40

iptables -t nat -nvL | grep kube-dns
0 0 KUBE-MARK-MASQ all -- * * 10.200.0.35 0.0.0.0/0 /* kube-system/kube-dns:dns */
0 0 DNAT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */ udp to:10.200.0.35:53
0 0 KUBE-MARK-MASQ all -- * * 10.200.0.35 0.0.0.0/0 /* kube-system/kube-dns:metrics */
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:metrics */ tcp to:10.200.0.35:9153

elhombre

Nov 30, 2017 14:40

/ # nslookup google.com 10.32.0.10
Server: 10.32.0.10
Address 1: 10.32.0.10

nslookup: can't resolve 'google.com'

elhombre

Nov 30, 2017 14:38

# nslookup 10.32.0.10
Server: 10.32.0.10
Address 1: 10.32.0.10

Name: 10.32.0.10
Address 1: 10.32.0.10

elhombre

Nov 30, 2017 14:38

waiting ....

elhombre

Nov 30, 2017 14:37

nslookup 10.32.0.10

elhombre

Nov 30, 2017 14:36

from busybox?

elhombre

Nov 30, 2017 14:32

iptables -t nat -nvL | grep kube-dns
0 0 KUBE-MARK-MASQ all -- * * 10.200.0.35 0.0.0.0/0 /* kube-system/kube-dns:dns */
0 0 DNAT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */ udp to:10.200.0.35:53
0 0 KUBE-MARK-MASQ all -- * * 10.200.0.35 0.0.0.0/0 /* kube-system/kube-dns:metrics */
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:metrics */ tcp to:10.200.0.35:9153

elhombre

Nov 30, 2017 14:28

@Radek'Goblin'Pieczonka One lookup is going over the coredns service the other goes directly over the coredns pod. Where am I missing the suprise?

elhombre

Nov 30, 2017 14:23

nslookup google.com 10.32.0.10
Server: 10.32.0.10
Address 1: 10.32.0.10

nslookup: can't resolve 'google.com'
/ # host google.com 10.200.0.35

Name: google.com
Address 1: 2a00:1450:400a:805::200e XXX04s06-in-x0e.1e100.net
Address 2: 172.217.16.142 XXX04s06-in-f14.1e100.net

elhombre

Nov 30, 2017 14:17

Doing ...

elhombre

Nov 30, 2017 14:14

host google.com 10.32.0.10
Using domain server:
Name: 10.32.0.10
Address: 10.32.0.10#53
Aliases:

google.com has address 172.217.16.142
google.com has IPv6 address 2a00:1450:400a:805::200e
google.com mail is handled by 40 alt3.aspmx.l.google.com.
google.com mail is handled by 30 alt2.aspmx.l.google.com.
google.com mail is handled by 50 alt4.aspmx.l.google.com.
google.com mail is handled by 20 alt1.aspmx.l.google.com.
google.com mail is handled by 10 aspmx.l.google.com.
root@cka-cert:~# host google.com 10.200.0.35

elhombre

Nov 30, 2017 14:07

Chain PREROUTING (policy ACCEPT)
target prot opt source destination
KUBE-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
KUBE-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination

elhombre

Nov 30, 2017 14:06

Done. Do you want me to iptables -t nat -nL

elhombre

Nov 30, 2017 14:03

iptables -t nat -nL
Chain PREROUTING (policy ACCEPT)
target prot opt source destination

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
KUBE-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes postrouting rules */
CNI-f38165a782ddcf73d57378a6 all -- 10.200.0.0/16 0.0.0.0/0 /* name: "bridge" id: "d53327bf9d3660232cf44b0617eb36…

elhombre

Nov 30, 2017 13:59

@Radek'Goblin'Pieczonka

elhombre

Nov 30, 2017 13:59

iptables --list-rules
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N KUBE-FIREWALL
-A INPUT -j KUBE-FIREWALL
-A OUTPUT -j KUBE-FIREWALL
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP

elhombre

Nov 30, 2017 13:59

Seems I have a problem with masking the service (Failed to execute operation: Invalid argument). I just disabled it and rebooted.

elhombre

Nov 30, 2017 13:49

doing that....

elhombre

Nov 30, 2017 13:46

Yep there was a preexisting firewalld in place (ubuntu/ufw) which I disabled after realising that I forgot about that. Can it be that ufw mixed up the iptables which after a restart cant be reset?

elhombre

Nov 30, 2017 13:40

@Radek'Goblin'Pieczonka Do you know of a test method like pinging service clusterIP's?

elhombre

Nov 30, 2017 13:38

@Radek'Goblin'Pieczonka Thank you

elhombre

Nov 30, 2017 13:15

@Radek'Goblin'Pieczonka What are you refering to with lack of clusterip connection?

elhombre

Nov 30, 2017 12:58

@Radek'Goblin'Pieczonka Regarding if the API server is restarting at the same time, you are right it does. I haven't declared an order of services to start.

elhombre

Nov 30, 2017 12:45

@Radek'Goblin'Pieczonka You there?

elhombre

Nov 30, 2017 12:44

Back to my problem now....

elhombre

Nov 30, 2017 12:44

@SureshVishnoi Have a nice day pressing the bank. Should I find it out I will give you the update ;)

elhombre

Nov 30, 2017 12:38

tcp LISTEN 0 128 127.0.0.1:10251 *:*
tcp LISTEN 0 128 127.0.0.1:2379 *:*
tcp LISTEN 0 128 127.0.0.1:10252 *:*
tcp LISTEN 0 128 127.0.0.1:2380 *:*
tcp LISTEN 0 128 192.168.0.217:2222 *:*
tcp LISTEN 0 128 192.168.0.217:80 *:*

elhombre

Nov 30, 2017 12:38

should be normally like this

elhombre

Nov 30, 2017 12:38

ss -tulpn | grep 30469
tcp LISTEN 0 128 :::30469 :::*

elhombre

Nov 30, 2017 12:37

@SureshVishnoi I don't get it neither, but I can remember that I found thread where this was discussed with a solution. I am just wondering why on my machine it now want's to expose ipv6 ports instead of ipv4?

elhombre

Nov 30, 2017 12:20

let me see what happens if I expose manually

elhombre

Nov 30, 2017 12:18

@SureshVishnoi It's a bit weird on my side. So I setup as you told in the instructions. but now the nodeport is open on a ipv6 port????

elhombre

Nov 30, 2017 12:15

@SureshVishnoi Regarding the port-foward comment: kubectl port-forward forwards internal cluster traffic to your kubectl client. Like this you can test cluster internal communications. (ex. reaching all internal services and pods from your desktop)

elhombre

Nov 30, 2017 11:57

@SureshVishnoi looking at it....

elhombre

Nov 30, 2017 11:54

@SureshVishnoi We all are on a continuous journey of learning ;) Let's see where this one is going to get us.

elhombre

Nov 30, 2017 11:52

@SureshVishnoi You too thank you for helping out!

elhombre

Nov 30, 2017 11:45

@Radek'Goblin'Pieczonka sure, will be here. Thank you very much for taking time to help. Have a good lunch :)

elhombre

Nov 30, 2017 11:43

Let me see....

elhombre

Nov 30, 2017 11:41

@Radek'Goblin'Pieczonka Here you go -- Logs begin at Thu 2017-11-23 14:34:27 CET, end at Thu 2017-11-30 12:40:37 CET. --
Nov 30 12:40:07 cka-cert systemd[1]: Started Kubernetes Kube Proxy.
Nov 30 12:40:11 cka-cert kube-proxy[973]: W1130 12:40:11.783697 973 server.go:191] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
Nov 30 12:40:12 cka-cert kube-proxy[973]: I1130 12:40:12.526452 973 server_others.go:117] Using iptables Proxier.

elhombre

Nov 30, 2017 11:39

@Radek'Goblin'Pieczonka That would be chat.stackexchange.com/transcript/message/41451943#41451943 But I will restart and post a full log. Give me a moment....

elhombre

Nov 30, 2017 11:36

@Radek'Goblin'Pieczonka Is my suspicion correct that the error may be with RBAC??

elhombre

Nov 30, 2017 11:31

apiVersion: v1
kind: Pod
metadata:
annotations:
kubernetes.io/created-by: |
{"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"busybox-56db8bd9d7","uid":"0b422fda-d5a1-11e7-8a78-080027cc30d9","apiVersion":"extensions","resourceVersion":"82753"}}
creationTimestamp: 2017-11-30T07:35:31Z
generateName: busybox-56db8bd9d7-
labels:
pod-template-hash: "1286468583"
run: busybox
name: busybox-56db8bd9d7-fv7np
namespace: default
ownerReferences:

elhombre

Nov 30, 2017 11:31

your command: kubectl get pod <busyboxname> -o yaml

elhombre

Nov 30, 2017 11:29

curl --cacert ca.pem https://192.168.0.218:6443/api/v1/endpoints?resourceVersion=0
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {

},
"status": "Failure",
"message": "endpoints is forbidden: User \"system:anonymous\" cannot list endpoints at the cluster scope",
"reason": "Forbidden",
"details": {
"kind": "endpoints"
},
"code": 403

elhombre

Nov 30, 2017 11:29

.... dpoints: Get 192.168.0.218:6443/api/v1/endpoints?resourceVersion=0: dial tcp 192.168.0.218:6443: getsockopt: connection refused

elhombre

Nov 30, 2017 11:28

@Radek'Goblin'Pieczonka You know what is even more interesting :)

elhombre

Nov 30, 2017 11:26

nslookup kubernetes.default.svc.cluster.local 10.200.0.31
Server: 10.200.0.31
Address 1: 10.200.0.31

Name: kubernetes.default.svc.cluster.local
Address 1: 10.32.0.1 kubernetes.default.svc.cluster.local

elhombre

How to troubleshoot udp port unreacha…