Discussion between user1847927 and A.B

Feeds

16:31

2

Q: Routing table behaviour coupled with bound sockets - unexpected behaviour

Problem I am experiencing some rather odd behavior where a seemingly unrelated default gateway route is having unexpected side-effects. I managed to replicate this issue with a minimal example. The aim here is mostly educational and I stumbled upon this while experimenting with a more complex sce...

A.B

Can you also provide ip -br addr after having deleted the routes (because there are more than one way to do that)? Details matter here.

user1847927

Most certainly. I updated the post to include the commands I used to delete and add routes and provided the output of ip -br addr

Sorry, I was just adding a comment (I updated the post so that I can ask you this below): If ip route get oif wlp2s0 192.168.0.3 gives back the same route in both cases (please see updated post), why does curl --interface wlp2s0 192.168.0.3 block when I remove the related 192.168.0.0/24 routes and then work when I add a bogus default route to 172.17.0.2?

A.B

Indeed there's something else to figure out.

While I made a mockup to reproduce the experiment, this wasn't a complete mockup: no Docker is running. I can only guess the presence of Docker and iptables rules it's adding can change the behavior. In my test, it works the same with or without the route through docker0. I won't be able to go further and hope someone else will provide the final explanation.

user1847927

Does curl --interface wlp2s0 192.168.0.3 always succeed in your case? This is useful information. Indeed Docker could be affecting this. I will try to disable it temporarily and check whether anything changes.

A.B

Yes it always succeeds. Disabling docker is not enough, unless you erase its firewall/nat rules and also do rmmod br_netfilter.

user1847927

16:33

The site suggested that I start a chat to avoid lengthy discussions. I hope that's ok.

I disabled docker using

`sudo systemctl disable docker.service`
`sudo systemctl disable docker.socket`

and rebooted.

On boot docker was not running, iptables seems to be empty and `lsmod | grep filter` gives no result.

I retried the whole scenario and unfortunately the result is the same

A.B

Ok. well I don't know what can differ then.

You can use tcpdump and see when you get or don't get ARPs and on what interface this happens.

user1847927

Yes, I just ran tcpdump using :

sudo tcpdump -i wlp2s0 -n host 192.168.0.3 and port 80

A.B

add 'or arp or icmp'

user1847927

ok

before I add arp or icmp, let me show you the results because I think there is an indication

A.B

anyway the ARP case takes 3s. I noticed you didn't always get 3s. no 3s means no ARP even attempted.

or anyway no ARP failure

ok

user1847927

16:41

this is the case when I do not modify the routes (so when everything works):

kevin@kevin-UX305LA:~$ sudo tcpdump -i wlp2s0 -n host 192.168.0.3 and port 80
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on wlp2s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
18:41:03.600766 IP 192.168.0.210.37614 > 192.168.0.3.80: Flags [S], seq 1925889800, win 64240, options [mss 1460,sackOK,TS val 1662348692 ecr 0,nop,wscale 7], length 0
18:41:03.603854 IP 192.168.0.3.80 > 192.168.0.210.37614: Flags [S.], seq 1919582977, ack 1925889801, win 65160, options …

now, let me remove the routes and show the result (might take ma a few mins)

A.B

moved where?

Indeed that means it succeeds

user1847927

what sorry? ah moved is the http reply, that's expected

A.B

an never mind ok the working case

user1847927

ok im back online, so i removed the routes as per usual:

kevin@kevin-UX305LA:~$ ip r
default via 192.168.0.1 dev wlp2s0 proto dhcp metric 600
169.254.0.0/16 dev wlp2s0 scope link metric 1000
192.168.0.0/24 dev wlp2s0 proto kernel scope link src 192.168.0.210 metric 600
kevin@kevin-UX305LA:~$ sudo ip route del default via 192.168.0.1
[sudo] password for kevin:
kevin@kevin-UX305LA:~$ sudo ip route del 192.168.0.0/24
kevin@kevin-UX305LA:~$ ip r
169.254.0.0/16 dev wlp2s0 scope link metric 1000
kevin@kevin-UX305LA:~$ curl --interface wlp2s0 192.168.0.3

this time round i waited enough for it to timeout

and here is tcpdump result

kevin@kevin-UX305LA:~$ sudo tcpdump -i wlp2s0 -n host 192.168.0.3 and port 80
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on wlp2s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
18:44:24.556021 IP 192.168.0.210.46406 > 192.168.0.3.80: Flags [S], seq 3302209512, win 64240, options [mss 1460,sackOK,TS val 1662549647 ecr 0,nop,wscale 7], length 0
18:44:24.559109 IP 192.168.0.3.80 > 192.168.0.210.46406: Flags [S.], seq 2206567877, ack 3302209513, win 65160, options [mss 1460,sackOK,TS val 805860058 ecr 1662549647,nop,wscale 7], length 0

A.B

what about running these 2 commands?

sysctl -w net.ipv4.conf.all.rp_filter=0

sysctl -w net.ipv4.conf.wlp2s0.rp_filter=0

Seems I can reproduce the timeout when it's =1 so I assume it's =1 on your system.

or maybe =2

before doing it, what's the result of sysctl -ar \\.rp_filter (with twice a \ )

user1847927

16:55

now it worked! It was 2 on my system

A.B

yes 2 would explain everything (including the default route elsewhere)

without going into details 2 is similar to 0 only in the presence of a default route

user1847927

I have no idea what that config is for in the first place.. need to read into it

A.B

I can update my answer to explain all this, but this can take quite some time

user1847927

I must thank you very much!

I think updating your answer would be best (if you have time to do so)

A.B

I'll add an UPDATE to the answer (rather than rewriting completely) explaining this part

everything written was correct, but the other way around wasn't considered

user1847927

17:02

sound great. Will accept it as the correct answer when you are ready

user1847927

17:34

made some final tests: with a value of 1, curl doesn't work (regardless of the presence of a default gateway), with 2 it only works if there's a default gateway, and 0 works always. This confirms what you said earlier :)

A.B

17:48

answer updated

user1847927

17:58

accepted. Thanks for explaining thoroughly, much appreciated!

Transcript for

Discussion between user1847927 and A.B