last day (16 days later) » 

8:17 PM
0
Q: Docker timeout port 53 - ISP to blame?

Dennis ZaragosaI'm running a Synology NAS with docker and used to have multiple containers running perfectly fine for months. Last week I switched ISP and use their router to connect to the internet. Since then my containers are unable to make any outbound requests. I first thought there was a problem with Dock...

 
It looks like you are using dnsmasq for DNS server (for cache DNS etc..). The resolver is not able to request 8.8.4.4:53 (which is a Google Public DNS server). You got a timeout, so it is a firewall blocking issue. You can't resolve anything from your containers.
 
@hidigoudi Its indeed the new router that I received. I tried completely turning off its firewall (just for testing purposes), but that did not resolve the issue. What other options are there left for me?
 
What is the error ? You got a timeout or something else ? - Are you able to run from the container : host github.com or nslookup github.com without error ? - You can also try to force the DNS resolver by using : dig @8.8.4.4 github.com
 
@hidigoudi Still a timeout, yes. My containers are unable to find both dig and host but nslookup returns the following: ;; connection timed out; no servers could be reached. Still, even if I were to force the DNS, it wouldn't matter because the very request is blocked, is it not?
 
Yes you still have a firewall blocking. If you disabled the firewall, something else in the network blocks DNS requests. You should open UDP port 53 (you can also open TCP port 53 but DNS protocol uses UDP by default).
A good test would be to run nslookup github.com 8.8.4.4 in another machine in the same network as your container. - If the nslookup can resolve the name, the issue is probably in your container, if it's the same behavior (a timeout), then it's probably a network issue.
 
8:17 PM
@hidigoudi I wish I could. When I try to do so in the router, I get the following warning: The following reserved ports cannot be forwarded because of safety reasons: (53,135,137,139,161,162). Would putting this one in bridge mode and using an external router allow me to bypass this restriction?
 
This warning is not related to your problem. It's about port forwarding, so the NAT in your internal network (when traffic comes from the internet). This is probably a firmware restriction from your ISP. - Your problem is located on the outgoing traffic (inside the firewall). - Have you tried to put the default gateway (the router IP) instead of 8.8.8.8 and 8.8.4.4 for the DNS resolver inside the container ? - Normally the router itself is a DNS resolver.
 
@hidigoudi Thanks for the explanation. So not related to the ISP, gotcha. So if I understand you correctly, you want me to add the default gateway (in my case 192.168.2.1) to my container? In this case I added it to /etc/resolv.conf and it did change the errors, albeit slightly: context deadline exceeded (Client.Timeout exceeded while awaiting headers)
 
Thanks for testing, could you please provide me your /etc/resolv.conf
 
Of course. In this case I lazily added the ip without altering the rest of the file

nameserver 192.168.2.1
nameserver 127.0.0.11
options ndots:0
 
Ok try to remove the line : nameserver 127.0.0.11
And test again the nslookup
 
8:23 PM
Oh wow that instantly worked
 
Yes ^^
 
Server: 192.168.2.1
Address: 192.168.2.1:53

Non-authoritative answer:
Name: github.com
Address: 140.82.121.4

Non-authoritative answer:
 
In your first question on SuperUser
Not this one
You sent a config file for your container DNS
You need to remove 8.8.8.8 and 8.8.4.4 and only put 192.168.2.1
Inside this file : /etc/docker/daemon.json
 
I will do that
 
Then you can restore your `/etc/resolv.conf` like that :
nameserver 127.0.0.11
options ndots:0

Without 192.168.2.1
Let me know
 
8:27 PM
While this will probably work, and I cannot thank you enough that it will, I have to ask; why is this necessary? What broke so dramatically on my side?
 
If it works, I will add an answer to your question
 
Yes, on it
 
yes
I will explain you
Let me 2 minutes
I will also explain in my answer no worry
 
Great :)
 
By the way, some ISP blocks DNS query to the external DNS resolvers like Google (8.8.8.8, 8.8.4.4), why ? To prevent their clients use another DNS resolver than their. It is also used to block some connection to pronography website or others malware.
You cannot bypass this behavior, it is directly integrated into the router firmware
It's for this reason you can't use Google DNS but you can use the router DNS resolver
Because the router DNS resolver is managed by the ISP and they have the full control on it.
 
8:33 PM
I see. And them forcing me to use their DNS is beneficial to them as it allows them to harvest my browsing data, I assume? Would putting their router in bridge mode and connecting an external router have also resolved this?
It does make sense in my head but perhaps bridge mode doesn't sideline every single thing in that router
 
Yes it can solved by configured the router in bridge mode BUT it's more of a waste of time than anything else. Whatever happens, if it's not your ISP that collects your information, it will be Google or another company.
It does not really matter
But I can't stop from making your choice
By the way, your traffic will pass through the ISP router.
(and network too)
 
I think on my side that you can stay like this
 
After changing the daemon file (actually the /var/packages/ContainerManager/etc/dockerd.jsonfile), I still get timeouts, weirdly enough
and restarting the docker engine
etc/resolv.conf is back to normal after a restart
 
Ok weird, can you try to get error ?
Do you see trace of 8.8.8.8 or 8.8.4.4 ?
 
8:42 PM
The exact content of this file is now {"data-root":"/var/packages/ContainerManager/var/docker","dns":["192.168.2.1"],"log-driver":"db","registry-mirrors":[],"storage-driver":"btrfs"} and the error is the old one again Get "https://acme.zerossl.com/v2/DV90": dial tcp: lookup acme.zerossl.com on 127.0.0.11:53: read udp 127.0.0.1:51113->127.0.0.11:53: i/o timeout
 
What do you have inside : /etc/dnsmasq-dns.conf ?
 
This file neither exists in my container nor on my host machine
 
Ok let me 2 minutes
Do you have dnsmasq service that runs on container ?
 
Inside a separate container? No I do not
 
No I mean inside the container with the timeout ?
systemctl status dnsmasq
 
8:48 PM
My container is alpine linux which either doesn't have systemctl (could that be) or its not on PATH
 
Oh OK
Check in this file : /etc/nsd/nsd.conf
 
Also does not seem to exist
 
Weird, do you know what is the DNS resolver inside your container ?
 
I do not have that info unfortunately. I will see if I can find that information
 
Apparently it's the docker-compose DNS
Do you have a docker-compose.yml file ?
For this container
 
8:53 PM
Not for this particular container, I think I set it up manually, but I do for all the others
 
Ah ok
So in your /etc/resolv.conf :
nameserver 192.168.2.1
options ndots:0
And it's good
 
Gotcha. I will see if the logs change. I could always convert it into a docker-compose file and manually add the dns flag
 
Oh just wait
Do you have a file : /etc/systemd/resolved.conf
 
checking
also not there..
This is the entire list in etc/
 
Ok so you can let like I mentionned
 
8:58 PM
alpine-release fstab mailcap nsswitch.conf resolv.conf sysctl.conf
apk group mime.types opt secfixes.d sysctl.d
busybox-paths.d hostname modprobe.d os-release securetty udhcpd.conf
ca-certificates hosts modules passwd services
ca-certificates.conf init.d modules-load.d periodic shadow
Caddy seems to still not like it completely
context deadline exceeded (Client.Timeout exceeded while awaiting headers)
 
Can you send me your resolv.conf ?
 
/etc # cat resolv.conf
nameserver 192.168.2.1
options ndots:0
 
Maybe try to restart
 
Won't that reset the contents of the file?
 
It shouldn't
We can test
 
9:01 PM
Restarting now
 
Ok
Let me know
 
It did reset it indeed. The timeout errors are back. I also set the default gateway inside another container, but that one does it like it either. I will update it docker compose file to see if that does anything
 
Ok I see
Maybe it does not like the modification of /etc/docker/daemon.json
You can try to reset everything as before
And do the modification we did for that works
 
Only do the direct manipulation of the resolv.conf, right?
 
Yes
I'm clearly not an expert of Docker in general, I wanted to help you on your DNS resolver issue initially ;)
Unfortunately I never used Alpine distribution...
 
9:08 PM
You already helped me way more than I helped myself the past two days, I appreciate it regardless
 
We will solved the issue no worry
 
I hope so! Im going to try one more container as the previous one is a bit trickier
 
Ok let me know
By the way we know the reason of the initial issue, it's a good point
You need to use 192.168.2.1 for DNS resolver
 
Indeed. Its now just a matter on why docker does not seem to take it
 
Yes
 
9:15 PM
So I removed everything from the daemon file. I manipulated a docker compose file to include dns: - 192.168.2.11 and restarted that container. While I don't have timeouts in that log, I do not get a response in the web interface as I expect (its Radarr)
 
You mean 192.168.2.1 ?
 
Yes, its correctly in the file luckily
I just said it wrong
 
Ok if it's an Alpine distro
We really need to know what is the internal DNS resolver
It's the key to find the solution
What do you have inside :
/etc/network/interfaces
 
The only files inside network are
if-down.d if-post-down.d if-post-up.d if-pre-down.d if-pre-up.d if-up.d
This includes folders
 
Inside :
/etc/udhcpd.conf
 
9:20 PM
A ton of things
 
Something relative to DNS ?
 
Sample udhcpd configuration file (/etc/udhcpd.conf)
# Values shown are defaults

# The start and end of the IP lease block
start 192.168.0.20
end 192.168.0.254

# The interface that udhcpd will use
interface eth0

# The maximum number of leases (includes addresses reserved
# by OFFER's, DECLINE's, and ARP conflicts). Will be corrected
# if it's bigger than IP lease block, but it ok to make it
# smaller than lease block.
#max_leases 254

# The amount of time that an IP will be reserved (leased to nobody)
 
I have a number of network for each of these containers
 
Containers use the same DNS servers as the host by default, but you can override this with --dns
 
9:24 PM
Yes so for my test container I overrode it with the dns flag
to 192.168.2.1
 
Ok and you have error in logs ?
surprisingly, we managed to make it work just now
 
So this test container doesn't show proper logs like timeout X or Y but it throws java logs, I think
 
There must be something different
 
I could destroy all networks and start over with those, if the problem could be in there
 
Do you have exactly the same configuration as when it worked earlier?
Yes you can try to reset networks
And only mention 192.168.2.1 for DNS
 
9:27 PM
Only the nslookup worked, but even then I think the logs of the first container still showed issues
I will try so tomorrow, its getting quite late here
Not to be ungrateful, you helped me a ton already
 
I did my best !
I will write an answer by the way
 
And I appreciate every second of it
 
And answer to the question
 
That sounds great
I will thinker so more with it and follow up if I do get it completely fixed
 
You can send me message here tomorrow if needed
 
9:29 PM
Tomorrow if going to be a long day but if I do find some time I definitely will
Again, thank you so much
 
Yes no problem
When you want
No problem !
With pleasure
 
Noted. Have a great evening and catch you later :)
 
The only problem now is the DNS management inside Docker
You too !
See you
 

  last day (16 days later) »