Discussion between adamstrawson and Michael - sqlbot

11:58 AM

Q: Route53 Multivalue answer persisting single record

We use Route53 for our private DNS, specifically the Multivalue Answer Routing Policy, with no Health Checks for dump/lazy load balancing. This has worked well for us, up until yesterday where we're noticing odd behaviour. We have a record set for ro.mysql.foo.bar.com, which contains the IPs of...

Michael - sqlbot

If you dig @ns-xxxx.awsdns-yy.zzz your-record.example.com where ns-xxxx.awsdns-yy.zzz is one of the name servers assigned to your hosted zone by Route 53, does the returned record ordering vary? Or, if this is a private hosted zone, the NS records are possibly dummy values, so try dig @169.254.169.253 your-record.example.com from inside the VPC.

adamstrawson

@Michael-sqlbot Yes, the ordering varies. We've tested this on the AWS nameservers and our own, both show the same behaviour.

Michael - sqlbot

So dig varies. Does ping your-record.example.com repeatedly try to ping only one host, or does that also vary, but only the mysql client doesn't?

adamstrawson

@Michael-sqlbot ping shows the same behaviour too, returns the single IP and does not vary

Michael - sqlbot

Initial impression, this sounds like it could be the work of my old nemesis, nscd.

adamstrawson

11:58 AM

@Michael-sqlbot I read similar things too, but we don't have nscd installed on our servers

Michael - sqlbot

And this has been working all along, but suddenly changed?

adamstrawson

@Michael-sqlbot Yep, up until around 10:30AM yesterday

Seems to be across our entire infrastructure (but varies on which IP sticks), including some servers that haven't changed/restarted in a number of months

Feels like an AWS thing, as i've just tested outside of our infrastructure and the same results happen

Sorry, scrap what I said about ping, that does not seem to show the issue

I didn't run the test long enough

Michael - sqlbot

12:20 PM

That's confusing. If the responses from dig are varying, there's nothing that can easily be pinpointed to say how or why Route 53 has a role in the behavior you observe.

adamstrawson

12:32 PM

I think it's our PowerDNS resolver

I've changed that to point to Googles DNS as a test and it's behaving as it should..

considering that has been running for 2 years without an issue, unsure why it's just started to cause us a problem

last day (15 days later) »

all rooms