@MikeyB MF won't fess up if it's a BSD based appliance, but nmap -O says it's 100% probability of being Linux and support replied that -4 isn't a valid flag for the appliance's telnet.
@Wesley well maybe the telnet on that box is preferring the IPv4-compatible IPv6 address corresponding to the hit-nxdomain record. Same solution. Stop using OpenDNS.
@ewwhite are you strangely proud of being from Evanston (warts and all) too? Evanstonians seem to be the proudest bunch of people for one reason or another
@MikeyB I'm very tempted to rip the hard drive out of this effing thing, clone it, and twiddle it in a virtual machine just to figure out what the hell this thing is based on and why it sucks so hard.
@pk. yes, extremely proud. It was a unique melting pot... racially and economically and politically diverse... good culture and history... Wonderful university.
@JoelESalas Explain to me why it's not OK to try IPv6 first and fail back to IPv4? If it was handed a valid IPv6 address, tried to connect and failed, then fell back to IPv4, OK.
@MikeyB It's basing security / authentication on a DNS record. So it's failing on updating because "oh noes the DNS record isn't what I expect it to be" and then says "Invalid machine indentification. Please contact support at ##"
@pk. well, that's historical. Evanston's black community was insular. More like a rural small town than anything else. Being black and not originally from Evanston... I was excluded from the town's Black community. "Oh, you're not Ray-Ray's cousin!?!"
@ScottPack It does: Pinging star.c10r.facebook.com [2a03:2880:2050:1f08:face:b00c:0:1] with 32 bytes of data: Reply from 2a03:2880:2050:1f08:face:b00c:0:1: time=98ms
@ewwhite you sure? I've met perfectly pleasant pepole in small town that won't have a thing to do with you after an initial introduction once they've figured out that you're not related to anyone they need to stay on the good side of.
@Magellan Or the south. Where I grew up meeting a new person resulted in two questions: "Who is your family?" "What church do you go to?". Frequently that data was requested even before your name.
@JoelESalas Yeah, the first question Kentuckians wouldn't ask you would be about your family or church, it would be "Are you done with my hedges already?"
@Iain Meaning your primary identity, and trust level, is dependent on which Christian sect you belong to and whether or not one of your cousins ever failed to say "Hello" to them.
@MikeyB I had to ping OpenDNS support the other day because a .tk domain for a client wasn't resolving at all. On any server. I checked zones and everything. Every other DNS service I tested against resolved it fine. Client went silent for a while, I waited... still no resolution after a week. Finally got OpenDNS to "flush caches" or whatever they said they did and then they responded with zone records, but still.
I have a RHEL system set up as an iSCSI target with 2 luns shared out via one target via the following configuration in /etc/tgt/targets.conf (which I built following the commented out examples in /etc/tgt/targets.conf):
<target iqn.2014-04.com.local.box:ESXi.target1>
backing-store /dev/...
@MikeyB So apparently at least one application on the appliance is requesting a AAAA record and the tech support was either unaware of it, or didn't seem to think there was a correlation. Instead, no, there must be some kind of packet tampering and ipv6 tunneling going on!
Hex is scaaaaary!
But I have the packet capture so they can eat it sideways.
Hexspeak, like leetspeak, is a novelty form of variant English spelling using the hexadecimal numbers. Created by programmers who wanted a magic number, hexspeak words can serve as a clear and unique identifier with which to mark memory or data.
Using hexadecimal notation, which includes the digits 0123456789ABCDEF, it is possible to spell several words. Further words can be made by treating some of the decimal numbers as letters - the digit "0" can represent the letter "O", and "1" can represent the letters "I" or "L". Less commonly, "5" can represent "S", "7" represent "T", "12" represen...
@MikeyB Right, I know, but it's requesting a v6 address. So imagine the grimace on my face as I was trying to make them understand "No IPv6 is in this equation. It's simply a DNS query, over v4, that's returning a DNS AAAA response that in essence simply has a text payload representing a AAAA IP address but there's no IPv6 on the wire you filthy animals."
@MikeyB What I fault them for is thinking that v6 is the problem when it obviously wasn't v6, it was their app failing on a AAAA record because they use it as some form of authentication, and they didn't force their app to query for A records. They assumed the default of a non-responsive AAAA record rather than explicitly defining their logic.
Having said that, yes OpenDNS sucks big wads and shouldn't tamper with zones
Summary
Risks of using LVM:
Vulnerable to write caching in kernel and hard drive/SSD on kernels before 2.6.33, or in VM guests
Harder to recover data due to more complex on-disk structures
Harder to resize filesystems correctly
Snapshots are hard to use, slow and buggy
Requires some skill to c...
@Wesley OK, fair enough. But still, expecting for them to define records on one side and get the same records on the other is not unreasonable. Oh, and that's not a "default of a non-responsive AAAA records". It specifically didn't exist until OpenDNS added it.
@MikeyB The assumption is that "Certainly our query will return a single-A record" when there is apparently an initial request for a AAAA record that they're not aware of.
@MikeyB Maybe I'm not terming it right. They don't know that their application is asking for a AAAA record first. So when a AAAA gets returned, the app doesn't handle it right.
At least that's what I'm understanding from their support emails. They assume that any AAAA response is because my firewall is tunneling something somewhere. They don't think that their app is asking for a AAAA record. Now certainly that's not a problem if the DNS provider doesn't insert shit into their zone on the fly
@Wesley Yeah that's just failure on their part to understand IPv6 (which is kind of sort of reasonable given they haven't yet implemented it) but throwing up their arms about it is kind of weak.
@MikeyB Correct. They get a response and connect to the IP address. I fault the programmatic logic that even asks for an IPv6 AAAA record. I wouldn't deal with anything dual stack unless I actually had the infrastructure to support v6.
But yes, OpenDNS is 99.9999999% at fault here.
I'm just all ragey that my systems were called into question because someone else shat themselves over seeing a IPv6 address. =P
@DennisKaarsemaker Oh and tell me more.
@MikeyB Oh and furthermore, I write detailed observations on the problem, and even include a case summary, and I get no response. I'm less inclined to cut them slack because they're just not nice people. Even though in this case the original problem isn't their fault, necessarily.
@Wesley "Vulnerable to write caching in kernel and hard drive/SSD on kernels before 2.6.33, or in VM guests" - no worse than without lvm. "Harder to recover data due to more complex on-disk structures" - ahbullshit. And that's what backups are for. "Harder to resize filesystems correctly" translation: I can't read manpages, mommy help!
"Snapshots are hard to use, slow and buggy" - I find them easy to use, but I'll grant that they're not the fastest. Haven't seen bugs yet and we use them a LOT "Requires some skill to configure correctly given these issues" - well yes, that's what your boss is paying you for goddammit
We use lvm on about 8000 servers and have yet to have our first problem caused by it.
@MikeyB Well, my guess is the app is using the OS's name resolution, which by default is sending both the A and AAAA queries in the same transaction. The fact that OpenDNS is adding crap into the response that doesn't work, I blame OpenDNS and not the software vendor (even though they seem to not understand what the OS they built on is doing).
@DennisKaarsemaker Lawl. Yeah the on disk complexity seems to only be an issue for filesystem designers or forensic examiners. Just take backups, you shouldn't be parsing a filesystem's raw datastructures unless you're part of a law enforcement investigation, and if you are you'll have mega expensive tools and consultants to work with.
@DennisKaarsemaker And just because something is slow doesn't mean it's bad. It's just a different tool with a different use-case.
@DennisKaarsemaker And when is requiring skill to configure something considered a negative? "Just give me the clicky clickies!"
@ShaneMadden Yes, that's a good summary. It's a dual stack OS apparently and everything would work fine if OpenDNS didn't muck with things. I just got all panty-twisted over them not liking seeing a v6 address and assuming somehow v6 was to blame.
@DennisKaarsemaker Write a better answer. I'll give you a bounty.
@kce Check if there's big spikes in logs in the event log. My guess is something reading the event log - is there a monitoring system or similar that would be doing that?
@MikeyB I have a similar story that I haven't posted on my blog yet. Not quite to the level of backdoory-ness, nit there's a CenturyLink router made by Zyxel that will hijack any traffic to any IP address that it routes for on port 4567 and 51080.
@MikeyB So for example, I had a client with a /29. If you went to the DSL modem/router's IP address:80 you got the remote admin page. Then if you turned off remote administration, but went to IP:4567 you'd get the HTTP (not s) login for the... remote admin page. Then if you went to any one of the other /29 addresseseven if it was an address that wasn't assigned to any host over port 4567, you'd get that same HTTP admin page.
@MikeyB It took six weeks of emailing CenturyLink before, finally, I got a phone number to call and the answer was "Nothing can be done about it for that modem. You need a PK5001 model." and I had to have a tech come out and replace the modem.
@ChrisS The port scan on the thing showed about ten different PCI-failing vulnerabilities on the two ports that were forced open and scooping up traffic bound for any one of the /29 addresses.