I replaced my consumer wireless router with a linux box that has a quad-gigabit NIC PCIe card and a single gigabit NIC on the motherboard (for the WAN). After turning on IP forwarding, masquerading (via iptables), and setting up subnets on each of the four LAN interfaces I ran some speed tests.
...
@Pedro - retested from two wired devices both directly attached on different subnets (no layer 2 switch this time on the one subnet). Essentially identical results
@DavidSchwartz Yes, the idea will be to eventually firewall one of the subnets to just WAN access and then firewall one of the others to just allow access to certain ports -- but this point iptables is as bare as possible to reduce variables, and I'm sure I've reloaded it's configuration.
Could you run the iperf3 with --bandwidth at a couple points like 20m, 40m, and 80m and see if those eventually slow down? This will show if you're overflowing a buffer on the router.
Can you please do a full bandwidth iperf from a host on enp3s0f0 to one on enp3s0f1, enp3s0f1/enp4s0f1, and enp3s0f0/enp4s0f1 to see if one of the ports is bad?
On the NC364T, can you just verify that the mac addresses are different for each port? It looks like you're only using two of the ports enp3s0f0 and enp3s0f1, correct?
These results are unexpected. I would expect a buffer overflow to crash after a consistent amount of data has transferred. The more you limit the bandwidth, the more I expect to squeak through before overflowing. However I see 40mb at ~80mbps and only ~14mb at 20mbps.
Your information about duplicate packets makes me suspect the quad NIC isn't behaving exactly like four normal NICs, but doing some deception.
3: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000 link/ether 00:26:55:00:00:01 brd ff:ff:ff:ff:ff:ff 4: enp3s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000 link/ether 00:26:55:00:00:00 brd ff:ff:ff:ff:ff:ff 5: enp4s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN mode DEFAULT group default qlen 1000 link/ether 00:26:55:00:00:03 brd ff:ff:ff:ff:ff:ff
I ran tests between each of the three subnets I have hooked up to each of the others. Similar behavior where it starts strong and then falls over after a few seconds
Doing longer 60 second tests, it looks like it actually oscilates
Still finishing the matrix, but every 12-15 seconds the speed comes back for a second or two
Here's the full matrix of router against all three subnets and then the permutations of the subnets against each other: pastebin.com/HqLTNSWM
Okay, so those four interfaces are the four on your quad NIC. You have another NIC on the motherboard that is presumably connected to your internet modem and doing pppoe. Could you plug in a host on that interface and do an iperf across there? Also fine would be doing an iperf over the uplink if you have a host outside.
With oscillation you can't rule out a buffer problem, but you should get way more data than that through and it shouldn't take 12 seconds of silence to catch up.
Also to clarify I mean iperf from a host on the 4xNIC to a host on the 1xNIC. Don't need any more from the router to hosts, those all appear to be good.
You're correct, I'm using three of the four NICs on the quad-nic card for LANs. The fourth is not connected. The WAN NIC is on the motherboard. I have an EC2 instance that I can quickly test from.
From 10.0.0.0/16 to host on WAN I get 40 Mbps / 5 Mbps as expected
Discussion between Pedro and Huckle
Discussion between Pedro and Huckle