[SOLVED]

... well, this is in many ways very unsatisfying, because I have no idea why this worked, but I seem to have fixed it.

Server A has two Ethernet ports, an Intel I219V and a Killer E3100. Several months ago, when trying to debug sporadic btrfs errors (I had my RAM installed incorrectly!), I had disabled some unused devices in BIOS, including the Killer Ethernet port.

Since I had no other ideas, and it seemed like this was somehow specific to this server, I just re-enabled the Killer port and switched the Ethernet cable to that port. I'm now getting 300 Mb/s transfers from my wireless devices to my server, exactly as expected.

I'm gonna like... go for a walk or something. Thank you so much to everyone who helped me rule out all of the very many things this could have been! I love this place, you all are so kind and supportive.

Original:

I'm trying to debug a perplexing networking situation, and I could use some guidance if anyone has any.

Here's my setup:

• UniFi Security Gateway
• UniFi Switch Lite
• Two UAPs
• Two servers, A and B, connected to the USW-Lite with GbE
• Many wireless devices, connected to the UAPs

Here's what I'm experiencing:

• Network transfers from the wireless devices to server A (as measured by iperf3 tests) are very slow. Consistently between 10 and 20 Mb/s.
• Network transfers from server A to all devices are expected speeds. 900-1000 Mb/s to server B, 350-ish Mb/s to wireless devices.
• Network transfers between server B and all devices (in both directions!) are expected speeds.
• Network transfers from the USG to server A also seem slow, which is odd. Only about 60 MB/s.
• Network transfers from the USG to server B and the wireless devices is about 300 MB/s

So, specifically network transfers from any wireless device to server A are slow, and no other connections have any issues that I can see.

Some potentially relevant details:

• Server A is running Unraid
• Server B is running Ubuntu
• Wireless devices include a Fedora laptop, an iPhone, and a Macbook Pro
• UniFi configuration is pretty straightforward. I have a few ports forwarded, a guest WiFi network (that none of these devices are on), a single default VLAN, and two simple "Allow LAN" firewall rules for Wireguard on the USG. No other firewall or routing config that I'm aware of.

If anyone has any thoughts at all on how to continue debugging, I would be immensely grateful! I suppose the next step would be to try to determine whether it's the networking equipment or the server itself that is responsible for the throttling, but I'm not sure how best to do that.

I’d like your help – and your stories!

I’m working on an article with a tentative title of “Tales of the Crypt: Horror stories (about your past) where database performance caused a real problem.” It’s meant to be schadenfreude nostalgia, about your late nights coping with a performance issue (with, hopefully, a happy ending of “…and this is what we did to fix it”).

So, what happened? Tell me about it.

I do want to quote you, but we can be oblique about the attribution – especially because sometimes these stories are from a previous employer and do not represent any current affiliation. But I do want the verisimilitude that demonstrates that these tales-of-woe come from real people. As a result, I’m fine with writing, “Kim’s first job was as a mainframe programmer at a hotel chain, where database transactions required tape changes. ‘Yada yada story,’ says Kim, who now is CIO of a Midwest insurance firm.” Real person, but you don’t need to worry about getting anyone to approve your words. (Though if you’re happy with full name, company, and role, I’m even happier; send in a private message if you prefer.)

I used an ancient example above, but I’m hoping for more recent database performance stories. Ideally some of the “here’s how we fixed it” become practical suggestions for developers who are enduring such a situation today.