Anyone have advice (or horror stories) on setting up a 100GbE NAS with RDMA / SMB Direct?

Greg
2025-09-03T12:50:34Z

Pretty much what the title says - I'm building out a smallish compute cluster and hoping to set up some centralised storage that won't be a bottleneck, but I'm very much not a networking specialist. Most of the load will be random reads from compute nodes pulling in the bits of various datasets they need to work on.

Is it plausible to throw a 100GbE ConnectX-5 card and 256GB RAM into a consumer AM5 machine, format everything in ZFS, and set up a network share with KSMBD? My understanding is that I want to ensure everything's using mirroring rather than worrying about RAIDZ parity if I'm optimising for speed, which is fine, and I know that I'll only get full throughput as far as things can be cached in RAM - but is it reasonable to expect ZFS ARC to do that caching for me? Dare I hope that the SMB driver will just work if I drop it in there between the filesystem and the NIC? Or have I crossed the line into enterprisey-enough requirements that it's going to be an uphill battle to get this working anywhere near line speed?

Comments URL: https://tildes.net/~comp/1pza/anyone_have_advice_or_horror_stories_on_setting_up_a_100gbe_nas_with_rdma_smb_direct

Votes: 15

Comments: 3