Looking for feedback on a homelab design

This turned out way longer than I intended, but I hope it's useful! There are a ton more details and nuances floating around in my head, so absolutely feel free to ask if there are specifics you want to discuss more.

I've had to do a kind of absurd deep dive into ZFS and the hardware/OS config/general NAS setup to run it recently, so happy to get as far into the details as you'd like! Something I've really found when searching is that a lot of the accepted wisdom is quite dated, even when it's being repeated in newer discussions - I've consciously gone against that in a few places, so if something doesn't seem to add up just ask, 50/50 whether it's deliberate or whether I've just made a mistake.

JBOD with either 90 bay or 45 bay storage

Roughly how much data are you expecting to store? This seems like a lot, especially when 100TB usable is five or six 28TB drives (the currently optimal price-per-TB). You might want to go 16TB or so just to give yourself a bit more granularity for VDEV configuration, but even then you're pushing 200TB usable space in a 4U case alongside the NAS server itself, no JBOD needed.

For what it's worth, you want to be buying the recertified with warranty Exos drives - redundancy is one of the major purposes of a ZFS setup, and no individual user is buying enough drives to be a statistically significant sample anyway, so as long as you're covered against DOA and early failure it's really not worth worrying about the difference between new and refurb/recertified. I've even seen some people suggesting that recertified have a lower failure rate due to the extra rounds of testing, although I haven't seen enough data to corroborate that.

Im tentatively planning for an 11 wide Raidz2 vdev configuration. This would hopefully scale to 8 vdevs with 2 hot spares or 4 vdevs with 1 hot spare.

If you're thinking of more drives for better throughput, I'm going to make a potentially controversial point and say that (within reason) you shouldn't be worrying about the HDD speed on a modern NAS. SSD and RAM caching are multiple orders of magnitude faster, and cheap enough that it's just not worth trying to optimise the bits of the system that are always going to be limited by a physically moving arm and platter.

AMD ryzen CPU and motherboard
64 or 128GB ram

ZFS loves RAM, so more tends to be better, but you can always start at 64GB and see if you actually hit any bottlenecks from cache misses first.

You'll probably want to run the memory at lower than its rated speed, too. This isn't a problem - you will absolutely never notice a difference in memory speed when serving files unless you're also running $10k in networking hardware - but Ryzen only has two memory channels, so four DIMMs is running two per channel (unless you're planning on using 64GB DIMMs, but last I checked they were way too expensive compared to 32GB) and they tend to get unstable if you don't knock the speeds down a bit.

You'll also want to consider the PCIe layout of your motherboard, and especially look out for one with bifurcation support, because that'll let you drop in a whole lot more NVMe drives as and when you need them.

AMD epyc and motherboard if I can get my hands on a less expensive one

There are really good bundle deals on eBay at the moment for a Supermicro H11SSL-i or H12SSL-i along with RAM and an Epyc Rome (7002 series) or Milan (7003 series) CPU. I bought one a month or so ago and it's been great so far - I'd say it's worth paying a little extra to go for Milan or newer, Rome is getting pretty dated at this point. You can also save a little by getting a Gigabyte board rather than Supermicro, but everything I could find said they're incredibly picky about PCIe devices, whereas the Supermicro ones just work (which I can attest to). Shipped from China, which is the only thing that might be an issue if you're in the US.

I know Proxmox can do zfs right out of the box so I know I don’t need the TrueNAS server, but splitting it this way just seems more flexible. Is this a realistic setup or would it just be better to let Proxmox do everything?

Separate NAS is always a good idea in my opinion. Proper separation of concerns and the ability to add/repurpose more machines later is more than worthwhile for a setup like this.

You could always run Proxmox on the NAS box as well, just with a single TrueNAS VM inside it, if you want the extra layer of abstraction for system backups etc. - I'd consider this if using TrueNAS, but I'm using NixOS so I'm less worried about losing system installs or OS config. If I were going down that route I'd still let TrueNAS handle the ZFS side, rather than giving it to Proxmox. But honestly as long as you have config backups for TrueNAS I'm probably overengineering things by even suggesting virtualising it.

Does anyone have experience creating NFS shares in TrueNAS for mounting in Proxmox? I would be interested in thoughts on performance, and stability among any other insights.

If you're sharing sensibly sized files (few megabytes to tens of gigabytes) I'd expect this to be fine - large numbers of much smaller files are where you start getting problems. I've found that NFS pretty much just works, and has decent throughput, but does add enough latency that you'll notice it when you need to do something like recursively list a directory tree with a few tens of thousands of files and subdirectories.

Is SLOG and/or L2ARC necessary for this setup? What capacity and configuration would be best?

L2ARC: yes. You don't want to be waiting for hard drives if you can reasonably avoid it, and you definitely don't want to be waiting for hard drives to look up metadata. Ideally you'll be hitting ARC rather than L2ARC more often, but even 64-128GB in comparison to HDD sizes means that things definitely will be pushed out relatively frequently, and 2TB NVMe really doesn't add much cost in the context of the hardware you're talking about here. If you find cache misses are a bottleneck once you get a feel for things, it's easy to add more NVMe drives later as long as you have the PCIe lanes for it.

You'll see a lot of people online saying SLOG is unnecessary because it's only required for sync writes (async writes are acknowledged as soon as they're cached in RAM, and are flushed to persistent storage every 5 seconds or less by default, whereas sync writes are only acknowledged after they're in persistent storage). What they don't tend to mention is that NFS defaults to using sync writes for everything!

Your options here are:

• Pass the async flag when mounting the NFS shares. In the worst case this can cause data loss in case of network disconnection, power outage, or system failure, because NFS on the client system will confirm writes before they've been written by the server.
• Mount the NFS shares synchronously, but force ZFS to write them asynchrounously. This removes the network as a failure point, because NFS waits for the NAS to confirm writes, but allows ZFS to send that confirmation while they're still cached in RAM on the server. Worst case risk is data loss of up to the last 5 seconds of writes in the case of a power outage or system failure on the server, but not the network. Bear in mind that "risky" here is by ZFS standards, which are still more reliable than most filesystems' defaults.
• Keep everything running synchronously without a SLOG drive. I wouldn't do this - sync writes on spinning disks take a comparative eternity, and it also leads to the drives constantly thrashing and creating noise and vibration - but you can always test it out if you want. If you have a known workload with large files and correctly tuned block sizes it might just about be acceptable. Maybe. This is safe against data loss in all circumstances, with writes only being acknowledged after they're in persistent storage.
• Keep everything running synchronously, use a SLOG drive. Same safety guarantees as the previous point, but much faster because you're caching those writes to an SSD - persistent, unlike RAM, and responsive, unlike HDDs.

But there is a major caveat on the last point: consumer SSDs don't actually write things persistently in the way most people expect. The faster ones have an internal DRAM cache, which is itself volatile before the drive controller can flush it to the NAND. For most consumer-grade drives, this means the latency on SLOG writes is a lot higher than you'd expect it to be, because ZFS is waiting for the NAND write rather than the DRAM write: still a decent amount faster than a HDD, but nothing like the snappiness you get from a "normal" filesystem that just assumes the drive's DRAM cache is safe enough to consider successfully written. For some drives, the controller will falsely report to ZFS that data was written when it's actually in the cache: this is much worse, because ZFS can deal with known data loss or corruption, but isn't built with an expectation of drives giving incorrect information. A power loss on the latter type of drive could theoretically leave ZFS itself in an inconsistent state.

If you're going for a SLOG drive, make sure it's an enterprise grade one with PLP (power loss protection) - those can correctly acknowledge writes even when they're in the DRAM cache, because they have an onboard capacitor that will allow the controller to flush the DRAM to NAND even in the case of total power loss, and without any interaction from the filesystem or OS. You'll see a lot of discussion about Optane drives, which were basically this setup with an unusually large (for the time) cache, but those haven't been in mainstream production for years and are now even hitting the end of extended enterprise support. Modern drives have large enough caches and fast enough flash that I don't think it's worth considering Optane, although if another manufacturer picked them up and made a modernised and supported version I'd be interested!

Micron and Kingston both make M.2 2280 NVMe drives with PLP, or you can look at used 2.5" U.2 enterprise drives and get an adapter to connect one to a PCIe or M.2 slot. Either way, you'll only need a few tens of GB for a SLOG drive, so pretty much anything on the market will be fine in terms of size - since SLOG is inherently a write-heavy workload, you can consider the extra space to be overprovisioning for wear levelling.