I wanted some help with a homelab server I am in the beginning stages of designing. I am looking for a flexible and scalable media and cloud system for home use, and I thought this community would...
I wanted some help with a homelab server I am in the beginning stages of designing. I am looking for a flexible and scalable media and cloud system for home use, and I thought this community would be a good place to source feedback and recommendations before taking any real next steps! I really want to check that I am approaching the architecture correctly and not making any bad assumptions. I am open to all feedback, so please let me know what you think!
I already run a simple home server and I have typical homelab FOSS apps, such as jellyfin, navidrome and audiobookshelf, but I am also interested in migrating away from cloud storage using nextcloud, immich, etc. In an ideal world, this setup would also allow me to leave windows on my main machine and use a windows vm for business related work that can’t be done on Linux. I will likely be the one primarily using the services, however I could expect up to 10 - 20 users eventually.
High level setup is with two machines:
- Proxmox Server
- TrueNAS Scale server
- JBOD with either 90 bay or 45 bay storage
- 10G switch
This might be a stupid setup right off the bat, which is why I wanted to discuss it with you all! I have read a ton about using TrueNAS as a WM within Proxmox, but I just like the idea of different machines handling different tasks. The idea here would be to set up the TrueNAS server so it can be optimized for managing the storage pool to allow for easy growth. While the Proxmox server can handle all the VMs and connecting users, with higher IO, etc.
TrueNAS System Specs:
- AMD ryzen CPU and motherboard
- 64 or 128GB ram
- Mirror 500GB M.2 NVMe OS Drives
- GPU if necessary, but hopefully not needed
- Dual 10gb pcie card if the motherboard doesnt already come with them
- An hba for the JBOD something like the LSI SAS 9305-16e
- SLOG and L2ARC as necessary?
JBOD enclosure
- While I am interested in a 90-bay enclosure, I would only realistically be starting with two vdevs which is why I think a 45 bay enclosure wouldn’t be an issue.
- Im tentatively planning for an 11 wide Raidz2 vdev configuration. This would hopefully scale to 8 vdevs with 2 hot spares or 4 vdevs with 1 hot spare.
- All drives would be HDDs
Proxmox Server Specs:
I am less familiar with the specs I will need for a good Proxmox server, but here is what I am thinking.
- AMD epyc and motherboard if I can get my hands on a less expensive one. Otherwise I was thinking a higher end AMD ryzen cpu
- 128 or 256GB ram
- Mirror 500GB M.2 NVMe OS Drives
- Somewhere between 2 and 8 TBs of SSD storage. Depending on the number of drives, I think this would be a single drive, mirror or raidz1.
- This storage will be used for all the vm configuration and storage, except for something like Nextcloud where the main storage will go onto the TrueNAS mount.
- I would also use this for temporal storage such as downloading a file before transferring it to the TrueNAS mount.
- A dedicated GPU primarily for transcoding media streams, but also for testing and experimenting with different AI models.
- Dual 10gb pcie card
Questions:
- I know Proxmox can do zfs right out of the box so I know I don’t need the TrueNAS server, but splitting it this way just seems more flexible. Is this a realistic setup or would it just be better to let Proxmox do everything?
- Does anyone have experience creating NFS shares in TrueNAS for mounting in Proxmox? I would be interested in thoughts on performance, and stability among any other insights.
- Do any of the system specs I listed seem out of line? Where and how do you think things should be scaled up or down?
- If I ever did expand to a second JBOD shelf, assuming the first one was full first, is it be possible to create new vdevs that spanned across the shelfs without losing data?
- Is SLOG and/or L2ARC necessary for this setup? What capacity and configuration would be best?
- What else have I missed?
Lastly, a quick blurb:
I have been building PCs for a while and undertook building a home server a few years ago. I loved the experience of learning Linux (the server is running Ubuntu), picking up docker, and learning more about the FOSS community has been a joy! Part of this project is to learn along the way but also have a setup that I can build towards over time! Proxmox, TrueNAS and zfs would all be new to me so I really see it as an opportunity to explore. I want a solid media and cloud server setup, while also giving myself the freedom to explore new operating systems and general hypervisor functionality.