Advice on expanding storage in starter homelab/media server
I've been slowly fiddling around with setting up a little homelab and media server for the last few months. As a web developer, I've always wanted to learn a bit more of the infrastructure side of things, hence the homelab part. The deteriorating quality of major streaming services finally pushed me to set up a media server as well.
Right now, my setup is very basic. I've been using an old repurposed office laptop. It's a simple dell latitude 5540 I got ridiculously cheap due to it's barely usable crusty keyboard, but since I mainly SSH into it that's not really an issue. I formatted it, doubled the ram, and installed the latest stable Debian release. (Headless)
After that, I chose to install yams which was recommended here. Definitely saved a lot of time there! Finally, I added an old unisex raspberry pi I had lying around. The idea is that it's the only part of the setup that is on 24/7, since it has an almost negligible footprint. Whenever I want the main server running, I SSH into the raspberry and use wakeonLAN to start the main server. I'm probably gonna make a tiny web interface for that soon.
Now on to the part I need advice for. The laptop and attached HD are quickly running out of space. I know just slapping on extra hard drives has a limit, and am vaguely aware of things like unraid existing, but am a bit overwhelmed right now with all the information and options in this space.
Does anyone have some advice on something I can tackle for a reasonable amount of work/budget? Something basic, but with the possibility of expansion in the future?
Any other tips on where to go next in general are of course also appreciated. (On that note, I'm right now not opening up the server to ingress from outside. I only interact with it on the home network, as I primarily work from home)
What are you planning on using the server for? File storage? Streaming media? Do you want something that's plug and play or are you willing to build something yourself?
I've been using Unraid for a few years now and have nothing but good things to say about it. In terms of usability, it's the best homelab server OS I've tried. It runs on basically anything, so you can run whatever hardware you want/have/can afford to get. There's also a pretty sizable user base, so there are tons of tutorials out there on how to do just about anything you can imagine.
If you want something premade, Synology makes a bunch of great NAS boxes that you just have to throw some drives in and you're good to go. It has apps for just about everything you can imagine.
I will second the Synology NAS -- been running one for over five years. The only issue I had was when I failed to check for accumulating drive errors and my disks became almost unusable, but I was able to copy things off to an external drive then replace the RAID disks. Now I have the monthly email report configured. We use a mix of dropbox and Google drive on our computers, and I have the NAS configured to download everything from our dropbox and Google drive accounts without deletion. That still has the potential to lose old versions of documents, but for the important media stuff it's fine.
Mostly it's going to be for streaming media, though I'm probably also going to host some hobby projects on it, maybe use it as a game server for friends and other small things. But daily usage is predominantly media streaming.
I do like the IDEA of building something myself, but I've learned throughout the years that I should prioritize a bit more if I don't just want to add to my perpetually growing list of unfinished hobby projects. Do you know if the Synology NAS boxes still allow some tinkering? Unraid sounds really great, but I'm not sure if it's realistic to take that on with my current workload. (I can maybe spend a few hours on this per month, nothing more, alas.)
Once it's up and running, Unraid basically runs itself. Until last week when I restarted to do an update to the OS, my server was up and running for nearly 3 months (when I also restarted it to do an update). It's a pretty resilient OS.
Synology NAS systems are pretty locked down. You can upgrade the RAM and network cards on the consumer models but that's about it. If you want to run anything relatively CPU heavy, then a Synology NAS will be pretty limiting.
I will say that software-wise, DiskStation (the OS that Synology NAS boxes run) and Unraid are extremely similar in terms of adding apps and services. They both mostly rely on Docker to get ready made app templates that you can customize however you want.
I have to disagree a bit. All of that is true if you buy an Atom chipset, but the AMD and Intel versions can run Docker and there are many many things runnable that way. There is also a FOSS package registry that I use at https://synocommunity.com so you don't even need Docker.
Just a comment on the game server idea which is something I also do and really enjoy. In that case fast storage starts to become valuable, so SSD and in particular NVMe storage can really improve server performance. For most modern games, 4GB to 8GB of RAM is enough, but if you intend to host more than half a dozen friends at the same time more RAM is helpful, and the bandwidth (and latency) of your internet connection will start to matter.
While we’re waiting on more experienced Tilders to chime in, I’m honestly interested in the setup you have now.
I would love for you to elaborate on how you configured that setup, and how you get the Pi and laptop talking. I’m looking at options for a home lab as well and found this much more realistic than going all out for a whole new server/NAS.
It actually was surprisingly easy! I thought this would be the hard part, but debugging the Dell network card and getting that to work took way more time.
I wrote some of it down in my notes. This is assuming you have a laptop up and running (Debian) and connected to your network, ssh configured. (The Debian installer actually took care of setting up ssh for me, which was a nice time saver)
First of all, you have to access your BIOS or UEFI settings. In there, you should enable the Wake On LAN setting, and disable Deep Sleep. Both of these might have slightly different names depending on your laptop. Reboot your laptop after applying these settings.
Next, ssh in your laptop and run
ip ato get general networking info. In this output, we're interested in three things: The name of the NIC, it's MAC address, and the broadcast address of your home network. My NIC was calledenp0s31f6, and the MAC address was displayed on the second line of that entry. Next to theinetvalue which has the IP address of this laptop (which should be the one you're using to ssh into), it had the broadcast address asbrd 192.168.1.255.With this information, you can first check if WoL (Wake on LAN) is enabled. Run
ethtool enp0s31f6, substituing my NIC name for yours. You're looking for the value of theWake-onsetting. With mine, it saidWake-on: g, which means WoL is enabled. (You can find the meaning of the various letter options on the ethtool manpage.Now that we verified WoL is enabled, turn of your laptop. Ssh into your raspberry, and install etherwake. (
sudo apt-get install etherwake). Once this is done, you should be able to wake up your laptop withwakeonlan -i <HOME_NETWORK_BROADCAST_ADDR> <NIC_MAC_ADDRESS. So for me that'swakeonlan -i 192.168.1.255 a4:4c:c8:6b:de:9b.Hope that helps you a bit!
This is the kicker which has made me give up getting WoL going on everything...I've had things that wouldn't let me change it. Also Windows does some really funky stuff now that interferes with WoL stuff which has been a major source of irritation.
That said, I had Sleep on LAN going for awhile, and is also some great functionality to add into that setup.
Ugh, there's so much going on with windows, and sudden inexplicable walls it throws at you that just straight up prevent you from using something with no good reason. I'm seriously considering switching full time to Linux, but as long as I want to keep gaming and using Adobe I'm kinda being held hostage by it.
Thanks for the sleep on LAN recommendation though, will check that out.
Multiplayer gaming is the only reason I still have a Windows box. The Steam Deck covers damn near everything else.
I have a similar setup. Now, please take the following with a grain of salt, I do realize this is far from optimal and I'm sure there will be plenty of better alternatives.
That said, what worked for me was attaching a portable 2tb SSD to my raspberry pi. It's more than enough space for what I do and was "set-and-forget", been rocking it for about 2 years.
I will second that. I have a few rpi computers with 2TB hard drives attached. One is a simple Samba drive, the other is the data directory for nextcloud.
Here is my recommendation for the "simple" upgrade
Yeah, turns out I fill 2TB up pretty fast. :D I honestly thought I would last longer with this, but apparently our media consumption (and desire for high quality 4k movies) burns through about 1TB a month. I could of course delete watched content, but uhm, I like to give back to the community, if you catch my drift.
I've got a Pi 4 with two 4TB SSDs attached via USB. Works great for a lot of small homelab stuff -- a jellyfin server, Pihole ad blocker, samba nas, vpn, calibre server, freshrss RSS server, and a couple of other small things. SSDs consume less power than HDDs, so I haven't even needed a powered USB hub for stability. Had it running for 3 or 4 years now with no downtime or file corruption.
This is literally the cheapest and lowest power option I know of. Though when I exceed my current space constraints, I'll probably look into Synology options or at least upgrade to a Pi case with powered NVMe and M2 connectors.
I decided to build a home-lab/media server a couple of years ago as a project during the pandemic and I decided I wanted to build the entire thing from scratch so I went in a slightly different direction than most of the commenters here. I'll try to provide a brief overview and if you have any additional question just let me know!
Intro:
First, this was mostly a learning project for me, I don't have a formal programming or hardware background so it was primarily to challenge myself to learn newer technologies. I was specifically interested in learning linux, docker and networking configurations so instead of using something like UnRaid I just used Ubuntu. I specifically targeted uses cases of streaming media and document storage so I had a pretty good idea what I needed from the hardware. The goal was for the server to always be on so I didn't bother including any type of wake scripting, since it wouldn't be needed.
Hardware:
Roughly here was the build: Micro-atx case and mother board, 16 GiB of ram, 10th gen i5 intel cpu, one Nvme for the OS and four 4TB Ironwolf drives for storage. At the time this cost around $900 for everything.
Software:
I loaded the most recent version of Ubuntu and off I went. I decided that I didn't want to go headless since I still wasn't that comfortable with linux so I went with a standard install. Then I implemented a software raid on the storage drives, I went with raid 5 as it seemed to give the best tradeoff between redundancy and efficient drive use. Since docker was one of the things I was keen on learning all of my services are dockerized, and I manage all of them through Portainer. I use Jellyfin for movies/tv, Navidrome/Symfonium for music and nextcloud for document storage. I was also interested in being able to access my services externally so I also bought a cheap domain and setup a reverse proxy for the external connections. Lastly, I use wireguard to securely access the services that are not exposed externally. I also put in several more containers for vpns and sourcing content, but if you have questions about that then just DM me.
Thoughts:
Overall it was a great project and I have a lot of fun with it. On the 'expand-ability' front I really like this setup, docker makes it very easy to spin up/down new services I want to try without interfering with my main storage or UI applications and the storage can be easily expanded.
Happy to answer any questions you might have!
Something else you might be interested in: you can send the Wake on Lan (wol) packet directly from windows without having to ssh into the raspberry pi, if you want.
On linux, I use the etherwake command. On windows clients, the easiest method I found is this wol.exe (https://www.gammadyne.com/wol.htm) which you invoke from the windows commandline. I just had to put wol.exe in the windows PATH environmental variable, and now I can do things like, "wol.exe xx:xx:xx:xx:xx:xx /d 192.168.1.255" directly instead of sshing into the rpi and doing it there with etherwake.
I also used to ssh into an rpi to send the WOL packet, but this way ended up being a lot easier.
Oh, right! That would normally be true and was indeed the first thing I was going for. But it turns out that my router actually blocks magic packets if they're coming from WiFi, and only allows them from LAN. (This could probably be circumvented somehow, but I don't want to break the warranty on the thing as it is 'borrowed' from the telecom provider)
The raspberry is my way of circumventing that. It is always connected by LAN, so now I can easily ssh into it from our other devices which are all on WiFi.
That makes sense! And it's too bad the router blocks it. But having an always-on and connected raspberry pi can be useful for a lot of things so hopefully you can benefit from having it already set up now
Synology with SHR is pretty much plug & play, accepts any random disks you have laying around and still can recover from one disk failing without data loss.
The 9xx+ -versions are pretty good bang for the buck. (I think the current one is the 923+) It has hardware acceleration for video decoding and decent power to run applications via Docker. You can also install a bunch of "standard" stuff like the -arr stack just by searching "<application> synology" and most support it out of the box.
I personally ran a 918+ from 2017 to last fall with zero issues. Started by slapping in random drives I had around the house and upgraded them piecemeal with shucked WD MyBooks I got on sale off Amazon. Upgraded memory to 12GB and added an SSD cache for a bit of a performance boost. I'm up to 18TB of usable space and 12GB of memory on it currently.
The first issue (last fall) was that it just didn't boot. Which was fixed with a new PSU from Amazon, but it kinda opened my eyes. Because of the proprietary nature of the SHR system, the only way to get data off the drives is a new Synology unit and some luck.
So I went a bit overboard and bought a Node804 case and built a completely overpowered Unraid server (Ryzen 5700G, 64GB of RAM and a total of 48TB NAS-grade SATA drives with a 2TB SSD cache 😅 - I did grab all of the components from Black Friday sales over a few months, but it was still a bit pricey.
On the other hand now the data is stored on a standard XFS filesystem on every drive. If the computer fries, I can just pop the drives out in an USB dock and read them without any hassle. It's also powerful enough to do pretty beefy processing by itself without me having to use my desktop to reencode video over the network.
The 918+ is going to my parents when I visit next, to act as an offsite backup the next time I make a longer visit.