14
votes
What programming/technical projects have you been working on?
This is a recurring post to discuss programming or other technical projects that we've been working on. Tell us about one of your recent projects, either at work or personal projects. What's interesting about it? Are you having trouble with anything?
Not programming.
But I've been getting my head around Databricks, Lakehouses and Data Vault architecture recently. My new role uses all of these and they have my head spinning with a million possiblities.
As one of my tiamasomo 2023 goals, I want to turn my rPi4 into a streaming server. If anyone has tips to break analysis paralysis, that would be massively appreciated. I keep making zero progress due to researching too much.
Here are the
completedsteps and bolded decisions I expect in the process.Format external 4TB HDD(chose EXT4 b/c it was the default, and I don't think most Linux file systems are meaningfully different until you're using multiple discs)Install NavidromeReally could use some pointers on my overall plan here.
ext4 is totally fine. xfs is generally preferable if you're doing database-intensive work. I run ZFS on all my machines but it's way overkill for an external drive hooked up to a Pi.
if you don't care about Windows, NFS will probably be easier to set up. make sure it's version 4.
what hosts the DNS for your domains? if it's AWS Route 53 you can use this. if it's another provider, you'll need a dynamic DNS client that's compatible with that provider's update API. if one isn't available, an alternative is to set up a dynamic DNS entry using someone else's domain (freemyip is one that I've used in the past) and then set up a CNAME in your domain that points to the dynamic DNS hostname.
Jellyfin should work, I think:
If you have several types of media in a single folder you can also label it as mixed, which will be a generic folder view that displays all files in the library.
By default, the enables access to all libraries option will be enabled, disabling this option will enable you to give the user access rights per library, libraries can consist of several folders.
what problem are you trying to solve with the VPN?
do you need/want the server to be accessible outside your home network? if it's local-only you can simplify things greatly.
do you need/want it to be accessible by other people, or just you?
Wireguard is the current state-of-the-art for VPNs, I'd definitely recommend it over any other options (such as OpenVPN, L2TP, or god forbid IPsec).
a simple option would be to run Wireguard on the Pi, forward a port to the Pi from your router, and then set up each of your Macs as a peer (using the dynamic DNS hostname as the endpoint). when the tunnel is up you'll be able to connect to the Pi even if you're not at home. when you're at home you'd need to connect using a different IP (the one assigned to the Pi on the local network, rather than its Wireguard IP).
it also may be possible to run Wireguard on your router directly, which would let you connect over the VPN to your entire home network rather than just the Pi. this would also allow you to use the same IP to connect to the server whether or not you're at home.
Works for me. IIRC, NFS will auto-mount on Macs while SMB needs to be manually reconnected each time.
For the domains I want to use for streaming (thinking of setting up one for Navidrome and one for Jellyfin), Pork Bun; for most of the rest, Cloudflare. I believe they both have dDNS clients.
So I can offload excess files while on the road. I'm planning on some extended travel (again), so I'd like to be able to connect and move files over to the NFS drive.
No, just me on two iPhones and two Macs (and perhaps an iPad in the future). If I set it up for just me and later change my mind, is this a scrap it and start over change or merely adding another user to the list of authorized accounts?
OK, that's the bit of context I was missing - you as a road warrior, with multiple devices phoning home to a centralized media server. here's what I recommend:
install Wireguard on the Raspberry Pi, and your client devices (the two laptops and two phones)
set up port-forwarding for the Wireguard UDP port from your router to the Raspberry Pi
192.168.0.10if your home network uses192.168.0as its prefix)set up Wireguard
192.168.202.50as the Pi's Wireguard address, and then addresses starting with192.168.202.100for the IPs of each peer device you assign. these are very intentionally different from the192.168.0addresses above. I can go into more detail about this if you want.the crucial thing Wireguard gives you is what in five-dollar software engineering terms is called an overlay network. your home network might use
192.168.0and your hotel or campground wifi might use10.0addresses. WG creates literally a "virtual, private, network" where your laptop at192.168.202.101is talking to the Pi at192.168.202.50and the entire thing is happening within an encrypted tunnel.in terms of "checkpoints" to get this thing running:
while on your home wifii, connect from your laptop to the local IP of the Pi (eg 192.168.0.10) and access both Navidrome and Jellyfin on their HTTP ports, and mount the NFS volume
go to a coffeeshop (or tether your laptop through your phone's cell connection - either way gets you off your home wifi) and connect Wireguard to your Pi and check that you're able to ping
192.168.202.50(or whatever you chose for the Wireguard IP of the Pi)only once the last two are working, should you try the two things together, and try to connect to Navidrome or Jellyfin through the Wireguard tunnnel.
As always, thanks for your informative answers. I hope Tildes is indexed so someone else can learn from you correcting my ignorance.
Neat, setting this up is like a slightly faster Tailscale but without a SAAS in between.
Already have that set up for the PiHole. Should I also set those up for my Macs? As I understand it, NFS works better with authorized IPs than addresses like
public-macbook.local. The Macs also get two 192.168.whatever addresses if they're connected to both Wi-Fi and ethernet—would that make a difference for assigning reservations on the local network?Sounds like my order of operations is
I presume this is to make it unambiguous which is the overlay and which is the local network—useful to keep devices not on the wireguard network separate even when on the local physical network. Is that right?
I've gotten a bit stuck getting NFS properly setup. When I try to connect, it times out.
Don’t mess with new filesystems, the risks are real but the benefits aren’t. That said, my media server has a two disk ZFS array with 2 14TB disks, it is certain the disks will be out of warrantee before they get filled.
If you're not using Windows, stick to NFS for directory sharing - it's just easier. If you're just using this to get the files onto the Pi, forget NFS and just rsync them over.
Jellyfin is great - it's a nice, slick Netflix-style UI. It can also handle TV tuners, if you get the Pi TV hat.
I expose my Jellyfin install to the Internet, with no VPN, so my parents can access it. Jellyfin has password-protected user accounts, so that's fine for my needs.
Setting that up to work remotely isn't much different than ensuring I can SSH remotely [setup DDNS first]. Even if I'm copying off the server, I can rsync. I was thinking of NFS b/c I do prefer a graphical environment for file management.
I presume the VNC can also work remotely once DDNS is set up. I want to use that so I can do things like log into Firefox on the Pi and directly download my Bandcamp purchases instead of downloading them to my Mac, then transferring over [or ripping discs].
I've been learning perl. It's a weird fun language.
3-sided cards:
For each design “card”, each “print” has been getting a unique URL for a few weeks, now there is a sqllite database for each card that keeps track of the prints and has a sequence number that gets incremented if a print is made successfully, otherwise the print is treated as defective or a work print. The back side visible here is “print #344.2” and is the second card to have a sequence number.
https://mastodon.social/@UP8/111168528593969946
There is likely to be a public database so I can do things like give somebody a business card and register a portrait I take against it, or let people register cards and trade them like NFTs (no money though!)
I updated my stereogram developer to output side-by-side stereo pairs in addition to the red-cyan anaglyphs I make because people on dpreview are asking for them. I am planning on stepping up stereogram production so there will be more updates in this area, I really need something to adjust exposures, crop, etc also I am still trying to produce stereograms with a light-field camera, the Lytro Desktop isn’t cutting it, and I may need to look at open source plentopic processing software.
Blog: no progress
YOShInOn RSS reaser: I have a long list of “little” things like an edit dialog for the HN submission queue, a submission queue for Mastodon (as opposed to immediate posts) but the thing that is looming is reworking the ingestion system so I can keep track which feed every item has come from, the processing of tearing the ingestion system down and rebuilding it could be a hassle so I am avoiding it, but a bunch of features I want will become easier to code up once I get it done so I should “just do it.”