Home network help part 2, SSH and Server
Edit: I've made some progress if you want to read the edits at the end.
Last year I started slowly planning out a home server setup with help from Tildes. I've gotten a few things up and running, but have been bouncing off a variety of walls trying to get to the next step.
The first goal was-
"Ok i've got Cosmos up and running for local access using self signed certs. I'd like to get it up and running using lets encrypt and a domain so I can eventually start giving a few family and friends proper logins and external access". Of note, ideally,
This led to a second goal of-
"Gosh it sure would be nice if I didn't have to be sitting at the physical server to do testing and could instead be at another computer in my house. I should probably configure ssh locally (working) and get it to forward windows so I can work in other rooms (not working...)"
"The stack":
Server - MS01 running LTS Ubuntu with Cosmos Cloud installed (well it was, but is currently not)
Router - Ubiquiti Dream Machine Pro (of note i've done some minimal guided config of this to try and harden it at a basic level so my cameras and IoT devices are better isolated. Not fully default, but the server is, for now, in the same network/vlan as the rest of my main computers so don't think this should matter.)
Clients - All local windows 10/11 machines for now, although in the off off chance it matters, i'm running nushell in the terminal
Domain Provider - Cloudflare
The SSH Problems:
I have a friend who's set SSH up for themselves with their home server, however they haven't had time to come over and troubleshoot. My rough understanding is "setup VcXsrv, change some configs, then it just works.". Windows these days has ssh built in, and I can SSH to the machine just fine with my key.
ssh -X...less so. I've read some docs, followed some guides, tried copilot, and it all leads to "yeah should work" and it just doesn't. I have configured a ssh config on both machines to allow X11 forwarding, i've started the XLaunch making sure I disable access control, made sure my unbuntu login isn't on wayland and so on. So far, no dice.
If someone has an end to end guide they trust to link, i'll gladly read and start from scratch. I've been cobbling together so many sources at this point i'm very lost. Lots of things jump quickly to "well just use WSL", which yeah ok i probably should test that next, but I was hoping I wouldn't need to (and am unclear if that'll even help).
The HTTPS/Domain Problems:
So..cosmos cloud.
I like the theory behind this software in that it helps enforce best practices so you don't blow your own head off when you screw something up. Maybe it's not the absolute best starting place, but getting it running without a domain was trivial, and more importantly, shockingly well documented. Not perfect, but for what I understand is mostly a one man show it's better than a lot of professional grade stuff i've dealt with.
And so I figured it'd be easy to just do the setup from scratch but choose https and point to my domain. There's been two attempts here, no DNS challenge and DNS challenge
No DNS Challenge Method
Per their docs it seemed easy enough. I'd never touched a DNS screen before but I configured an A record pointing at my WAN IP (eventually...) and disabled the cloudflare proxy.
Well going to that domain took me to my router login. Hmm. After screwing around with port forwarding and router DNS records I never got it to work and felt like I was playing with fire, so undid everything I'd done and decided I'd try the DNS challenge. Of note I could still access the cosmos cloud page from http directly to the IP, where it confirmed it failed to get the TLS cert, but https to the domain wasn't having it.
DNS Challenge Method
This seemed like I was close, and then nothing. I have no idea if i need to do internal routing on the router for this, it just sorta says "Do the DNS challenge, here's a form, you don't need to fill out all of it" which uh...ok.
I filled out what I think I needed to after setting up a token(not an API key) in cloudflare. I'm pretty certain I got that correct as I saw text files with keys created on cloudflare's DNS page and had I screwed that I'm guessing it couldn't have.
However from what I can tell, that's as far as it got. The files nuked themselves 2 minutes later when the TTL expired, and going to the domain locally gave me the cloudflare "our shit's fine, the server is timing out" page. From what I could tell diving into logs, cosmos had the same error, and I couldn't hit cosmos at all, even using the IP and http.
I do however wonder if maybe it did work BUT since I undid the router DNS record before trying this maybe that killed it? dunno.
Any ideas?
That's basically my situation. Figured i'd throw it here and see if anyone has some guidance or troubleshooting they'd recommend. Aforementioned friend who's done some of this before should be free one of these weekends and can probably help, and I haven't tried again since the second attempt. I've thrown some of the questions i've had on the discord and gotten minimal response(although I'm kinda using the thread as a rubber ducking spot as well). Next attempt is probably just DNS challenge again after more research on it and seeing if that works if I put back on the router DNS record, but i feel like logically that shouldn't work.
Oh also if anyone has some general recommended reading so that I can really understand what the hell it is I'm doing I'd love that. There's a ton of networking books/articles/etc, and in general I'd like to learn more about the subject, but I'm curious if there's a go to for people who are techy and trying to dip their toe in all of it the same way I am and setting up a proper home network and server.
Edit:
So after lots of testing, doc reading, and help from the cosmos discord I:
- Got the DNS challenge to work according to the cosmos logs.
- narrowed down that the main issue was my UDM pro router policies. Needed a firewall rule and a port forward, and had only done one of those at a time in my various attempts and not realized they were really different.
Now once that was all working and I could hit the site i was getting "likely a false cert" errors, but since i've got all the pieces I'm probably going to try another clean install later and see what we get. Hurrah for troubleshooting, good docs, rubber ducking, and helpful humans.
Edit 2:
Eventually required:
- Port forward rule in UDM pro
- Firewall rule in UDM pro
- Static IP and DNS entry in UDM pro.
One I’d done those things started working. Killed it after that as now I need to think about architecture
I just rent a 5$ VPS (and I'm sure you can go cheaper) with a public IP that runs a wireguard server. From there I can give myself remote access to my home network, and in theory could allow other people access to certain IPs or subnets on my network. Not sure if it works for your usecase, but it seems the simplest approach if the people you're giving access to can be bothered to connect to a VPN. I believe there are apps for most devices (Android TV, Xbox etc.) that offer QR code based setups. Then you can just point them at your local IP address.
For ssh my recommendation is to avoid
ssh -X. Even when it "works" it is painful. It seems like Cosmos has a web UI? Can you just use that and use plain old ssh for anything you need to do on the machine itself?Yeah i was looking into VPN options so i'll keep that in mind
For ssh, well it was supposed to be an easy diversion from the main problem. The server setup is somewhere mildly incontinent so I thought it'd be a quick win to learn how to ssh (yes) and ssh -X (god no) before diving back into troubleshooting cosmos with lets encrypt. I'm beginning to see that's not the case, and naturally i can't use the cosmos UI for this particular troubleshooting, but I guess i'll just have to live with the torment of moving to a slightly less comfortable setup instead of mastering ssh -x.
I don't know exactly what you want to achieve long-term, but since you already have your own domain name and are on Cloudflare you could use a tunnel rather than fiddling with port forwarding and potentially fighting with dynamic IPs. Is it absolutely best practice? Maybe not, but it should be easy to set up and easy to replace later if you desire to do so.
With tunnels Cloudflare takes over the TLS termination. Whether this is good or bad depends on if you care that Cloudflare has terminated TLS in their edge servers. You don't need to do any of the cert stuff with Cloudflare tunnels because they do it all on their edge servers. I haven't verified, but my recollection is that they then re-encrypt so that all in-flight traffic is encrypted.
As for visual ssh I've got nothing. I personally avoid ever trying to ssh visual applications because it's not fun. The only exception is that if the visual application is really just a web app because using a local browser connected to the remote server is easy.
Yeah part of the point of the project was to get a better understanding of https/lets encrypt in general, so while the tunnel would probably work I'd like to get it working.
Granted the point was to get it working and work backwards, since I sorta assumed it'd be closer to plug and play, not just the endless troubleshoot loop.
For X forwarding I've given up on most of the mishmash solutions and just use https://mobaxterm.mobatek.net/ which works most of the time, with a little trick for forwarding if sudoing.
Just to make sure i'm not on the wrong track, what is the standard way of remoting into a server if you actually need the desktop?
I know that's not common, but in order to do the initial cosmos setup you need to navigate to the main site and configure some stuff through the web app, which can only be done from the local machine until configured.
I've always heard "oh just use ssh" so i figured that'd be the way, but is there some other method everyone is using these days and i'm trying to make something ancient work?
I usually use SSH and X forward anything local that needs a GUI (but most is over CLI or maybe a web page) but there are other options, most of which I'd say are less used. They boil down to running a VNC or similar server and then connecting to that (I think TigerVNC and FreeRDP (freerdp-shadow-cli seems to be the server) are somewhat up to date) which gives you access to your entire desktop as with RDP to Windows.
I'd lean away from full remote desktop solutions unless you need to preserve a GUI session and access it remotely and locally.
Edit: For cosmos, anything that 'can only be accessed locally' as a webapp can probably be port-forwarded over SSH (aka a tunnel to localhost:80).
NoMachine works pretty well on low-bandwidth links. Getting NoMachine to run securely without paying money is more involved so I won't get into it here.
freerdp is also quite good! (but for some reason on my setup audio isn't forwarded). To use freerdp I run rdpserve.fish on the server and rdp.fish on the client in one go like this:
All that being said I usually just use ssh instead of GUI because there is still latency with remote X11. The feeling of
mstsc.execonnecting between two Windows machines is a lot better: the UI feels very responsive and clear.If you're on Wayland maybe the situation is different with waypipe, etc.
.....and yeah title should be "Home Network" not "Homnetwork" if someone can change that.
Fixed!