Advice Needed: Simple and Reliable notifications
I have a long standing problem that probably has several good solutions, I just haven't been able to figure them out. So here I am, asking you.
I'm selfhosting some services, a mix of selfbuilt and open source software. But some things I don't want to selfhost. Notably backups and alerts/notifications. For backups I have a solution which works well in every regard except one - I don't always get alerted when things fail, because the way I send myself those alerts is failing more than the actual backups.
Currently I'm using python and gmails smtp interface to send myself email, but gmail disables my smtp access from time to time, and it's really easy not to notice not getting an email. I've tried sending the email regardless of whether the backup failed or not, but I've noticed several times that I still don't notice if the they stop coming.
Now on to my requirements/wishlist.
- I'm already using s3 glacier at aws for the backups, so preferrably something in the aws space.
- I would like to get an popup/toast on my phone when a message is being sent. And the ability to review messages later.
- I would like as few moving parts as possible.
- I don't want to write my own client.
- I want it to be cheap, and if there's a cost I prefer to pay it at a place where I'm already paying, meaning aws (or possibly proton).
- I want a stable service.
- I prefer to manage as little as possible of the infrastructure.
- I'd like a simple programmable interface that can't easily fail. E.g. http based.
- It's no problem if messages are not received instantly, I could easily tolerate delays up to 24 hours.
As you may have noticed I'm pretty much expecting there to be something in aws that I can use, but aws documentation is so abstract, that I often don't understand what the point of something is or how I'm supposed to use it.
You could take a look at https://ntfy.sh/
You can self-host it as well.
A similar project is gotify: https://gotify.net
Note, no experience with it myself, I just have seen it around over the years.
Looks very neat, if I was interested in self-hosting for this, gotify would probably be my choice!
Hmm.. that could be an option, I'm gonna try one of the mail-based ones first I think, but this looks like a reliable fallback.
A straightforward AWS-centric option is SNS; you can just send yourself SMS. Assuming you have control over how your backup service sends out its notifications, you can route payloads to SNS however you'd like.
If that's not an option: Going off no self-host, AWS, and HTTP-based: A really low-brow way could be using Lambda + messaging service of your choice, with your phone configured to receive notifications on that service? I've done exactly that with Discord before when doing a one-off hacky project to annoy my friends.
Discord webhooks are great for near-zero effort notifications. One thing to keep in mind is that (depending on configuration) Discord will throttle notifications to your phone to keep it from driving you insane. I think that only applies to "all messages in the channel" notifications; @mentions are typically all delivered in my experience.
@Bwerf Second vote for SNS if you want amazon-based option. The nice thing about this isnthat you can send one notification to a channel and configure it to go to multiple places (e.g. email several people, email and text message, etc).
Slack is a good alternative to discord, has a free tier, and has a nice web hook option.
If you just want send emails using an API hook, Resend has a nice wrapper around AWS SES and has a generous free tier.
SMS would indeed fulfill my specs, but I guess it does limit me to only checking on my phone. SNS looks very capable and powerful, but a bit complicated? Or maybe I'm just overwhelmed that it can do so much, and I just need to figure out the few parts I actually need to do. This looks like the kind of thing I might be interested in learning because it could be fun, but maybe not the shortest path to the goal.
You said you were using Python to send yourself email notifications, why not send via Amazon's SMTP service (SES) instead of gmail? You wouldn't have to change very much and it's essentially free for low volumes of emails.
Alternatively you could read the response from gmail's SMTP server and handle it when it sends back whatever code is associated with your intermittent outages. Possibly a small delay and a retry would solve it.
Edit: Missed a word
SES looks like it's designed for bulk mail? But maybe I'm just sending bulk mail with a single recipient =) I think I'm gonna look into this first. Thanks!
I wouldn't say it's designed for bulk mail exactly, it works like any SMTP server.
Adding: If you run into issues setting it up, feel free to msg me
Do you have an easy way of checking on the status of e.g. backups? If so would setting a calendar notification with a link to the status check work? I.e. you set a notification to check your backups after they've run. It won't be pushing notifications when something fails (so might not be what you actually want), just a reminder to check the status.
Unfortunately I think I would go months without noticing, and right now there's no super-easy way to check the status other than these (potential) notifications. There is also the server logs, but the way it's set up I can really only check those from a specific computer and they're a bit time consuming to manually parse.
Hi @Bwerf not sure if my recommendation may be exactly what you need...but what i use is a private, dedicated matrix room to receive automated notifications. Why? Because I'm in matrix chatting with folks often enough, that receiving notifications - like a backup job finished or failed - is easy for me to actually see/not miss, etc. So, I also use my own python script (https://github.com/mxuribe/howler) to send http calls for each message directly into said matrix room...Of course, since the matrix api for this is quite straight-forward, there are plenty of scripts out there beyond just python.
If matrix messages are not your thing, and prefer email, have you considered using a service like mailgun, or messagebird for smtp...that is, separate from your gmail setup, so might fail a bit less? Before i started using my matrix message approach years ago, i used to use a python script to smtp send an email by using messagebird...and associated it to some generic, dedicated domain name just for this (to avoid messing with my main email address), and the usage was low enough that it fell under their free tier...so was easy api use, really great reliability, nice performance, and of course free cost....but i only moved away from this because i started chasing - at the time - the new hotness called matrix...this was back in 2018 or so...and so far matrix has worked great, but again, email is totally good option too. Thankfully, there are many such email providers with quite generous tiers.
Please feel free to reach out if you have any questions on matrix stuf...also, while i linked to my python script, there are many, many other matrix tools that can be used as well...one cli example is matrix-commander (https://github.com/8go/matrix-commander). Good luck!
EDIT: Fixed urls for the 2 links.
I'm very rarely on matrix, so I don't think it would be a good fit, but it's an interesting fallback solution. Thanks.
I suppose the same kind of concepot can be applied to several other platforms....for example, i actually got this idea for a matrix room from many, many years ago...where at my dayjob we used a private Twitter group (so long ago, i forget what they used to be called)...but any way, it was private and users culd only view contents sent to the twitetr feed if invited/added to said private group...It worked great because the VP at time was constantly on twitter...so while about to enter sales visits, he would view twitter to see if we had reported outages or automated system messages to the private group, etc.
Obviously separate of matrix, twitter, etc...you could use a similar approach....I have to imagine telegram, discord, wherever you spend enough time that behaves like a chat room or message inbox or soirts, you could setup some private space/area...and send notifications there. I hope that helps! :-)
Maybe against #7 but I’ll just leave my setup here.
I use Healthchecks.io to monitor my backup jobs, you can self-host (which I do) or you can use the service hosted by them (you get to monitor 20 jobs for free).
What you do is you set up a scheduled check with a specific interval, and if the check isn’t pinged (with something like curl or python, etc…) in the given interval it will be marked as failed and will trigger a notification to a wide variety of services (webhooks, smtp, ntfy, etc…). If your job returns an exit code upon completion you can also send that to healthchecks so if the job ran but failed you will be alerted. Signaling failures
I use it with ntfy (I self-host but one doesn’t have to) to get a notification on my phone.
Hope you find something that fits your needs!
Is there a reason you can't use SES instead of Gmail to send your email alerts? For low volumes, the cost is a rounding error, and you can keep using pretty much all your exiting infrastructure; SES supports SMTP for sends.
SES looks like it might the right solution, also I'm not married to smtp itself, but it will be very easy to implement, so that is indeed a plus.
Telegram is free and easy to use.
Same way I send mine. It's the only thing I use Telegram for.
Kinda goes against requirement #7 but if you by chance are already hosting Home Assistant some place it's simple enough to setup web hooks that can trigger notifications in the companion app on your phone. Not sure on iOS but android app has a history of notifications under setting>Companion app>Notification history.
I needed a way to send myself notifications that would work both when I'm using my Android phone and (more often) when I'm daily driving my dumb phone. I settled on textbelt.com, since it was the easiest way to send myself SMS messages I could find. Basically you create a secret API key, buy credits for that API key, then call a dead simple HTTP API with that key to send messages (I just use curl from the shell script that runs my backups, plus some cron jobs to send occasional reminders). Haven't had any issues with late or lost messages in the couple years I've been using them.
For non-SMS solutions I'll add my voice to the Ntfy recommendations--I've self-hosted an Ntfy server and used it for Android push notifications and had good experiences with that as well.