Advice Needed: Simple and Reliable notifications
I have a long standing problem that probably has several good solutions, I just haven't been able to figure them out. So here I am, asking you.
I'm selfhosting some services, a mix of selfbuilt and open source software. But some things I don't want to selfhost. Notably backups and alerts/notifications. For backups I have a solution which works well in every regard except one - I don't always get alerted when things fail, because the way I send myself those alerts is failing more than the actual backups.
Currently I'm using python and gmails smtp interface to send myself email, but gmail disables my smtp access from time to time, and it's really easy not to notice not getting an email. I've tried sending the email regardless of whether the backup failed or not, but I've noticed several times that I still don't notice if the they stop coming.
Now on to my requirements/wishlist.
- I'm already using s3 glacier at aws for the backups, so preferrably something in the aws space.
- I would like to get an popup/toast on my phone when a message is being sent. And the ability to review messages later.
- I would like as few moving parts as possible.
- I don't want to write my own client.
- I want it to be cheap, and if there's a cost I prefer to pay it at a place where I'm already paying, meaning aws (or possibly proton).
- I want a stable service.
- I prefer to manage as little as possible of the infrastructure.
- I'd like a simple programmable interface that can't easily fail. E.g. http based.
- It's no problem if messages are not received instantly, I could easily tolerate delays up to 24 hours.
As you may have noticed I'm pretty much expecting there to be something in aws that I can use, but aws documentation is so abstract, that I often don't understand what the point of something is or how I'm supposed to use it.
You could take a look at https://ntfy.sh/
You can self-host it as well.
A similar project is gotify: https://gotify.net
Note, no experience with it myself, I just have seen it around over the years.
A straightforward AWS-centric option is SNS; you can just send yourself SMS. Assuming you have control over how your backup service sends out its notifications, you can route payloads to SNS however you'd like.
If that's not an option: Going off no self-host, AWS, and HTTP-based: A really low-brow way could be using Lambda + messaging service of your choice, with your phone configured to receive notifications on that service? I've done exactly that with Discord before when doing a one-off hacky project to annoy my friends.
Discord webhooks are great for near-zero effort notifications. One thing to keep in mind is that (depending on configuration) Discord will throttle notifications to your phone to keep it from driving you insane. I think that only applies to "all messages in the channel" notifications; @mentions are typically all delivered in my experience.
You said you were using Python to send yourself email notifications, why not send via Amazon's SMTP service (SES) instead of gmail? You wouldn't have to change very much and it's essentially free for low volumes of emails.
Alternatively you could read the response from gmail's SMTP server and handle it when it sends back whatever code is associated with your intermittent outages. Possibly a small delay and a retry would solve it.
Edit: Missed a word
Do you have an easy way of checking on the status of e.g. backups? If so would setting a calendar notification with a link to the status check work? I.e. you set a notification to check your backups after they've run. It won't be pushing notifications when something fails (so might not be what you actually want), just a reminder to check the status.
Hi @Bwerf not sure if my recommendation may be exactly what you need...but what i use is a private, dedicated matrix room to receive automated notifications. Why? Because I'm in matrix chatting with folks often enough, that receiving notifications - like a backup job finished or failed - is easy for me to actually see/not miss, etc. So, I also use my own python script (https://github.com/mxuribe/howler) to send http calls for each message directly into said matrix room...Of course, since the matrix api for this is quite straight-forward, there are plenty of scripts out there beyond just python.
If matrix messages are not your thing, and prefer email, have you considered using a service like mailgun, or messagebird for smtp...that is, separate from your gmail setup, so might fail a bit less? Before i started using my matrix message approach years ago, i used to use a python script to smtp send an email by using messagebird...and associated it to some generic, dedicated domain name just for this (to avoid messing with my main email address), and the usage was low enough that it fell under their free tier...so was easy api use, really great reliability, nice performance, and of course free cost....but i only moved away from this because i started chasing - at the time - the new hotness called matrix...this was back in 2018 or so...and so far matrix has worked great, but again, email is totally good option too. Thankfully, there are many such email providers with quite generous tiers.
Please feel free to reach out if you have any questions on matrix stuf...also, while i linked to my python script, there are many, many other matrix tools that can be used as well...one cli example is matrix-commander (https://github.com/8go/matrix-commander). Good luck!
EDIT: Fixed urls for the 2 links.