Introducing Backtick API: An open-source, unofficial Tildes API
Hi everyone! Today I am releasing Backtick API, an unofficial Tildes API. I originally wrote it to power Backtick but quickly realized that it could be useful for other clients and projects that require Tildes data. The project is open-sourced, both because I would love community contributions and because I want to make the project as transparent as possible.
Q: Does it require or store any user credentials?
No. All the endpoints that are implemented scrape publicly available data that do not require any user credentials.
Q: Will this overload Tildes?
To prevent Backtick API from being a nuisance to Tildes, the code caches calls to Tildes for a short period of time. Depending on usage, it should reduce the number of calls to Tildes (vs a client querying Tildes directly).
Q: How can I use it?
A hosted version of the code is available through RapidAPI here: https://rapidapi.com/asnewman/api/backtick-api
Q: Are there any limitations?
To prevent the project from being abused and becoming a financial burden on myself, I'm using RapidAPI to manage usage. There is a generous free tier and a reasonably priced paid tier which will be used to cover server costs. In addition, I will be donating a portion of the paid plan revenue (my goal is 50% but will depend on load and subscription count) directly to Tildes via Github Sponsors. It's really important to me to keep this project and Tildes as sustainable as possible. I really would love to provide everything for free, but unfortunately, there are countless examples where that doesn't work out down the road.
Of course, if you do not want to use the hosted version, you are welcome to deploy the code yourself!
Q: Isn't scraping prone to breaking?
Yes! Users beware! Obviously, I will try my best to fix things if the Tildes code changes in a way that breaks Backtick API, but it's definitely a possibility.
Q: Why not just work on and contribute to creating an official Tildes API?
An official Tildes API will always be better than something like this and I look forward to when/if it gets implemented. That being said, I'm choosing the route that would be the quickest to continue the development of the Backtick mobile app. In addition, it's also the choice that is most fun for me. To work on a Tildes API, if the powers that be even want to have one, would require lots of coordination and codebase learning, something that will take a lot of time. I highly commend anyone who chooses to take on that task, but I do too much of that kind of work for my real job 😂
Please report any issues or feature requests through Github, this post, or the Backtick Discord server. Finally, I just want to thank everyone in the community for all the great conversations I've had, for providing helpful feedback for Backtick, and for creating a place on the internet that I very much enjoy being a part of 😊
Well done! I hope this encourages people to try building interesting Tildes apps, integrations, frontends, experimental new features, and more.
Thank you 😊 I hope so too!
Any plans for a Backtick android app?
I'm not opposed to it, but it will take a lot of work. Have you checked out @talklittle's app that just went into alpha testing?
Ok, maybe I'm completely ignorant of the way APIs are supposed to work, but what exactly is rapidapi doing here? I was expecting that you created a piece of code that I call with something like a tildes URL and it gives me a JSON or similar object describing the contents of the underlying page. Instead, I call rapidapi with something like a URL, and it gives that back to me? So rapidapi is handling all the scraping and caching? Can't that be done locally? I guess that's what the "development" instructions on the github are for, right? How to do locally what rapidapi does. So you've built a typescript thingy that does the scraping and caching, and offers it as a http API, and I can spin it up locally too. And what we gain from this additional layer of indirection is that I can use any language that can do http requests to query backtick, instead of being limited to typescript. Also, it lets us centralize all the issues around accidentally DDOSing Deimos, because rapidapi will just refuse to, and it acts as a central cache too.
Am I getting that roughly right?
I should have been more clear in the post, Backtick API is a http REST API which you call via GET requests and it returns JSON data for Tildes information.
All RapidAPI does is wrap my server's url will their url and adds (RapidAPI) authentication around it for limiting usage. For example, when you call the RapidAPI endpoint for Feed data using curl like this:
what it does is call the underlying server Backtick API server endpoint, as long as your RapidAPI account for Backtick API is free-tier and within the limits or paid-tier. In short, people use RapidAPI to monetize and document their APIs. It doesn't really do anything beyond that.
All scraping and caching is done in the code, not RapidAPI. You can run the code locally and hit your own local endpoint to get data if you want to avoid going through RapidAPI and are fine hosting the code yourself.
What you gain from it is that your client no longer needs to call Tildes directly and parse the HTML manually because it's provided by Backtick API through a REST endpoint (which is language agnostic).
Hmmm, ok. I figured RapidAPI was running your code on their server, but it seems they're just running an authentication wrapper of some sort, and otherwise forwarding everything to your server, which is ultimately running the code. From the outside, there should be no difference there I suppose.
From the perspective of someone who would be using your API, if at all, merely to generate data from the website, rather than build a service around the API, all those layers of indirection seem unnecessary, hence the silly questions. I wouldn't care about caching (I'd hope to query every URL only once) or where the scraping code is being run, because from my perspective it doesn't really matter. But I think I understand now.
It sounds like it’s a shared cache so there’s some benefit in reducing load on Tildes. If someone else downloaded the same URL recently, it would be downloading it zero times rather than once. That’s probably more important if there are lots of clients using it, though?
And whether that’s even necessary depends on Tildes’ own caching.
Exactly right! The cache is only 30s (right now), but it helps a lot for endpoints that are used often. For example, when you open the Backtick mobile app, it loads the front page feed. If 10 people open the app in the cache span, then Tildes is hit only once (instead of 10 times). It's done to be respectful to Tildes. I wouldn't want anything I build to become too much of a burden on the Tildes server.