Hi everyone! Today I am releasing Backtick API, an unofficial Tildes API. I originally wrote it to power Backtick but quickly realized that it could be useful for other clients and projects that require Tildes data. The project is open-sourced, both because I would love community contributions and because I want to make the project as transparent as possible.

Q: Does it require or store any user credentials?

No. All the endpoints that are implemented scrape publicly available data that do not require any user credentials.

Q: Will this overload Tildes?

To prevent Backtick API from being a nuisance to Tildes, the code caches calls to Tildes for a short period of time. Depending on usage, it should reduce the number of calls to Tildes (vs a client querying Tildes directly).

Q: How can I use it?

A hosted version of the code is available through RapidAPI here: https://rapidapi.com/asnewman/api/backtick-api

Q: Are there any limitations?

To prevent the project from being abused and becoming a financial burden on myself, I'm using RapidAPI to manage usage. There is a generous free tier and a reasonably priced paid tier which will be used to cover server costs. In addition, I will be donating a portion of the paid plan revenue (my goal is 50% but will depend on load and subscription count) directly to Tildes via Github Sponsors. It's really important to me to keep this project and Tildes as sustainable as possible. I really would love to provide everything for free, but unfortunately, there are countless examples where that doesn't work out down the road.

Of course, if you do not want to use the hosted version, you are welcome to deploy the code yourself!

Q: Isn't scraping prone to breaking?

Yes! Users beware! Obviously, I will try my best to fix things if the Tildes code changes in a way that breaks Backtick API, but it's definitely a possibility.

Q: Why not just work on and contribute to creating an official Tildes API?

An official Tildes API will always be better than something like this and I look forward to when/if it gets implemented. That being said, I'm choosing the route that would be the quickest to continue the development of the Backtick mobile app. In addition, it's also the choice that is most fun for me. To work on a Tildes API, if the powers that be even want to have one, would require lots of coordination and codebase learning, something that will take a lot of time. I highly commend anyone who chooses to take on that task, but I do too much of that kind of work for my real job 😂

Please report any issues or feature requests through Github, this post, or the Backtick Discord server. Finally, I just want to thank everyone in the community for all the great conversations I've had, for providing helpful feedback for Backtick, and for creating a place on the internet that I very much enjoy being a part of 😊

Hello!

Some of you may know that I have been developing an iOS client for Tildes.

I’ve seen other devs like @wababa and @talklittle also working on awesome apps, and have seen some discussion there around collaborating on some of the core parts (scraping the website, etc.).

I’ve also been in touch with @efraimbart, who is working on the awesome Everything protocol project.

I’m sure there are other devs out there that I haven’t seen, and this post is a call-out for all of you.

Until an API is available, any similar project will face the same initial blocker: needing to scrape the website for data.

Is there any interest from the community on collaborating on that part? For example, developing a public library that encapsulates all of the interactions with Tildes and any client can talk to with a consistent API. Or, even better, collaborating on an official API for Tildes (which is, of course, itself an open-source project).

To that end, I’ve released some of the code I use to accomplish this in Surfboard. Maybe it can help someone build an iOS client, or inspire some of us to build a more generic solution :D. It’s far from perfect or complete, and isn’t even an actual package you can just import and use at this time, but at the very least it documents many of the endpoints and interactions.

Have a great weekend :)

I understand there are no plans to develop an official app, which is understandable to me as that would require a lot of time and resources for a project like this.

But it's clear that there are lots of devs eager to chime in and create an app by themselves, just like they did for Reddit. There's even one iOS app already in progress, although it relies on HTML parsing to retrieve the data obviously.

So is there any official stance on this? Why not publish the API and let the community do its magic? Any thoughts?

A lot of people want to talk about Reddit and that will likely continue. This is a place to post minor news updates, so that Reddit topics don't fill up the front page of ~tech.

(Up to you what counts as "minor.")

My typical approach is one that I believe is pretty common: Reading the response header for current count and waiting if the limit is reached.

However, I am currently working with a couple of APIs which don't implement that and are currently set up with rate limits on an honesty system.

Is it a case of throwing sleep statements into you code, or using some kind of "bucket" and "lock" system?

I'd be interested to see any simple implementation people have used (the simpler the better).

We have a backend that kind of does REST APIs but cannot handle simple Bearer tokens for authorization and cannot produce the full set of HTTP error codes (the platform just doesn't allow, for example HTTP 501 to be returned programmatically). There is no Swagger for the API.

The thought was to use Microsoft API Management Services as a proxy of sorts. It would handle the Bearer token upfront, and then just proxy / wildcard the requests/responses to the backend. The hard part is that it needs to parse the return response, and if there is something like "{ errorCode: 501 }" property in the JSON, it needs to return HTTP 501 instead of the regular payload.

Does anyone have any experience in setting this up? It seems like the basic policy processing won't cut it, and so function apps and logic apps seem to be the ticket. We want to keep this facade layer as thin as possible. Microsoft APIM is the only platform we're allowed to consider at this time.

I have never been a leader at a big company (or anywhere...), and honestly I am pretty ignorant when it comes to money and business, so maybe that's why I feel this way but... isn't this what for-profit companies ultimately are supposed to do? (make money?)

Reddit is blowing up today over his internal memo, and that's when I kind of started to feel bad for him. Wouldn't an internal memo be expected at a time right now? Wouldn't it say that kind of stuff? I'm just curious but for others, if you were in his position, what would you do right now? Is there a better move to be made? What should he have said in that memo? I kind of feel bad for him. At the end of the day he helped create reddit, and it must kind of suck to watch your own project devolve and people come to hate you.

The thing about this API decision that got to me is how abrupt it was - 30 days or thereabout. That doesn't seem like very long. But aren't these decisions usually made by multiple people? (not just a CEO?) I also think it sucks that reddit app hasn't been made accessible to vision impaired folks. So maybe he sucks as a leader, but is that a reason to hate him?

I'd love to better understand.

Official Announcement

NYTimes Article

Apollo Apollo (well known iOS client) developer talking about the specifics. Sounds like the API will now be paid based on usage.

It's a bit easier to have an opinion after the Apollo developer revealed the specifics Reddit gave him. Other than the NSFW part, which seems odd considering the API will be one of the revenue streams that isn't advertiser supported, it seems reasonable, of course waiting on the final price per usage.

It was never going to be sustainable for Reddit's API to be fully free. It was just silly - you could use the whole site, which certainly costs money in both AWS fees and developers doing KTLO, and not see any advertisements via the API.

App developers will pass the costs along to the user, many will likely fold because it won't be commercially viable with the additional cost, but, well, that's the way of things.

Starting on Thursday night, Twitter cut off API access for some of the biggest third party clients.

From The Verge:

It’s hard to tell whether the third-party client outage is due to the API. Attempting certain calls from my individual Twitter developer account seemed to work, while Twitter’s own API explorer tool is currently broken.

It definitely seems like it is on purpose. For it to last this long without any update definitely makes it feel like it was done on purpose. Many developers' apps have started showing up as "suspended". In looking at my own account, I can see that both Tweetbot and Fenix are gone from my list of connected apps.

The Icon Factory (makers of Twitteriffic) have a blog post about it as well.

The complete silence from Twitter is completely baffling. Burning more than a decade of working with developers overnight seems incredibly stupid. As Paul Hadad, one of the makers of Tweetbot said:

Even during the darkest Twitter 1.0 days they were pretty open about what they were doing. I remember getting a call prior to the 4 quadrants token limit where they explained what was going to happen and answered questions. I wasn't happy but at least felt there was respect.

Hey everyone,

I've been programming for some time now but notice without any formalized education in CS I often get lost in the weeds when it comes to developing larger applications. I'm familiar with the principles of TDD and SOLID - which have helped with maintainability - however still feel that I'm lacking in the ability to architect a properly structured system. As an example, I'm currently developing a flask REST API for a website (just for learning purposes). This involves parsing a html response and serializing the result as JSON. I'm still quite unclear as to structuring this sort of thing. If any more experienced developers could point me in the right direction/offer up their opinion I'd be very appreciative. Currently I have something like this (based - I hope correctly? - on uncle bob's clean architecture).

Firstly, I'm defining the domain model. i.e the structure of the API response. Then, from outside in.

1. Infrastructure (Flask): User makes request via interface (in my case a request to some endpoint)
2. Adapters: request object checks if the request is valid (on the way back it checks if the response is valid) - Is this layer only for error handling?
3. Repository: I'm struggling a bit here, AFAIUI this layer is traditionally a database. In my case however, where the request is valid, is this where I should handle the networking layer? i.e all the requests to return the website source? I'm also confused given at this stage I should be returning the relevant domain model, like an ORM, but as my data is unstructured, in order to do this I need to transform the response first. Where would it be best to handle this?
4. Use Cases: Here I transform the domain model depending on the request. For example, filter all objects by id. Have I understood this correctly?
5. Serializers: Encode the domain model as JSON to return from flask route.

If you got this far, thanks so much for reading. I really hope to hear the opinions of more experienced devs who can steer me in the right direction/correct me should I have misunderstood anything.

I'm looking for some feedback on a feasible mechanism for structuring a few API endpoints where a purely RFC-spec compliant REST API wouldn't suffice.

I have an endpoint which returns $child entries for a $parent resource, let's call it: /api/parent/:parentId/children. There could be anywhere from a dozen to several hundred children returned from this call. From here, a child entity is related to a single userOrganization, which itself is a pivoting entity on a single user. The relationship between a child and user is not strictly transitive, but can each child only has one userOrganization which only has one user, so it is trivial to reach a user from a child resource.

Given this, the data I need for the particular request involves retrieving all user's for a parent. The obvious, and incorrect solution to the problem is to make the request mentioned above, and then iterate through and make an API request to retrieve each user. This is less than very good as this would obviously be up to several hundred API calls.

There's a few more scalable solutions that could solve this problem, so any input on these ideas is great; but if you have a better proposal that also works, I'm keen to explore that!

Include user relationships in the call by default.

This certainly does solve the problem, but it's also pumping down a load of data I don't necessarily need. This would probably 2x the amount of bytes travelling along the wire, and in 8 out of 10 calls, that extra data isn't needed.

Have a separate /api/parent/:parentId/users call.

Another option that partially solves the issue: I need data from both the child and the user to format this view, so I'd still need to make the initial call I documented earlier. Semantically, it feels a bit odd to have this as a resource because I don't consider a user to be nested under a parent in terms of database topology.

Keep the original call, but add a query parameter to fetch the extra data

This comes across as the 'least worst' idea objectively, in terms of flexibility and design. Through the addition of the query parameter, you could optionally retrieve the relationship's data. This seems brittle and doesn't scale well to other endpoints where it could be useful though.

Utilize a Stripe expands-style query parameter.

Stripe implements the ability to retrieve all related records from an API endpoint by specifying the relations as strings. This is essentially the same as the above answer, but is scaled to all available API endpoints. I love this idea, but implementing it in a secure way seems fraught with disaster. For example, this is a multi-tenancied application, and it would be trivial to request userOrganization.user.organizations.users. This would retrieve all other organisations for the user, and their users! This is because my implementation of expands simply utilises the ORM of my choice to perform a database join, and of course the database has no knowledge about application tenancy!

Now, I do realise this problem could easily be solved by implementing a GraphQL API server, which I have done in the past, but unfortunately time and workload constraints dictate implementing a GraphQL-based solution is infeasible. As much as I like GraphQL, I'm not as proficient in that area as compared to implementing high quality traditional APIs, and the applications I'm working on at the moment are focusing on choosing boring technology, and not using excessive innovation tokens.

Furthermore, I do consider the conceptuals around REST APIs to be more of an aspirational sliding scale, rather than a well defined physical entity, because let's face it, the majority of popular APIs today aren't REST-compliant, even Stripe's isn't, and it's usually both financially healthier and feature-rich to choose a development path that results in a rough product that can be refined later, than aiming for a perfect initial release. All this said, I don't mind proposals or solutions to my problem that are "good enough". As long as they aren't too hacky! :)