Open-sourcing some Tildes-related code
Hello!
Some of you may know that I have been developing an iOS client for Tildes.
I’ve seen other devs like @wababa and @talklittle also working on awesome apps, and have seen some discussion there around collaborating on some of the core parts (scraping the website, etc.).
I’ve also been in touch with @efraimbart, who is working on the awesome Everything protocol project.
I’m sure there are other devs out there that I haven’t seen, and this post is a call-out for all of you.
Until an API is available, any similar project will face the same initial blocker: needing to scrape the website for data.
Is there any interest from the community on collaborating on that part? For example, developing a public library that encapsulates all of the interactions with Tildes and any client can talk to with a consistent API. Or, even better, collaborating on an official API for Tildes (which is, of course, itself an open-source project).
To that end, I’ve released some of the code I use to accomplish this in Surfboard. Maybe it can help someone build an iOS client, or inspire some of us to build a more generic solution :D. It’s far from perfect or complete, and isn’t even an actual package you can just import and use at this time, but at the very least it documents many of the endpoints and interactions.
Have a great weekend :)
An official API is the way to go. Eventually there should be client-side SDKs too, but those could be unofficial, to support different use cases and programming styles.
The following is what I had envisioned, and expressed to Deimos, some months before there was any sign Another Site would make any API changes. Of course the situation has changed, which may change priorities a bit, given the influx of users and developers to Tildes.
My plan back then was: I release an app using scraping, no API. Then based on learnings, I help build out the official API, with endpoints and JSON formatted to support apps from Day 1 of the API.
The alternative would have been to build the API alongside the app, but that would have resulted in many breaking changes as I discovered problems along the way.
(Example breaking change: Reading the main topics feed, I originally naively returned a top-level JSON array of topics. Then I realized it would be better to push the topics down into a
"topics"subkey, so we can return other stuff like"notifications"alongside it, therefore requiring only a single HTTP request instead of two.)Before the official API is made, it looks like there will already be several unofficial APIs/SDKs, based on the comments here. This is fine of course. [Edit: On second thought, it's fine given my assumption they are read-only APIs.] IMO there is not much added value consolidating to a single unofficial API/SDK. On the other hand, when it comes to the official one, we will need to coordinate and contribute our learnings to design an efficient and intuitive API.
Official API development can be split into a few phases:
It wouldn't be too hard to start on #1 already. We just need to agree on the format of the JSON.
#2 we need to come up with something better than cookies. So people can block bad clients. Probably OAuth 2, despite its flaws, as AFAIK there isn't anything newer that is a big enough improvement over OAuth. Please correct me if I'm wrong.
#3 will be harder because we haven't talked at all about bots and scripts on Tildes. That will require some serious planning. IMO we can't rely on the invitations system forever to keep bad actors at bay. The current lack of API creates enough of a barrier to entry that we don't need to worry that much about automated abuse today.
A note regarding the client SDKs: They need to have a practical license. For iOS apps, this means permissive: MIT or BSD or Apache. GPL does not really work for iOS apps because AFAIK the App Store does not let you publish GPL'd apps if you don't own the copyright. Worst case, as soon as someone contributes a pull request, assuming the contributor didn't assign ownership of their code to you, now the updated codebase isn't usable by anyone in publishing to the App Store. (Unless the contributors all form a single company to publish, or something cumbersome like that.)
I still personally have no plans to open source my app for reasons explained in my announcement, so for Android too, I would avoid any client-side SDK using the GPL. It would have to be Apache/MIT/BSD.
Agree, and that's why I wanted to open this conversation with the community. On further thought, I wish I could change the title of the post to better reflect that, but alas.
I don't really understand this part; there already exists a client for Tildes, the website. Shouldn't the features and UX offered there be the base for the official API ?
Without more context I suppose it's hard to judge but that just seems odd to me. I get the appeal of making fewer requests, but notifications and topics shouldn't be under the same endpoint IMO 🤔
It does mean that clients might build out incompatible features, although I suppose an official API doesn't prevent that either.
Of course :D
I don't believe this part is correct ? As far as I understand it, the only real restriction the GPLv3 imposes over MIT is that you have to release the source-code of anything making use of GPLv3 licensed code. So, I can see that in your case it would mean you would avoid an SDK licensed under GPL since you don't want to open-source your client, but I don't think it prevents you from publishing to the App Store. You still own the copyright of the derivative work you create using an SDK licensed under GPLv3, you simply are required to also release the source of that derivative work under the same license.
I am more than happy to be corrected by someone more familiar with the legalese than myself.
Tildes itself is licensed under GPL, and I imagine any official API/SDK would be as well, so it seems moot anyways.
Agree, and it seems like there's a decent amount of interest in contributing to an API, too, which is exciting.
OAuth is really the only sensible choice today, IMO. Others have different opinions 😄
Re: GPL not allowed in App Store, sources for that are: https://www.fsf.org/blogs/licensing/more-about-the-app-store-gpl-enforcement and https://www.fsf.org/news/2010-05-app-store-compliance - both from 2010, so a chance things have since changed, but as far as I'm aware, they have not changed. Further discussion at https://apple.stackexchange.com/questions/6109/is-it-possible-to-have-gpl-software-in-the-mac-app-store
Regarding official Tildes API and SDK, let's be precise: the API would be AGPL since it will be part of the server code. That does not mean clients consuming the API would be forced to be AGPL, as API output would not be covered by AGPL: https://www.gnu.org/licenses/gpl-faq.en.html#GPLOutput
A client-side SDK which consumes the API output, for the same reason, could have any license. It could be AGPL or it could be permissive like MIT/BSD/Apache. For the reasons I've stated, the permissive ones are the only ones that make sense to me.
Thanks for that ! Much appreciated. I’m far from an expert on the licenses. I didn’t realize the API being AGPL wouldn’t preclude a client SDK from using a different license.
any thoughts on using graphQL for the api?
i like the idea a lot, with a few caveats:
it's standard and open
it's flexible
it can easily be used in an app
versioning is nearly irrelevant, as is keeping track of versioned api endpoints
there are a fair number of mature libraries for all levels pf abstraction (server, client, cache, etc)
is very scalable and allows a single query's results to be generated by multiple servers/services
the data layer is json
as for caveats:
server side can be more complicated to implement
takes a little bit to grok the way a query needs to be structured in more complex cases
may need minor extensions for unbound/unknown count/non-paginated return sets
otherwise, if graphQL is chosen, dev can be arranged such that the server can send live data while the server is in development as ignoring certain parts of the input query and outputting structured data basically turns it into a ”normal” web-api¹.
please give a thought to this as it has nearly all the advantages of in-line sql, but without the security and development nightmare¹.
also, to whom it may concern: keep me in the loop on this and feel free to ping me on this api thing. i'm a low-level and embedded dev these days, but i have extensive experience in high performance data layers on websites with complex search, categorization, permutation, and caching requirements.
mostly real estate websites, though:
century21
coldwell banker
coldwell banker commercial
era
better homes and gardens
plus their international websites' data/cache/search layers.
thanks!
(oh, and /u/talklittle: thanks for rif, man: it helped keep me sane while stuck in bed with broken hands³ for far too long after the car accident)
^footnotes
1: i apologize for this last paragraph: it's late for me and my brain is not working with human languages well right now. if this sounds like a stupid paragraph, please let me know and i'll rewrite it as the underlying idea is firm and good² but i and my words are not working correctly at this moment.
2: and has worked well for me for a lot of major websites' data layers. sites like century21.com
3: among other things
GraphQL is interesting in concept. I think it can bring benefits when used by the right development team on the right project.
For Tildes, however, I need to echo some of the caveats you brought up. The big ones are: It's significantly more complicated to implement correctly on the server side, and there are security concerns. Tildes is a privacy-respecting site, and we need to choose technologies and site architectures that make it hard to access private data by default. GraphQL makes accessing all data very easy, which is good for some projects but not so desirable here.
Sure, a developer experienced with GraphQL will manage to avoid those security pitfalls, but we don't have as many of those around here.
Tildes' site implementation philosophy includes this piece: Use modern versions of simple, reliable, "boring" technology. In that context, I think we would prefer wider-adopted technologies for this site. Boring JSON endpoints would be preferred here, I think.
Simple restful APIs returning JSON data definitely feels more in line with Tildes. I’m not a huge GraphQL fan tbh, and I’m not sure the advantages it does offer are super relevant to the use case here. It also gives a lot more flexibility to the client, which can be super useful but also makes it harder to do things like try to ensure clients generally follow Tildes philosophy.
Hey, thanks for the call-out!
I will definitely be keeping an eye on this project and would love to contribute as well.
What are some possibile ways this can be multiplatform and framework agnostic? I'm thinking perhaps something built in Rust that can compile to to any language, and perhaps along with that we can collaborate on a spec for a simple scraper that we can implement across any framework using prexisiting scraping libraries.
I agree Rust is probably a great choice for an embedded lib for scraping. Outside of scraping, the most agnostic method would of course be simply collaborating to build an API for Tildes which is itself open-source. That part would likely require more discussion with the admins.. cc @Deimos
I could help with this potentially. Going an OpenAPI or Protobuf route would be ideal because it would enable clients in any language.
Full stack dev with typescript/express/nest/apollo experience happy to help if we get some direction on building out the backend API and there’s room for more contributors
I'd love to help out with this! I'll see if I can find some time to get a Rust API up and running, but it would definitely be amazing to have all the third party clients work off a shared API. Thank you so much for your work with the Swift app, especially making all source code public :D
Leaving a link here to Everything for Tildes, one potential API/project to contribute to. cc @tinyzimmer @brogeroni and others
It seems like there is some decent interest from the community in this, which is exciting :)
This is an early start.
The next step would be to extract this code into a module that conforms with IService from the Everything SDK - like has been done with Lemmy https://github.com/everything-gripe/lemmy-for-everything - so that tildes can be accessed via the API or imported into a js project.
I should also mention that this API is live now for anyone that would like to try it. The group endpoint is
https://tildes.z.gripe/r/<group>/hotwithout the tilde.I once upon a time started a kotlin android app with the goal of one day writing a ktor multi platform API client. Need an API to do that though.
Hi @earlsweatshirt! I'm currently working on an unofficial Tildes API which powers Backtick but is built in a generic way so any client should be able to utilize it. It's server side with a TypeScript/Express stack. I'm going to open source it once it's a little more feature complete (right now Posts and Profiles are supported) and would love community contributions when it's public.
In the end an official Tildes API will always be better, but my aim with it is to fill the gap until that happens.
Thanks for your comment ! Server-side seems like an interesting choice. It’s actually the main part I don’t love about Everything, having to basically proxy all requests through an extra server. I’m curious why you went that way ? Easier to make generic ? Particularly in the context of something specific to Tildes it seems to me like at that point working towards an official API would be a similar goal.
Good question! I have two main reasons:
Gotcha, that all makes sense. Btw I apologize, I edited my message to clarify my thoughts a bit before I noticed this response. Stale data strikes again 😆
Would you implement a cache or something of the sort? This way it’s not too much of a performance burden; one of the things I really enjoy about Tildes is the snappiness.
Yup! Right now the plan is to cache any calls to Tildes for a minute. Depending on usage, it has the potential to reduce overall load on Tildes (compared to if each individual was using the website).
I’m slightly confused about this. Tildes calls are authenticated — does this mean using your app would involve sending my credentials to your server rather than directly accessing tildes?
Right now it only supports publicly available data that does not require any credentials. Any POST endpoints where you submit data like comments or topics are already supported natively via Tildes's undocumented API so mine wouldn't need to support that (I might just create documentation for it).
So the only thing left is authenticated GET requests like notifications. The way to tackle this is to accept the authentication cookie that Tildes provides after logging in and just pass it through when making the call to Tildes (e.g. no password is sent). Backtick, and I presume the other apps, work similarly to this: it saves the cookie on the device and uses that to call load Tildes in the background.
I already know that passing the Tildes authentication token to a server will raise some alarm bells for some people, but I hope that making it open-source will ease people's minds. Ultimately, it's all about trust, regardless if something is on a device or a server, open-source or not.
To your point here, it really doesn’t matter much how authentication is implemented. At the end of the day, if you’re going to put your credentials into something other than the Tildes website itself, you’d better trust them :) Just because my app doesn’t proxy the requests through an extra server doesn’t mean I don’t upload everyone’s passwords or auth cookies to my server for malicious use (LittleSnitch users, imma need you to back me up here when I say I don’t 😆).
The advantage to doing it locally is that as an end user you can see your outgoing traffic and where it sends your credentials. Open-sourcing the server part should offer close to the same peace of mind, especially if you can verify the code is what’s actually deployed.
Well put!
I mean the fact that we can simply packet dump the communications is a big barrier to what you are taking about. It’s a big difference when I know that my credentials go directly to tildes server rather than somewhere else — I’m waaaaaaay more comfortable with that.
Yeah, I touched on that point too. I actually agree with you in principle, I’m never a fan of proxying requests. I simply wanted to point out that 1) you’d better trust anyone you hand your credentials to, whether they told you they are proxying your requests or not and 2) if the server portion is open-source it should be easier to trust it than otherwise. You might not trust @wababa to secure his server, but what if he deploys on Cloudflare, for example? Do you trust them? Many services you probably do trust certainly trust Cloudflare.
I mean yeah, I do kind of trust cloudflare. I deploy through them and other similar services like cloudfront all day long. I know their general governance and disclosure practices. I know they definitely take security seriously and have layers of accountability as well as signed contracts and defined obligations to me as a customer.
NGL i think that’s kinda sketchy. Would you disclose the fact that users’ authentication info is sent to a non-tildes server anywhere? I don’t feel super comfortable with this mechanism, am I alone in that?
Yeah, I have no problem disclosing that.
I'm curious to know what the difference is to you between storing the cookie on a device vs passing it through on a server. In this case, the cookie isn't even stored anywhere on the server while the device has to store it, so to me, it's technically even more secure from a malicious developer. Like @earlsweatshirt mentioned, you can be just as malicious via an app vs a server.
In the end, I'm okay knowing that no matter what the argument is, some people will feel that sending the token as a pass-through to a server is scarier. The best I can do is open-source the code and be as transparent as possible. The rest is on the user to decide whether they want to use the app/api or not.
Trusting your server is different than trusting my device. How do I know whether you are capable of securing a server? I’m more confident in Apple’s ability to keep out hackers than I am in some random person keeping their server secure.
In essence, it’s not so much that I distrust you personally, than it is adding a second point of failure in the chain of custody for my credentials.
It’s definitely less secure than storing it on my device.
Ah, that makes sense to me and is a totally fair perspective.
I haven't implemented any of this yet so there is still lots of room for adjustment. Technically, it would still be possible to get things like notifications manually without requiring credentials, but it would require a good amount more code. But if the community feels that it's worth it, I definitely don't mind trying to go down that path with the understanding that some things won't be possible (off the top of my head inbox functionality is definitely not possible).
In any case, I appreciate you voicing your concern!
I have zero webdev experience, but I think people developing a 3rd party ecosystem for Tildes are treasures. At the moment, I'm pretty happy with the mobile browser interface (since this site is pretty clean and simple to begin with) but I will absolutely check out these projects as a user. Godspeed y'all.
I moved away from coding a couple years ago but once a backend developer, always a backend developer.
If there is a common plan to write some APIs, I'm in, even if just to give my insight on security/scalability etc. etc.
Just a thought from really amateur programmer - what about collaborating on implementing actual Tildes API instead of making scraper for web? Wouldn't that be better for the whole project andfuture 3rd party apps and also making yourown apps more futureproof (against web interface changes etc.)?
I agree completely ;) This post is basically a call to see if any devs are interested in collabing on such a thing.
I definitely should read more before commenting... :-) Thanks for pointing me the right way.
I'm a more Frontend focused dev but I'd love to help get an api up and running
I’m a budding front-end developer working towards building industrial skills and this sounds like a great opportunity to get involved in.
I’d love to be apart of this project to contribute any way that I can.
What if the tildes website was just one interface for a decoupled backend? In other words, maybe just think of the website, api and database as three discrete entities.
Each component would require different resources, expertise and funding of course.