The Tildes code is now open-sourced
Disclaimer: this post/information will probably only be interesting to more technical people
It's long overdue, but Tildes is now open-sourced: https://gitlab.com/tildes/tildes
There's still a ton to do in terms of writing more documentation, filling out the issue tracker with known issues/plans and so on, but it should be ready enough to get started.
I'm planning to make a post on the Tildes blog tomorrow or Thursday announcing this more formally, but I'd like to keep it a bit quiet until then, in case there are any issues or major gaps in the docs discovered when some of you start looking at it. I'd especially appreciate it if any of you would like to try setting up a local development version and let me know how it goes, and if the instructions all make sense.
There are two new pages on the docs site as part of this as well:
- Development Setup - describes how to do the initial setup necessary to get a local development version of the site running
- Development - a general page with information and instructions related to a lot of aspects of site development. It's very scattered right now and I intend to split it into multiple pages eventually, but that will require some more work on the blog to support being able to organize it well. There are still some definite gaps in here, but it should describe most of the major pieces.
I'd appreciate any feedback about the documents, code, etc. As mentioned, I'd be especially grateful if some of you want to try setting up a local development version by following my instructions, and give feedback about if any parts of the process are unclear, incomplete, or otherwise need more work. Please feel free to ask any questions you have as well.
Good time for a reminder that Tildes is entirely supported by donations.
Quick link to Patreon: https://www.patreon.com/tildes
Taking advantage of the fact that this is the top comment:
It is the smaller, more sustainable pledges that make the projects survive in the long time, so even if you don't have the means to make a big contribution, even $1 a month is a big contribution in its own right. I myself can not pledge more ATM (TRY has stalled and USD is becoming more expensive every day), so I'm pledging a dollar a month, which I can happily forget about. And not only, but thus I can support multiple projects instead of one only.
So if you can pay for your internet, electricity, and mobile data; and desire that Tildes continue on and become a great, high quality community, you most probably have the means to help sustain it, even if that help is little. In Turkish we say: "Damlaya damlaya göl olur" = "Many drops make a lake".
I started getting back on my feet a little bit before I found out about tildes so I've been meme-ing a $4.20 monthly pledge to tildes and some friend projects.
I figure tildes and those buddies are going to be more frugal with that $4 than i would be now so it's probably for the better ahah
Vagrant containers, web frameworks, and templating languages, oh my!
This all looks very civilized. Haven't dug into it yet, but the documentation looks fantastic so far! Thanks so much for your work on this.
I can't wait to explore this, it looks like I can learn a lot about the multitudes of abstractions being used here, and "how the kids are doing things these days".
Haha, this is a lot closer to "how the kids were doing things 5-10 years ago". I'm definitely leaning more towards older technology than what's trendy now. Trendy stuff now would probably be Docker, Kubernetes, React, Vue, and a couple others.
I was actually just considering a snarky "what, no Docker container??" comment, but you beat me to it.
Real talk: should I look into testing docker on my VPS? Been looking into ways to make setups easier and to separate instances and apps from each other.
Given the stated goals of "make setups easier" and keeping "separate instances," probably. But if whatever you're doing isn't stateless, then you need to be mindful about where you're storing persistent data. There was a good talk at Dockercon 2018 about this. Third link down. (You might need to make a Docker account to watch.)
I'll take a looknat that, thanks :)
I feel like most web software these days follow the car salesman meme approach: "this bad boy can fit so much fucking javascript in it".
I'm not sure when 80% of the web turned into IKEA; we'll just give you all the parts and you put everything together yourself on your own time. It's kind of gross.
On the other side, just using on JS like if it was the same language of the late 90 denotes only ignorance.
Don't get me wrong, you (generic) can still find your job in which you just need to learn that one technology and be good with it for life. But luckily or not, IT is a constantly evolving field and you should really have a mindset that allow you to be comfortable with learning something new every 6 months.
There is no Swiss army knife tech in IT. When you close yourself against technologies, you hurt only yourself.
To be clear, I spend half or more of my time in big javascript projects; I'm intimately familiar with current JS. There are a lot of great things about it, but I think that it's really overengineered for a lot of things, and there can be significant drawbacks to it if it's done poorly. Many sites don't actually need to be an SPA, and rely on javascript for things that there is no reason to rely on javascript for.
Oh yeah, I totally agree with that.
With HTML5 and CSS3 there are lots of things that can be done without JS at this point.
Also lots of people just learn "react" or any other framework and apply it to literally everything but that's just how web development always worked. 90% (personal perception) of the "professionals" stay at a junior level and never really become something more, despite their company promoting them just because time passes.
They never ask themselves the "how" or "why" about the tools thay're using.
But that's not the technology's fault (being it JS, PHP, Ruby, Python, Java, whatever else) but the people's fault that just think (hope maybe) that they found the panacea for every problem in the technology they learned.
I 100% agree... I kind of wonder if you think that I think something else? You phrased your first response to me as "on the other hand", but this is the exact point I was bringing up.
To be clear, I never said anything negative about Javascript, I just talked about this exact issue. People build stuff badly because they don't know how to not do that. That's never the fault of the language being used, with the exception of PHP, which is just garbage.
I suppose it was a stupid misunderstanding :)
I don't think it was stupid; it led to this conversation! I just wanted to make sure that we were on the same page; just I was on the back of the page and you were on the front.
Lol, I kid, I kid... On the other hand, I feel sooo old all of the sudden. Why does no one use frames, imagemaps, and the marquee tag anymore?
Frames, as in iframes? Those are rather limited and have very strict security surrounding them, making them non-responsive and non-interactive with the rest of the web page. In general, they're clunky and difficult to work with, so they're typically relegated to more specific roles now that take advantage of their restrictive nature (e.g. Stripe payments). If you're talking about actual frame tags, then there were apparently a number of problems with them that really don't work well with modern, dynamic page content. Marquee tags inherently affect readability by forcing movement of text (consider visually-impaired users in particular), so they directly oppose modern (good) design practices. I'm not even sure what you mean by imagemaps, however, so I have no answer for you on that one.
Heh, I was being facetious. Yes, I was talking about frames, as in the <frameset> and <frame> tags. The <marquee> and <blink> tags were never good ideas. Imagemaps defined regions of an image that were hyperlinks. It was silly and convoluted, but it allowed you to make different parts of an image different clickable things, without having to cut your image into a bunch of carefully designed pieces, then arrange them using tables with colspan and rowspan attributes.
The last time these things were relevant was before CSS was invented, before the browser wars, like, Netscape 3.0 era or so.
Ah, gotcha! Sorry, it's difficult to gauge intent in a text-only medium :)
It doesn't help you're clearly talking to a wise old greybeard speaking an arcane and long forgotten dialect. :)
You're going to love Vagrant. I haven't really touched Docker or Kubernetes, but my understanding is that they solve a completely different problem space. I use Vagrant heavily in my own work and it makes things so much easier :)
Docker solves a lot of the same problems as Vagrant, but also solves another problem (that Vagrant is starting to move into as well) which is deployment.
To be clear, I still love and use Vagrant a lot, and I'm a bit bummed that I'm moving away from it professionally (we're starting to replace it with Docker, so that it meets deployment needs as well).
Finally! Not like I had the knowledge to contribute but now other people two are smarter than me can. Also stallman can now use tildes :)
Also can we add source code link at the bottom of the page?
Yes, I'll add links to the source code and issue tracker / roadmap later today or tomorrow, thanks for the reminder.
Neato. I'm setting it up to hack on it right now.
Edit: The instructions should include how to configure the hostname. It keeps redirecting to tildes.net.
Edit: After replacing all instances of tildes.net, I'm getting 400 Bad CSRF Origin when trying to log in from a non local hostname.
Edit: The external port was different from the internal one used in dev config, to fix:
Edit: I'll probably contribute to docs later, but to add invites to a user:
That's weird, it shouldn't be doing anything with tildes.net except for a very few links like the Blog/Docs/etc. ones in the footer, it should all just be going through localhost. All links on the site were trying to go to tildes.net?
I've fixed the redirect issue by modifying the hostname in
salt/salt/nginx/tildes.conf.jinja2. After doing that, internal links are fine, but the footer links todocs.tildes.netandblog.tildes.netare obviously hard coded.But yeah, in the default configuration, accessing it from
localhost:4443works fine. I'm accessing it from another hostname.Edit: I'm dumb, forgot to run
vagrant provisionWould you mind explaining exactly what you did? I'm actually having some trouble trying to do the same. I changed server_name in
salt/salt/nginx/tildes.conf.jinja2to<my IP address> localhost, but I'm still getting redirected to tildes.net. I also tried changing the www. redirect, which I don't think is relevant but was the only other place I could find tildes.net.I just made a mirror of the repository at notabug.
That's odd, sounds like the initial data didn't get inserted into the database for some reason. I'll look and see if I can tell why that might be happening, thanks. You could try running from a command prompt (inside a SSH session on the vagrant machine):
Just a warning, so it shouldn't matter. Strange that it didn't run in the first place though, I'll see if I can figure out why.
Oh? I didn't disable them deliberately, let me see why it's not working. Does it give a more specific error or anything, or just says merge requests aren't enabled?
Ah ha, thanks. Try it again now.
insert-dev-data isn't running because final-setup.initialize-db fails.
The output from which is:
I don't really understand vagrant so not sure what's causing this yet.
Thank you, that tells me exactly what the issue is: I deleted all the alembic migration files when I squashed the commit history down to a single commit, so now git isn't even creating the folder because it's empty, and that's crashing the script.
Huge kudos for open sourcing it already!
I have a background in devops, would you be interested in me taking a look and dockerizing the setup, maybe submitting a PR with a working dockerfile? (Or docker-compose, if required)
Thanks, I appreciate the offer. I don't have any interest in using Docker for the site itself, but what application are you thinking of using it for? An alternate setup method for local dev versions?
Mostly for local development and testing, yeah. Vagrant is nice and all, but I find the feedback cycle way too long whenever I have to work on with it.
I love Docker, and use it extensively both at home and at $dayjob, but it might be more trouble than it's worth in this case.
From a brief look at the Vagrant + Salt setup, you'd have to duplicate nearly all of the configuration defined by Salt (things like teaching Postgres that it should use RabbitMQ as a message broker for NOTIFY commands). If the plan was to use Docker in production, I think it'd be worthwhile, but if the production stack continues to use Salt for config management, a Docker setup will probably be a perpetual second-class citizen and source of unnecessary duplication.
It was two days that i was pondering if I should have sent a message to you asking about the open source day. I was growing worried, glad to finally see it! :)
I think that people try to abstract things too much. Topics and comments serve totally different purposes, and while you can find some common things between them, overall they'll almost never be treated the same way. You just end up moving the complexity somewhere else, and probably even end up with more complexity in the end than just treating them as separate things in the first place. Something I notice in general is that people are way too worried about any kind of repetition and will try to eliminate it, but repeating similar code really isn't that bad in most cases. It will often end up diverging later anyway, in which case your abstraction makes it more complex than it would have been.
I'm coming from spending the last few years working on reddit code, which was way over-abstracted by trying to treat almost everything as a
Thingthat has upvotes and downvotes and some other common attributes, and it just turned into a giant disaster. The number of subscribers to a subreddit is stored insubreddit._upsbecause... subscriptions are... kind of... like upvotes, I guess. And the subreddit's "popularity" is stored insubreddit._downsbecause... well, because there's no downvotes on subreddits and we needed somewhere else to store a number, so we might as well, and so on.The reddit code is an excellent example of people trying to be far too clever. It has some pretty neat ideas in it and was probably great for prototyping the site in originally, but it's completely insane to still be using it as the foundation for one of the top sites on the internet.
Reddit example is not an example of why applying too much DRY approach or abstraction in general is bad, but of clear bad design choice.
You don't store apples in the containers that says "oranges" to not waste a container.
You create a basic container without labels and then extend it with the additional attributes it needs.
Having said so, abstraction can absolutely become a problem if you just apply it blindly. Knowing when to use a paradigm or strategy is what differentiate junior from senior developers.
Probably that decision was taken back in the early day and no one challenged it in the following years.
"Clever" is something that's painful to deal with. I'm trying to avoid that as much as possible in my current work because it's just such a massive headache trying to deal with the steaming pile of technical debt that inevitably arises. No code is completely free of technical debt and it's pretty much a guarantee that there will be some pretty bad parts of my own, but I'm at least going to try to mitigate it as much as possible. I had some pretty bad messes to deal with in my previous work and that was for a small startup, so I don't even want to begin to imagine what the reddit code base looked like.
I think it's part of people's evolution as a programmer. When you've never had to maintain a large, complex system for a long period of time, solving things in clever ways is appealing. It's kind of exciting to figure out a fancy approach to a problem, other people will be impressed if you show your code to them, and so on. If you're just a hobby programmer or in school, you never have to deal with the longer-term maintenance impacts of that clever code, you finish with the assignment/project/whatever and never look at it again.
As you get more experience though, you start to recognize the benefits of having code written clearly and simply. Yeah, it's boring and it's never going to impress anyone that looks at it, but that's the point—it's easy to understand, debug, and modify when you need to, even if you've completely forgotten how it works.
This is something I have a hard time explaining to others. It's great and amazing that an intern has come up with this fancy solution, but it's too fancy, and will be extremely inflexible later on. It's clever now, but will be a bane later.
Reminds me of an intern I worked with, many years ago, and a function they wrote with a comment like "// this is pretty much a masterpiece of recursion".
Unfortunately many startups use students/novices at the beginning because they're cheap/naive, and they build exactly the kind of code that leads to these long-term maintenance issues that are a nightmare for later employees to deal with (and rewriting/refactoring always takes a back seat to the "feature customer needs yesterday").
Hot damn Deimos, my clients project just launched today and I was planning on catching up on paperwork tomorrow. :)
Oh, python stuff! This is something I want to sink my teeth into.
Thanks!
How do you envision collaborative development? Do you foresee lots of contributions with site users contributing new features, or would you rather keep a small team?
I don't really have a particular vision for it. Significant contributions can be a lot of work, so my expectation is that there will probably only be a small group of people that do larger changes, but I'd be happy to see lots of smaller contributions as well (for example, minor tweaks/cleanups of things I'm not great at like sysadmin-side stuff, frontend, etc.). Overall, I'd be happy even if there aren't any contributions anyway, just because I like doing the development publicly.
Typed Python! You're killing it, Deimos!
Hurray! :D
What's the caching situation like? It looks like everything is redis? Are you passing through queries/objects through to them automagically or is the caching not in place yet?
No caching yet. Postgresql is extremely fast as long as you have good indexes (and has its own internal caching-like behavior). Eventually I'll probably need to add some, but it likely shouldn't be necessary for a long time.
From a quick scan of the code, only place I saw Redis used was for a bloom filter of commonly hacked passwords, while RabbitMQ is used as a broker for notifications flowing through Postgres. Are there any other big moving parts?
Redis is also used for session storage and rate-limiting. The RabbitMQ usage is huge overkill for now (and was honestly implemented more as a proof of concept than something that's necessary yet), but it will be useful for a lot of scraping/metadata uses as well as some other things in the future.
Those are pretty much all the major technologies involved though, yes.
True, Postgres is quite solid. Though even for small Django websites I've used Redis for custom data structures, so at least that's available (even if it's not specifically used for caching).
Most sites end up with a background processor that needs to be turned into a message queue. Lot of sites use Celery or Resque and then have to code shells for whatever they want and have to deploy it with the main web app. Kinda nice to be able to run a message queue that you can just hook into and spin up a separate service for.
Curious to see the scale of Tildes so far.
Yeah, that's pretty much what rabbitmq is there for. It's "connected" to
NOTIFYevents from postgresql through pg-amqp-bridge. That way, postgresql sends messages liketopic.created,topic.editedand so on when the relevant table has rows inserted/updated/deleted, and background processor jobs can subscribe to the queues for the events they care about.Like I said, it's definitely overkill for what I'm using it for at the moment, but it has some great possibilities for the future.
That's awesome, didn't realize that existed!
So, I am learning Python and I definitely want to contribute in any way possible because this is definitely a project I can get behind. However, I have been more on the data science track with my self-teaching. Would you recommend any resources that I could look at and work through that would give me a better understanding of how Tildes is built?
If you've never done any web development, a great place to start would be the Pyramid tutorial. That's the web framework underneath everything, and the tutorial takes you through Jinja2 templates and SQLAlchemy database queries, both of which Tildes uses.
Thank you for this! I will add this to my list for a resource to work through!
Hmm, I think it's difficult because web applications have so many different parts involved that it's pretty hard to say how you could start. Something I'd like to do is write up kind of a "walkthrough" of implementing a whole feature, explaining everything that needs to be changed as part of it. I think that could be very helpful to be able to give people a better understanding of the different pieces and how they interact with each other.
I think it'd be helpful to start simpler than that - explain all the moving parts of the system as-is, before I make any changes to it.
When I worked at Amazon, one of the things they had new hires for AWS do is make some requests to S3 - create a bucket, upload a file to it, then download that file, etc - then use the request IDs and logs to trace the path each request took through the various backend services.
I think a similar high-level walkthrough would be helpful here. When I click "submit" on a new post, what happens? When someone comments on my post, what happens? What triggers the orangered notification for me, how do the "X new comments since your last visit" counters get maintained, etc.
That's a great suggestion, thanks. I added it as an issue on the docs site to make sure I remember it as something I should do: https://gitlab.com/tildes/tildes-static-sites/issues/1
This is something I just did for a new codebase at work, we call them “codelabs” at Google. I can look into at least writing the outline for a Tildes codelab if you want to make an issue for tracking it.
Thank you for responding!
I will keep up reviewing the code and tinkering here and there to get a better understanding of how everything ties together. I hope I can contribute in any way possible eventually!
Without getting extremely lucky with a full-feature tutorial that uses the exact same stack as this site, you should look at learning each piece separately.
Start with the parts of the site that you'd want to eventually contribute to, and start reading up. If it's the backend that you're most interested in, start there with generic Python tutorials to get the syntax solid in your mind, and then move onto tutorials for Pyramid. Then, Jinja so changes you make can be rendered on the site. If you're not interested in learning the database too much, then don't spend a lot of time learning Postgres - just accept that it exists and is running in the background. Same with the other "ops"-y stuff, like Nginx.
Thank you for responding!
I will take your advice and go through things I know and am interested in step-by-step. Everything is pretty new to me but even stuff I don't think I would be interested in has something that I can learn that should help me elsewhere.
That's definitely true, and a good way of thinking. Take it one piece at a time, and if something seems fun and interesting, go for it!
Fantastic, thanks for open sourcing!
Update: setup the environment on MacOS 10.13, Virtualbox 5.2.12, and vagrant 2.1.2. I got some error messages after the first
vagrant upand the default user/pass in the docs didn't work. I followed the instructions for adding a user and I was able to log in to that without issue.I'll repro the steps in another env to see if I was doing it wrong.
This is the same problem I had, see below.
If you create the folder
versionsundertildes/alembicthenvagrant upshould just work.Had to recreate my vagrant box today and can confirm that having the versions folder there before running
vagrant upworked great.Ah, good find. I ended up creating that file when I went to perform a table alteration; the alembic command complained until I created it manually.
This is fantastic - great use of typed Python throughout. Going to spend a few days going through the code. Thank you.
Super exciting - congrats on getting this all taken care of.
Just in time for my wife and kids to be gone for a week, so I think I'll actually have a look at it this coming week!
Good stuff man. Aside from helping with documentation and issues if I'm hacking around and find something useful or fun to implement, will you deny the pull request based on the road map or is it OK to try?
It's hard to say without knowing what it is. As it says in the CONTRIBUTING doc, it's best to "claim" (or create) an issue for the thing you're going to work on first, so that you know it's actually wanted. If you just want to tinker around and won't be disappointed if you do a bunch of work that doesn't get included, that's fine of course. It might be best to at least send me a quick message and ask before working on anything significant if you're hoping it'll be included in the site though.
🎉🎉🎉
Awesone, I’m excited to read through it and (possibly) contribute! The development documentation looks very thorough.
Wooooo! Awesome, I can't wait to dig in. Thanks for releasing it! :D
This is an exciting step for the site! I'm looking forward to digging into this later.
Fantastic organization, really cool to finally be able to see the code.
Congratulations on the release.
Does Tildes have any desktop or mobile apps?
The mobile site works pretty well. What features would you look for in an app?
I was only curious if it already existed for reading it's source code. I have much appreciation for the current website that just works and actually already feels like a native app (compared all of reddit's shitty mobile apps/webites).
Easy sharing to app. It can make posting really easy, and might even encourage more users to post, and more often.
Anyone got time to give me some feedback on: https://gitlab.com/jms301/tildes/commit/f1d5962e82f5e4769334919271679c237019bb22
I have zero experience with Pyramid, SQLAlchemy or Intercooler so I'm expecting obvious mistakes, don't worry about my feelings!
Just that I'd rather poke code first then read the docs and tutorials later when I've had a go. :)