232 votes

The Tildes code is now open-sourced

Disclaimer: this post/information will probably only be interesting to more technical people

It's long overdue, but Tildes is now open-sourced: https://gitlab.com/tildes/tildes

There's still a ton to do in terms of writing more documentation, filling out the issue tracker with known issues/plans and so on, but it should be ready enough to get started.

I'm planning to make a post on the Tildes blog tomorrow or Thursday announcing this more formally, but I'd like to keep it a bit quiet until then, in case there are any issues or major gaps in the docs discovered when some of you start looking at it. I'd especially appreciate it if any of you would like to try setting up a local development version and let me know how it goes, and if the instructions all make sense.

There are two new pages on the docs site as part of this as well:

  • Development Setup - describes how to do the initial setup necessary to get a local development version of the site running
  • Development - a general page with information and instructions related to a lot of aspects of site development. It's very scattered right now and I intend to split it into multiple pages eventually, but that will require some more work on the blog to support being able to organize it well. There are still some definite gaps in here, but it should describe most of the major pieces.

I'd appreciate any feedback about the documents, code, etc. As mentioned, I'd be especially grateful if some of you want to try setting up a local development version by following my instructions, and give feedback about if any parts of the process are unclear, incomplete, or otherwise need more work. Please feel free to ask any questions you have as well.

94 comments

  1. [3]
    talklittle Link
    Good time for a reminder that Tildes is entirely supported by donations. Quick link to Patreon: https://www.patreon.com/tildes

    Good time for a reminder that Tildes is entirely supported by donations.

    Quick link to Patreon: https://www.patreon.com/tildes

    41 votes
    1. [2]
      cadadr Link Parent
      Taking advantage of the fact that this is the top comment: It is the smaller, more sustainable pledges that make the projects survive in the long time, so even if you don't have the means to make...

      Taking advantage of the fact that this is the top comment:

      It is the smaller, more sustainable pledges that make the projects survive in the long time, so even if you don't have the means to make a big contribution, even $1 a month is a big contribution in its own right. I myself can not pledge more ATM (TRY has stalled and USD is becoming more expensive every day), so I'm pledging a dollar a month, which I can happily forget about. And not only, but thus I can support multiple projects instead of one only.

      So if you can pay for your internet, electricity, and mobile data; and desire that Tildes continue on and become a great, high quality community, you most probably have the means to help sustain it, even if that help is little. In Turkish we say: "Damlaya damlaya göl olur" = "Many drops make a lake".

      11 votes
      1. ruspaceni Link Parent
        I started getting back on my feet a little bit before I found out about tildes so I've been meme-ing a $4.20 monthly pledge to tildes and some friend projects. I figure tildes and those buddies...

        I started getting back on my feet a little bit before I found out about tildes so I've been meme-ing a $4.20 monthly pledge to tildes and some friend projects.

        I figure tildes and those buddies are going to be more frugal with that $4 than i would be now so it's probably for the better ahah

        6 votes
  2. [20]
    pseudolobster Link
    Vagrant containers, web frameworks, and templating languages, oh my! This all looks very civilized. Haven't dug into it yet, but the documentation looks fantastic so far! Thanks so much for your...

    Vagrant containers, web frameworks, and templating languages, oh my!

    This all looks very civilized. Haven't dug into it yet, but the documentation looks fantastic so far! Thanks so much for your work on this.

    I can't wait to explore this, it looks like I can learn a lot about the multitudes of abstractions being used here, and "how the kids are doing things these days".

    30 votes
    1. [17]
      Deimos Link Parent
      Haha, this is a lot closer to "how the kids were doing things 5-10 years ago". I'm definitely leaning more towards older technology than what's trendy now. Trendy stuff now would probably be...

      I can't wait to explore this, it looks like I can learn a lot about the multitudes of abstractions being used here, and "how the kids are doing things these days".

      Haha, this is a lot closer to "how the kids were doing things 5-10 years ago". I'm definitely leaning more towards older technology than what's trendy now. Trendy stuff now would probably be Docker, Kubernetes, React, Vue, and a couple others.

      28 votes
      1. [4]
        Kraetos Link Parent
        I was actually just considering a snarky "what, no Docker container??" comment, but you beat me to it.

        I was actually just considering a snarky "what, no Docker container??" comment, but you beat me to it.

        12 votes
        1. [3]
          clerical_terrors Link Parent
          Real talk: should I look into testing docker on my VPS? Been looking into ways to make setups easier and to separate instances and apps from each other.

          Real talk: should I look into testing docker on my VPS? Been looking into ways to make setups easier and to separate instances and apps from each other.

          5 votes
          1. [2]
            Kraetos Link Parent
            Given the stated goals of "make setups easier" and keeping "separate instances," probably. But if whatever you're doing isn't stateless, then you need to be mindful about where you're storing...

            Given the stated goals of "make setups easier" and keeping "separate instances," probably. But if whatever you're doing isn't stateless, then you need to be mindful about where you're storing persistent data. There was a good talk at Dockercon 2018 about this. Third link down. (You might need to make a Docker account to watch.)

            8 votes
      2. [7]
        aphoenix Link Parent
        I feel like most web software these days follow the car salesman meme approach: "this bad boy can fit so much fucking javascript in it". I'm not sure when 80% of the web turned into IKEA; we'll...

        I feel like most web software these days follow the car salesman meme approach: "this bad boy can fit so much fucking javascript in it".

        I'm not sure when 80% of the web turned into IKEA; we'll just give you all the parts and you put everything together yourself on your own time. It's kind of gross.

        5 votes
        1. [6]
          crius Link Parent
          On the other side, just using on JS like if it was the same language of the late 90 denotes only ignorance. Don't get me wrong, you (generic) can still find your job in which you just need to...

          On the other side, just using on JS like if it was the same language of the late 90 denotes only ignorance.

          Don't get me wrong, you (generic) can still find your job in which you just need to learn that one technology and be good with it for life. But luckily or not, IT is a constantly evolving field and you should really have a mindset that allow you to be comfortable with learning something new every 6 months.

          There is no Swiss army knife tech in IT. When you close yourself against technologies, you hurt only yourself.

          4 votes
          1. [5]
            aphoenix Link Parent
            To be clear, I spend half or more of my time in big javascript projects; I'm intimately familiar with current JS. There are a lot of great things about it, but I think that it's really...

            To be clear, I spend half or more of my time in big javascript projects; I'm intimately familiar with current JS. There are a lot of great things about it, but I think that it's really overengineered for a lot of things, and there can be significant drawbacks to it if it's done poorly. Many sites don't actually need to be an SPA, and rely on javascript for things that there is no reason to rely on javascript for.

            3 votes
            1. [4]
              crius Link Parent
              Oh yeah, I totally agree with that. With HTML5 and CSS3 there are lots of things that can be done without JS at this point. Also lots of people just learn "react" or any other framework and apply...

              Many sites don't actually need to be an SPA, and rely on javascript for things that there is no reason to rely on javascript for.

              Oh yeah, I totally agree with that.
              With HTML5 and CSS3 there are lots of things that can be done without JS at this point.
              Also lots of people just learn "react" or any other framework and apply it to literally everything but that's just how web development always worked. 90% (personal perception) of the "professionals" stay at a junior level and never really become something more, despite their company promoting them just because time passes.

              They never ask themselves the "how" or "why" about the tools thay're using.

              But that's not the technology's fault (being it JS, PHP, Ruby, Python, Java, whatever else) but the people's fault that just think (hope maybe) that they found the panacea for every problem in the technology they learned.

              4 votes
              1. [3]
                aphoenix Link Parent
                I 100% agree... I kind of wonder if you think that I think something else? You phrased your first response to me as "on the other hand", but this is the exact point I was bringing up. To be clear,...

                I 100% agree... I kind of wonder if you think that I think something else? You phrased your first response to me as "on the other hand", but this is the exact point I was bringing up.

                To be clear, I never said anything negative about Javascript, I just talked about this exact issue. People build stuff badly because they don't know how to not do that. That's never the fault of the language being used, with the exception of PHP, which is just garbage.

                2 votes
                1. [2]
                  crius Link Parent
                  I suppose it was a stupid misunderstanding :)

                  I suppose it was a stupid misunderstanding :)

                  2 votes
                  1. aphoenix Link Parent
                    I don't think it was stupid; it led to this conversation! I just wanted to make sure that we were on the same page; just I was on the back of the page and you were on the front.

                    I don't think it was stupid; it led to this conversation! I just wanted to make sure that we were on the same page; just I was on the back of the page and you were on the front.

                    3 votes
      3. [5]
        pseudolobster Link Parent
        Lol, I kid, I kid... On the other hand, I feel sooo old all of the sudden. Why does no one use frames, imagemaps, and the marquee tag anymore?

        Lol, I kid, I kid... On the other hand, I feel sooo old all of the sudden. Why does no one use frames, imagemaps, and the marquee tag anymore?

        3 votes
        1. [4]
          Emerald_Knight Link Parent
          Frames, as in iframes? Those are rather limited and have very strict security surrounding them, making them non-responsive and non-interactive with the rest of the web page. In general, they're...

          Frames, as in iframes? Those are rather limited and have very strict security surrounding them, making them non-responsive and non-interactive with the rest of the web page. In general, they're clunky and difficult to work with, so they're typically relegated to more specific roles now that take advantage of their restrictive nature (e.g. Stripe payments). If you're talking about actual frame tags, then there were apparently a number of problems with them that really don't work well with modern, dynamic page content. Marquee tags inherently affect readability by forcing movement of text (consider visually-impaired users in particular), so they directly oppose modern (good) design practices. I'm not even sure what you mean by imagemaps, however, so I have no answer for you on that one.

          8 votes
          1. [3]
            pseudolobster Link Parent
            Heh, I was being facetious. Yes, I was talking about frames, as in the <frameset> and <frame> tags. The <marquee> and <blink> tags were never good ideas. Imagemaps defined regions of an image that...

            Heh, I was being facetious. Yes, I was talking about frames, as in the <frameset> and <frame> tags. The <marquee> and <blink> tags were never good ideas. Imagemaps defined regions of an image that were hyperlinks. It was silly and convoluted, but it allowed you to make different parts of an image different clickable things, without having to cut your image into a bunch of carefully designed pieces, then arrange them using tables with colspan and rowspan attributes.

            The last time these things were relevant was before CSS was invented, before the browser wars, like, Netscape 3.0 era or so.

            7 votes
            1. [2]
              Emerald_Knight Link Parent
              Ah, gotcha! Sorry, it's difficult to gauge intent in a text-only medium :)

              Ah, gotcha! Sorry, it's difficult to gauge intent in a text-only medium :)

              1 vote
              1. super_james Link Parent
                It doesn't help you're clearly talking to a wise old greybeard speaking an arcane and long forgotten dialect. :)

                It doesn't help you're clearly talking to a wise old greybeard speaking an arcane and long forgotten dialect. :)

                6 votes
    2. [2]
      Emerald_Knight Link Parent
      You're going to love Vagrant. I haven't really touched Docker or Kubernetes, but my understanding is that they solve a completely different problem space. I use Vagrant heavily in my own work and...

      You're going to love Vagrant. I haven't really touched Docker or Kubernetes, but my understanding is that they solve a completely different problem space. I use Vagrant heavily in my own work and it makes things so much easier :)

      7 votes
      1. aphoenix Link Parent
        Docker solves a lot of the same problems as Vagrant, but also solves another problem (that Vagrant is starting to move into as well) which is deployment. To be clear, I still love and use Vagrant...

        Docker solves a lot of the same problems as Vagrant, but also solves another problem (that Vagrant is starting to move into as well) which is deployment.

        To be clear, I still love and use Vagrant a lot, and I'm a bit bummed that I'm moving away from it professionally (we're starting to replace it with Docker, so that it meets deployment needs as well).

        3 votes
  3. [2]
    JustABanana Link
    Finally! Not like I had the knowledge to contribute but now other people two are smarter than me can. Also stallman can now use tildes :) Also can we add source code link at the bottom of the page?

    Finally! Not like I had the knowledge to contribute but now other people two are smarter than me can. Also stallman can now use tildes :)

    Also can we add source code link at the bottom of the page?

    12 votes
    1. Deimos Link Parent
      Yes, I'll add links to the source code and issue tracker / roadmap later today or tomorrow, thanks for the reminder.

      Yes, I'll add links to the source code and issue tracker / roadmap later today or tomorrow, thanks for the reminder.

      9 votes
  4. [4]
    lordpipe (edited ) Link
    Neato. I'm setting it up to hack on it right now. Edit: The instructions should include how to configure the hostname. It keeps redirecting to tildes.net. Edit: After replacing all instances of...

    Neato. I'm setting it up to hack on it right now.

    Edit: The instructions should include how to configure the hostname. It keeps redirecting to tildes.net.

    Edit: After replacing all instances of tildes.net, I'm getting 400 Bad CSRF Origin when trying to log in from a non local hostname.

    Edit: The external port was different from the internal one used in dev config, to fix:

    diff --git a/tildes/development.ini b/tildes/development.ini
    index a050c0f..57078db 100644
    --- a/tildes/development.ini
    +++ b/tildes/development.ini
    @@ -5,7 +5,7 @@
     # Vagrant box. This setting's value should match the port defined in the
     # Vagrantfile that's being forwarded to port 443 on the guest.
     # If not using Vagrant, this setting should probably be removed.
    -prefixmiddleware_force_port = 4443
    +prefixmiddleware_force_port = 443
     
     [app:main]
     use = egg:tildes
    

    Edit: I'll probably contribute to docs later, but to add invites to a user:

    from tildes.models.user import User, UserInviteCode
    user = request.db_session.query(User).filter_by(username="lordpipe").first()
    invites = [UserInviteCode(user) for _ in range(256)]
    print(', '.join(list(map(lambda invite: str(invite), invites)))
    request.db_session.add_all([UserInviteCode(user) for _ in range(256)])
    request.tm.commit()
    
    12 votes
    1. [3]
      Deimos Link Parent
      That's weird, it shouldn't be doing anything with tildes.net except for a very few links like the Blog/Docs/etc. ones in the footer, it should all just be going through localhost. All links on the...

      That's weird, it shouldn't be doing anything with tildes.net except for a very few links like the Blog/Docs/etc. ones in the footer, it should all just be going through localhost. All links on the site were trying to go to tildes.net?

      6 votes
      1. [2]
        lordpipe Link Parent
        I've fixed the redirect issue by modifying the hostname in salt/salt/nginx/tildes.conf.jinja2. After doing that, internal links are fine, but the footer links to docs.tildes.net and...

        I've fixed the redirect issue by modifying the hostname in salt/salt/nginx/tildes.conf.jinja2. After doing that, internal links are fine, but the footer links to docs.tildes.net and blog.tildes.net are obviously hard coded.

        But yeah, in the default configuration, accessing it from localhost:4443 works fine. I'm accessing it from another hostname.

        8 votes
        1. smores (edited ) Link Parent
          Edit: I'm dumb, forgot to run vagrant provision Would you mind explaining exactly what you did? I'm actually having some trouble trying to do the same. I changed server_name in...

          Edit: I'm dumb, forgot to run vagrant provision


          Would you mind explaining exactly what you did? I'm actually having some trouble trying to do the same. I changed server_name in salt/salt/nginx/tildes.conf.jinja2 to <my IP address> localhost, but I'm still getting redirected to tildes.net. I also tried changing the www. redirect, which I don't think is relevant but was the only other place I could find tildes.net.

  5. chewzerita Link
    I just made a mirror of the repository at notabug.

    I just made a mirror of the repository at notabug.

    11 votes
  6. [9]
    Comment deleted by author
    Link
    1. [8]
      Deimos Link Parent
      That's odd, sounds like the initial data didn't get inserted into the database for some reason. I'll look and see if I can tell why that might be happening, thanks. You could try running from a...

      That's odd, sounds like the initial data didn't get inserted into the database for some reason. I'll look and see if I can tell why that might be happening, thanks. You could try running from a command prompt (inside a SSH session on the vagrant machine):

      python -c "from scripts.initialize_db import insert_dev_data; insert_dev_data('/opt/tildes/development.ini')"
      
      6 votes
      1. [6]
        Comment deleted by author
        Link Parent
        1. [5]
          Deimos Link Parent
          Just a warning, so it shouldn't matter. Strange that it didn't run in the first place though, I'll see if I can figure out why.

          Just a warning, so it shouldn't matter. Strange that it didn't run in the first place though, I'll see if I can figure out why.

          5 votes
          1. [4]
            Bauke Link Parent
            Just deleted my comment thinking merge requests were enabled but they're not. So I guess I'll comment again, you should enable merge requests. :P

            Just deleted my comment thinking merge requests were enabled but they're not. So I guess I'll comment again, you should enable merge requests. :P

            5 votes
            1. [3]
              Deimos Link Parent
              Oh? I didn't disable them deliberately, let me see why it's not working. Does it give a more specific error or anything, or just says merge requests aren't enabled?

              Oh? I didn't disable them deliberately, let me see why it's not working. Does it give a more specific error or anything, or just says merge requests aren't enabled?

              5 votes
              1. [2]
                Bauke Link Parent
                When trying to create one GitLab gives me a 403 and tells me I don't have the "permission to access this page." The setting is under Permissions, maybe it's set to "Only Project Members?"

                When trying to create one GitLab gives me a 403 and tells me I don't have the "permission to access this page." The setting is under Permissions, maybe it's set to "Only Project Members?"

                3 votes
                1. Deimos Link Parent
                  Ah ha, thanks. Try it again now.

                  Ah ha, thanks. Try it again now.

                  4 votes
      2. [2]
        super_james (edited ) Link Parent
        insert-dev-data isn't running because final-setup.initialize-db fails. The output from which is: ID: initialize-db Function: cmd.run Name: /opt/venvs/tildes/bin/python -c "from...

        insert-dev-data isn't running because final-setup.initialize-db fails.

        The output from which is:

        
             ID: initialize-db
            Function: cmd.run
                Name: /opt/venvs/tildes/bin/python -c "from scripts.initialize_db import initialize_db; initialize_db('/opt/tildes/development.ini')"
              Result: False
             Comment: Command "/opt/venvs/tildes/bin/python -c "from scripts.initialize_db import initialize_db; initialize_db('/opt/tildes/development.ini')"" run
             Started: 15:36:26.232789
            Duration: 5332.59 ms
             Changes:
                      ----------
                      pid:
                          7328
                      retcode:
                          1
                      stderr:
                          /opt/venvs/tildes/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
                            """)
                          INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
                          INFO  [alembic.runtime.migration] Will assume transactional DDL.
                          Traceback (most recent call last):
                            File "<string>", line 1, in <module>
                            File "/opt/tildes/scripts/initialize_db.py", line 38, in initialize_db
                              command.stamp(alembic_cfg, 'head')
                            File "/opt/venvs/tildes/lib/python3.6/site-packages/alembic/command.py", line 508, in stamp
                              script.run_env()
                            File "/opt/venvs/tildes/lib/python3.6/site-packages/alembic/script/base.py", line 427, in run_env
                              util.load_python_file(self.dir, 'env.py')
                            File "/opt/venvs/tildes/lib/python3.6/site-packages/alembic/util/pyfiles.py", line 81, in load_python_file
                              module = load_module_py(module_id, path)
                            File "/opt/venvs/tildes/lib/python3.6/site-packages/alembic/util/compat.py", line 82, in load_module_py
                              spec.loader.exec_module(module)
                            File "<frozen importlib._bootstrap_external>", line 678, in exec_module
                            File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
                            File "alembic/env.py", line 80, in <module>
                              run_migrations_online()
                            File "alembic/env.py", line 75, in run_migrations_online
                              context.run_migrations()
                            File "<string>", line 8, in run_migrations
                            File "/opt/venvs/tildes/lib/python3.6/site-packages/alembic/runtime/environment.py", line 836, in run_migrations
                              self.get_context().run_migrations(**kw)
                            File "/opt/venvs/tildes/lib/python3.6/site-packages/alembic/runtime/migration.py", line 321, in run_migrations
                              for step in self._migrations_fn(heads, self):
                            File "/opt/venvs/tildes/lib/python3.6/site-packages/alembic/command.py", line 497, in do_stamp
                              return script._stamp_revs(revision, rev)
                            File "/opt/venvs/tildes/lib/python3.6/site-packages/alembic/script/base.py", line 365, in _stamp_revs
                              heads, revision, include_dependencies=True)
                            File "/opt/venvs/tildes/lib/python3.6/site-packages/alembic/script/revision.py", line 394, in filter_for_lineage
                              id_, branch_label = self._resolve_revision_number(check_against)
                            File "/opt/venvs/tildes/lib/python3.6/site-packages/alembic/script/revision.py", line 437, in _resolve_revision_number
                              self._revision_map
                            File "/opt/venvs/tildes/lib/python3.6/site-packages/alembic/util/langhelpers.py", line 239, in __get__
                              obj.__dict__[self.__name__] = result = self.fget(obj)
                            File "/opt/venvs/tildes/lib/python3.6/site-packages/alembic/script/revision.py", line 122, in _revision_map
                              for revision in self._generator():
                            File "/opt/venvs/tildes/lib/python3.6/site-packages/alembic/script/base.py", line 90, in _load_revisions
                              for file_ in Script._list_py_dir(self, vers):
                            File "/opt/venvs/tildes/lib/python3.6/site-packages/alembic/script/base.py", line 782, in _list_py_dir
                              return os.listdir(path)
                          FileNotFoundError: [Errno 2] No such file or directory: '/opt/tildes/alembic/versions'
        
        
        

        I don't really understand vagrant so not sure what's causing this yet.

        2 votes
        1. Deimos Link Parent
          Thank you, that tells me exactly what the issue is: I deleted all the alembic migration files when I squashed the commit history down to a single commit, so now git isn't even creating the folder...

          Thank you, that tells me exactly what the issue is: I deleted all the alembic migration files when I squashed the commit history down to a single commit, so now git isn't even creating the folder because it's empty, and that's crashing the script.

          6 votes
  7. [4]
    Luca Link
    Huge kudos for open sourcing it already! I have a background in devops, would you be interested in me taking a look and dockerizing the setup, maybe submitting a PR with a working dockerfile? (Or...

    Huge kudos for open sourcing it already!

    I have a background in devops, would you be interested in me taking a look and dockerizing the setup, maybe submitting a PR with a working dockerfile? (Or docker-compose, if required)

    11 votes
    1. [3]
      Deimos Link Parent
      Thanks, I appreciate the offer. I don't have any interest in using Docker for the site itself, but what application are you thinking of using it for? An alternate setup method for local dev versions?

      Thanks, I appreciate the offer. I don't have any interest in using Docker for the site itself, but what application are you thinking of using it for? An alternate setup method for local dev versions?

      6 votes
      1. [2]
        Luca Link Parent
        Mostly for local development and testing, yeah. Vagrant is nice and all, but I find the feedback cycle way too long whenever I have to work on with it.

        Mostly for local development and testing, yeah. Vagrant is nice and all, but I find the feedback cycle way too long whenever I have to work on with it.

        4 votes
        1. rabidfurby Link Parent
          I love Docker, and use it extensively both at home and at $dayjob, but it might be more trouble than it's worth in this case. From a brief look at the Vagrant + Salt setup, you'd have to duplicate...

          I love Docker, and use it extensively both at home and at $dayjob, but it might be more trouble than it's worth in this case.

          From a brief look at the Vagrant + Salt setup, you'd have to duplicate nearly all of the configuration defined by Salt (things like teaching Postgres that it should use RabbitMQ as a message broker for NOTIFY commands). If the plan was to use Docker in production, I think it'd be worthwhile, but if the production stack continues to use Salt for config management, a Docker setup will probably be a perpetual second-class citizen and source of unnecessary duplication.

          8 votes
  8. crius Link
    It was two days that i was pondering if I should have sent a message to you asking about the open source day. I was growing worried, glad to finally see it! :)

    It was two days that i was pondering if I should have sent a message to you asking about the open source day. I was growing worried, glad to finally see it! :)

    10 votes
  9. [10]
    Comment deleted by author
    Link
    1. [9]
      Deimos (edited ) Link Parent
      I think that people try to abstract things too much. Topics and comments serve totally different purposes, and while you can find some common things between them, overall they'll almost never be...

      I think that people try to abstract things too much. Topics and comments serve totally different purposes, and while you can find some common things between them, overall they'll almost never be treated the same way. You just end up moving the complexity somewhere else, and probably even end up with more complexity in the end than just treating them as separate things in the first place. Something I notice in general is that people are way too worried about any kind of repetition and will try to eliminate it, but repeating similar code really isn't that bad in most cases. It will often end up diverging later anyway, in which case your abstraction makes it more complex than it would have been.

      I'm coming from spending the last few years working on reddit code, which was way over-abstracted by trying to treat almost everything as a Thing that has upvotes and downvotes and some other common attributes, and it just turned into a giant disaster. The number of subscribers to a subreddit is stored in subreddit._ups because... subscriptions are... kind of... like upvotes, I guess. And the subreddit's "popularity" is stored in subreddit._downs because... well, because there's no downvotes on subreddits and we needed somewhere else to store a number, so we might as well, and so on.

      18 votes
      1. [8]
        lordpipe (edited ) Link Parent
        I haven't spent too much time exploring reddit source code but I've always found this incredibly strange. Glad to hear I'm not insane.

        I'm coming from spending the last few years working on reddit code, which was way over-abstracted by trying to treat almost everything as a Thing that has upvotes and downvotes and some other common attributes

        I haven't spent too much time exploring reddit source code but I've always found this incredibly strange. Glad to hear I'm not insane.

        6 votes
        1. [7]
          Deimos (edited ) Link Parent
          The reddit code is an excellent example of people trying to be far too clever. It has some pretty neat ideas in it and was probably great for prototyping the site in originally, but it's...

          The reddit code is an excellent example of people trying to be far too clever. It has some pretty neat ideas in it and was probably great for prototyping the site in originally, but it's completely insane to still be using it as the foundation for one of the top sites on the internet.

          9 votes
          1. crius Link Parent
            Reddit example is not an example of why applying too much DRY approach or abstraction in general is bad, but of clear bad design choice. You don't store apples in the containers that says...

            Reddit example is not an example of why applying too much DRY approach or abstraction in general is bad, but of clear bad design choice.

            You don't store apples in the containers that says "oranges" to not waste a container.

            You create a basic container without labels and then extend it with the additional attributes it needs.

            Having said so, abstraction can absolutely become a problem if you just apply it blindly. Knowing when to use a paradigm or strategy is what differentiate junior from senior developers.

            Probably that decision was taken back in the early day and no one challenged it in the following years.

            6 votes
          2. [5]
            Emerald_Knight Link Parent
            "Clever" is something that's painful to deal with. I'm trying to avoid that as much as possible in my current work because it's just such a massive headache trying to deal with the steaming pile...

            "Clever" is something that's painful to deal with. I'm trying to avoid that as much as possible in my current work because it's just such a massive headache trying to deal with the steaming pile of technical debt that inevitably arises. No code is completely free of technical debt and it's pretty much a guarantee that there will be some pretty bad parts of my own, but I'm at least going to try to mitigate it as much as possible. I had some pretty bad messes to deal with in my previous work and that was for a small startup, so I don't even want to begin to imagine what the reddit code base looked like.

            4 votes
            1. [4]
              Deimos Link Parent
              I think it's part of people's evolution as a programmer. When you've never had to maintain a large, complex system for a long period of time, solving things in clever ways is appealing. It's kind...

              I think it's part of people's evolution as a programmer. When you've never had to maintain a large, complex system for a long period of time, solving things in clever ways is appealing. It's kind of exciting to figure out a fancy approach to a problem, other people will be impressed if you show your code to them, and so on. If you're just a hobby programmer or in school, you never have to deal with the longer-term maintenance impacts of that clever code, you finish with the assignment/project/whatever and never look at it again.

              As you get more experience though, you start to recognize the benefits of having code written clearly and simply. Yeah, it's boring and it's never going to impress anyone that looks at it, but that's the point—it's easy to understand, debug, and modify when you need to, even if you've completely forgotten how it works.

              10 votes
              1. [2]
                devlinium Link Parent
                This is something I have a hard time explaining to others. It's great and amazing that an intern has come up with this fancy solution, but it's too fancy, and will be extremely inflexible later...

                This is something I have a hard time explaining to others. It's great and amazing that an intern has come up with this fancy solution, but it's too fancy, and will be extremely inflexible later on. It's clever now, but will be a bane later.

                4 votes
                1. rabidfurby Link Parent
                  Reminds me of an intern I worked with, many years ago, and a function they wrote with a comment like "// this is pretty much a masterpiece of recursion".

                  Reminds me of an intern I worked with, many years ago, and a function they wrote with a comment like "// this is pretty much a masterpiece of recursion".

                  1 vote
              2. danjac Link Parent
                Unfortunately many startups use students/novices at the beginning because they're cheap/naive, and they build exactly the kind of code that leads to these long-term maintenance issues that are a...

                Unfortunately many startups use students/novices at the beginning because they're cheap/naive, and they build exactly the kind of code that leads to these long-term maintenance issues that are a nightmare for later employees to deal with (and rewriting/refactoring always takes a back seat to the "feature customer needs yesterday").

                2 votes
  10. super_james Link
    Hot damn Deimos, my clients project just launched today and I was planning on catching up on paperwork tomorrow. :)

    Hot damn Deimos, my clients project just launched today and I was planning on catching up on paperwork tomorrow. :)

    8 votes
  11. NubWizard Link
    Oh, python stuff! This is something I want to sink my teeth into. Thanks!

    Oh, python stuff! This is something I want to sink my teeth into.

    Thanks!

    5 votes
  12. [2]
    joelthelion Link
    How do you envision collaborative development? Do you foresee lots of contributions with site users contributing new features, or would you rather keep a small team?

    How do you envision collaborative development? Do you foresee lots of contributions with site users contributing new features, or would you rather keep a small team?

    5 votes
    1. Deimos Link Parent
      I don't really have a particular vision for it. Significant contributions can be a lot of work, so my expectation is that there will probably only be a small group of people that do larger...

      I don't really have a particular vision for it. Significant contributions can be a lot of work, so my expectation is that there will probably only be a small group of people that do larger changes, but I'd be happy to see lots of smaller contributions as well (for example, minor tweaks/cleanups of things I'm not great at like sysadmin-side stuff, frontend, etc.). Overall, I'd be happy even if there aren't any contributions anyway, just because I like doing the development publicly.

      5 votes
  13. teaearlgraycold Link
    Typed Python! You're killing it, Deimos!

    Typed Python! You're killing it, Deimos!

    4 votes
  14. flaque Link
    Hurray! :D

    Hurray! :D

    3 votes
  15. [7]
    SourceContribute Link
    What's the caching situation like? It looks like everything is redis? Are you passing through queries/objects through to them automagically or is the caching not in place yet?

    What's the caching situation like? It looks like everything is redis? Are you passing through queries/objects through to them automagically or is the caching not in place yet?

    3 votes
    1. [6]
      Deimos Link Parent
      No caching yet. Postgresql is extremely fast as long as you have good indexes (and has its own internal caching-like behavior). Eventually I'll probably need to add some, but it likely shouldn't...

      No caching yet. Postgresql is extremely fast as long as you have good indexes (and has its own internal caching-like behavior). Eventually I'll probably need to add some, but it likely shouldn't be necessary for a long time.

      6 votes
      1. [5]
        rabidfurby Link Parent
        From a quick scan of the code, only place I saw Redis used was for a bloom filter of commonly hacked passwords, while RabbitMQ is used as a broker for notifications flowing through Postgres. Are...

        From a quick scan of the code, only place I saw Redis used was for a bloom filter of commonly hacked passwords, while RabbitMQ is used as a broker for notifications flowing through Postgres. Are there any other big moving parts?

        4 votes
        1. [4]
          Deimos (edited ) Link Parent
          Redis is also used for session storage and rate-limiting. The RabbitMQ usage is huge overkill for now (and was honestly implemented more as a proof of concept than something that's necessary yet),...

          Redis is also used for session storage and rate-limiting. The RabbitMQ usage is huge overkill for now (and was honestly implemented more as a proof of concept than something that's necessary yet), but it will be useful for a lot of scraping/metadata uses as well as some other things in the future.

          Those are pretty much all the major technologies involved though, yes.

          4 votes
          1. [3]
            SourceContribute Link Parent
            True, Postgres is quite solid. Though even for small Django websites I've used Redis for custom data structures, so at least that's available (even if it's not specifically used for caching). Most...

            No caching yet. Postgresql is extremely fast as long as you have good indexes (and has its own internal caching-like behavior). Eventually I'll probably need to add some, but it likely shouldn't be necessary for a long time.

            True, Postgres is quite solid. Though even for small Django websites I've used Redis for custom data structures, so at least that's available (even if it's not specifically used for caching).

            Most sites end up with a background processor that needs to be turned into a message queue. Lot of sites use Celery or Resque and then have to code shells for whatever they want and have to deploy it with the main web app. Kinda nice to be able to run a message queue that you can just hook into and spin up a separate service for.

            Curious to see the scale of Tildes so far.

            2 votes
            1. [2]
              Deimos Link Parent
              Yeah, that's pretty much what rabbitmq is there for. It's "connected" to NOTIFY events from postgresql through pg-amqp-bridge. That way, postgresql sends messages like topic.created, topic.edited...

              Most sites end up with a background processor that needs to be turned into a message queue.

              Yeah, that's pretty much what rabbitmq is there for. It's "connected" to NOTIFY events from postgresql through pg-amqp-bridge. That way, postgresql sends messages like topic.created, topic.edited and so on when the relevant table has rows inserted/updated/deleted, and background processor jobs can subscribe to the queues for the events they care about.

              Like I said, it's definitely overkill for what I'm using it for at the moment, but it has some great possibilities for the future.

              2 votes
              1. SourceContribute Link Parent
                That's awesome, didn't realize that existed!

                That's awesome, didn't realize that existed!

                2 votes
  16. [11]
    NubWizard Link
    So, I am learning Python and I definitely want to contribute in any way possible because this is definitely a project I can get behind. However, I have been more on the data science track with my...

    So, I am learning Python and I definitely want to contribute in any way possible because this is definitely a project I can get behind. However, I have been more on the data science track with my self-teaching. Would you recommend any resources that I could look at and work through that would give me a better understanding of how Tildes is built?

    3 votes
    1. [2]
      rabidfurby Link Parent
      If you've never done any web development, a great place to start would be the Pyramid tutorial. That's the web framework underneath everything, and the tutorial takes you through Jinja2 templates...

      If you've never done any web development, a great place to start would be the Pyramid tutorial. That's the web framework underneath everything, and the tutorial takes you through Jinja2 templates and SQLAlchemy database queries, both of which Tildes uses.

      4 votes
      1. NubWizard Link Parent
        Thank you for this! I will add this to my list for a resource to work through!

        Thank you for this! I will add this to my list for a resource to work through!

        2 votes
    2. [5]
      Deimos Link Parent
      Hmm, I think it's difficult because web applications have so many different parts involved that it's pretty hard to say how you could start. Something I'd like to do is write up kind of a...

      Hmm, I think it's difficult because web applications have so many different parts involved that it's pretty hard to say how you could start. Something I'd like to do is write up kind of a "walkthrough" of implementing a whole feature, explaining everything that needs to be changed as part of it. I think that could be very helpful to be able to give people a better understanding of the different pieces and how they interact with each other.

      2 votes
      1. [2]
        rabidfurby Link Parent
        I think it'd be helpful to start simpler than that - explain all the moving parts of the system as-is, before I make any changes to it. When I worked at Amazon, one of the things they had new...

        Something I'd like to do is write up kind of a "walkthrough" of implementing a whole feature, explaining everything that needs to be changed as part of it.

        I think it'd be helpful to start simpler than that - explain all the moving parts of the system as-is, before I make any changes to it.

        When I worked at Amazon, one of the things they had new hires for AWS do is make some requests to S3 - create a bucket, upload a file to it, then download that file, etc - then use the request IDs and logs to trace the path each request took through the various backend services.

        I think a similar high-level walkthrough would be helpful here. When I click "submit" on a new post, what happens? When someone comments on my post, what happens? What triggers the orangered notification for me, how do the "X new comments since your last visit" counters get maintained, etc.

        4 votes
        1. Deimos Link Parent
          That's a great suggestion, thanks. I added it as an issue on the docs site to make sure I remember it as something I should do: https://gitlab.com/tildes/tildes-static-sites/issues/1

          That's a great suggestion, thanks. I added it as an issue on the docs site to make sure I remember it as something I should do: https://gitlab.com/tildes/tildes-static-sites/issues/1

          2 votes
      2. smores Link Parent
        This is something I just did for a new codebase at work, we call them “codelabs” at Google. I can look into at least writing the outline for a Tildes codelab if you want to make an issue for...

        This is something I just did for a new codebase at work, we call them “codelabs” at Google. I can look into at least writing the outline for a Tildes codelab if you want to make an issue for tracking it.

        1 vote
      3. NubWizard Link Parent
        Thank you for responding! I will keep up reviewing the code and tinkering here and there to get a better understanding of how everything ties together. I hope I can contribute in any way possible...

        Thank you for responding!

        I will keep up reviewing the code and tinkering here and there to get a better understanding of how everything ties together. I hope I can contribute in any way possible eventually!

    3. [3]
      Celeo Link Parent
      Without getting extremely lucky with a full-feature tutorial that uses the exact same stack as this site, you should look at learning each piece separately. Start with the parts of the site that...

      Without getting extremely lucky with a full-feature tutorial that uses the exact same stack as this site, you should look at learning each piece separately.

      Start with the parts of the site that you'd want to eventually contribute to, and start reading up. If it's the backend that you're most interested in, start there with generic Python tutorials to get the syntax solid in your mind, and then move onto tutorials for Pyramid. Then, Jinja so changes you make can be rendered on the site. If you're not interested in learning the database too much, then don't spend a lot of time learning Postgres - just accept that it exists and is running in the background. Same with the other "ops"-y stuff, like Nginx.

      2 votes
      1. [2]
        NubWizard Link Parent
        Thank you for responding! I will take your advice and go through things I know and am interested in step-by-step. Everything is pretty new to me but even stuff I don't think I would be interested...

        Thank you for responding!

        I will take your advice and go through things I know and am interested in step-by-step. Everything is pretty new to me but even stuff I don't think I would be interested in has something that I can learn that should help me elsewhere.

        1 vote
        1. Celeo Link Parent
          That's definitely true, and a good way of thinking. Take it one piece at a time, and if something seems fun and interesting, go for it!

          That's definitely true, and a good way of thinking. Take it one piece at a time, and if something seems fun and interesting, go for it!

          2 votes
  17. [4]
    Celeo (edited ) Link
    Fantastic, thanks for open sourcing! Update: setup the environment on MacOS 10.13, Virtualbox 5.2.12, and vagrant 2.1.2. I got some error messages after the first vagrant up and the default...

    Fantastic, thanks for open sourcing!

    Update: setup the environment on MacOS 10.13, Virtualbox 5.2.12, and vagrant 2.1.2. I got some error messages after the first vagrant up and the default user/pass in the docs didn't work. I followed the instructions for adding a user and I was able to log in to that without issue.

    I'll repro the steps in another env to see if I was doing it wrong.

    3 votes
    1. [3]
      super_james (edited ) Link Parent
      This is the same problem I had, see below. If you create the folder versions under tildes/alembic then vagrant up should just work.

      This is the same problem I had, see below.

      If you create the folder versions under tildes/alembic then vagrant up should just work.

      2 votes
      1. Celeo Link Parent
        Had to recreate my vagrant box today and can confirm that having the versions folder there before running vagrant up worked great.

        Had to recreate my vagrant box today and can confirm that having the versions folder there before running vagrant up worked great.

        3 votes
      2. Celeo Link Parent
        Ah, good find. I ended up creating that file when I went to perform a table alteration; the alembic command complained until I created it manually.

        Ah, good find. I ended up creating that file when I went to perform a table alteration; the alembic command complained until I created it manually.

        2 votes
  18. danjac Link
    This is fantastic - great use of typed Python throughout. Going to spend a few days going through the code. Thank you.

    This is fantastic - great use of typed Python throughout. Going to spend a few days going through the code. Thank you.

    3 votes
  19. aphoenix Link
    Super exciting - congrats on getting this all taken care of. Just in time for my wife and kids to be gone for a week, so I think I'll actually have a look at it this coming week!

    Super exciting - congrats on getting this all taken care of.

    Just in time for my wife and kids to be gone for a week, so I think I'll actually have a look at it this coming week!

    3 votes
  20. [2]
    Godverdomme Link
    Good stuff man. Aside from helping with documentation and issues if I'm hacking around and find something useful or fun to implement, will you deny the pull request based on the road map or is it...

    Good stuff man. Aside from helping with documentation and issues if I'm hacking around and find something useful or fun to implement, will you deny the pull request based on the road map or is it OK to try?

    3 votes
    1. Deimos (edited ) Link Parent
      It's hard to say without knowing what it is. As it says in the CONTRIBUTING doc, it's best to "claim" (or create) an issue for the thing you're going to work on first, so that you know it's...

      It's hard to say without knowing what it is. As it says in the CONTRIBUTING doc, it's best to "claim" (or create) an issue for the thing you're going to work on first, so that you know it's actually wanted. If you just want to tinker around and won't be disappointed if you do a bunch of work that doesn't get included, that's fine of course. It might be best to at least send me a quick message and ask before working on anything significant if you're hoping it'll be included in the site though.

      3 votes
  21. what (edited ) Link
    🎉🎉🎉 Awesone, I’m excited to read through it and (possibly) contribute! The development documentation looks very thorough.

    🎉🎉🎉

    Awesone, I’m excited to read through it and (possibly) contribute! The development documentation looks very thorough.

    2 votes
  22. Silbern Link
    Wooooo! Awesome, I can't wait to dig in. Thanks for releasing it! :D

    Wooooo! Awesome, I can't wait to dig in. Thanks for releasing it! :D

    2 votes
  23. MajorMajorMajorMajor Link
    This is an exciting step for the site! I'm looking forward to digging into this later.

    This is an exciting step for the site! I'm looking forward to digging into this later.

    2 votes
  24. clerical_terrors Link
    Fantastic organization, really cool to finally be able to see the code.

    Fantastic organization, really cool to finally be able to see the code.

    2 votes
  25. jgb Link
    Congratulations on the release.

    Congratulations on the release.

    2 votes
  26. [4]
    StellarTabi Link
    Does Tildes have any desktop or mobile apps?

    Does Tildes have any desktop or mobile apps?

    2 votes
    1. [3]
      Archimedes Link Parent
      The mobile site works pretty well. What features would you look for in an app?

      The mobile site works pretty well. What features would you look for in an app?

      13 votes
      1. StellarTabi Link Parent
        I was only curious if it already existed for reading it's source code. I have much appreciation for the current website that just works and actually already feels like a native app (compared all...

        I was only curious if it already existed for reading it's source code. I have much appreciation for the current website that just works and actually already feels like a native app (compared all of reddit's shitty mobile apps/webites).

        3 votes
      2. helloworld Link Parent
        Easy sharing to app. It can make posting really easy, and might even encourage more users to post, and more often.

        Easy sharing to app. It can make posting really easy, and might even encourage more users to post, and more often.

        2 votes
  27. super_james Link
    Anyone got time to give me some feedback on: https://gitlab.com/jms301/tildes/commit/f1d5962e82f5e4769334919271679c237019bb22 I have zero experience with Pyramid, SQLAlchemy or Intercooler so I'm...

    Anyone got time to give me some feedback on: https://gitlab.com/jms301/tildes/commit/f1d5962e82f5e4769334919271679c237019bb22

    I have zero experience with Pyramid, SQLAlchemy or Intercooler so I'm expecting obvious mistakes, don't worry about my feelings!

    Just that I'd rather poke code first then read the docs and tutorials later when I've had a go. :)

    2 votes