18 votes

How to install + serve Tildes directly on a VPS?

Posted by SunriseMojo
Tags: tech support, ask.advice, vps

Hi,

I would like to run a Tildes instance on a VPS, using a custom domain.

QUESTION:

Is it possible to install and serve Tildes directly on a VPS? (eliminate Vagrant / VirtualBox entirely)

Being a solo dev, it feels like Vagrant / VB adds excess complexity for little benefit.

  • note 1: I tried the Vagrant / VB install method (on an Ubuntu VPS), and hit some errors - all related to Vagrant / VB.

  • note 2: I found this 3-year old comment of Deimos’ instructions, though I'm guessing it's out of date, since the code has changed a lot in 3 years (salts, minions, etc).

If it IS possible to install and serve Tildes directly on a VPS - what is the best / simplest way to do it in 2022?

I will very much appreciate any ideas.

16 comments

  1. [5]
    Deimos
    (edited )
    Link
    Vagrant/Virtualbox is only for local development, so that you can have all the right versions of everything inside a single contained VM. You don't want to use those if you're actually trying to...

    Vagrant/Virtualbox is only for local development, so that you can have all the right versions of everything inside a single contained VM. You don't want to use those if you're actually trying to host an instance of the site.

    If you're setting up a VM specifically to host Tildes, you can just install Ubuntu 20.04 Debian 10 on it and then run Ansible on it directly to install and set up everything it needs. I think there might be a couple of manual steps necessary like cloning the tildes repo somewhere and creating a symlink to it from /opt/tildes, but almost everything else should be done by Ansible. That comment of mine that you linked should have the manual steps in it, but it's mostly obsolete things related to Salt (which was replaced by Ansible) that you can ignore.

    I think the tildes.net domain is probably hardcoded in various places still too, so you'll probably need to do a search and replace on that with your own domain, but that should be pretty straightforward.

    9 votes
    1. [4]
      SunriseMojo
      Link Parent
      Hey, thank you for the detailed reply. I will set up a new VPS and give it a try.

      Hey, thank you for the detailed reply. I will set up a new VPS and give it a try.

      4 votes
      1. [3]
        Deimos
        Link Parent
        Please let me know if you run into any difficulties with it, or if anything else ends up being necessary. I'd like to add a document to the repo that just has the specific steps needed to get it...

        Please let me know if you run into any difficulties with it, or if anything else ends up being necessary. I'd like to add a document to the repo that just has the specific steps needed to get it up and running.

        7 votes
        1. [2]
          SunriseMojo
          (edited )
          Link Parent
          Okay great. I’m keeping track of the specific steps / commands as I go - I will clean it up and send it over. It seems there are two approaches: target localhost in ansible for a loopback install...

          Okay great. I’m keeping track of the specific steps / commands as I go - I will clean it up and send it over.

          It seems there are two approaches:

          1. target localhost in ansible for a loopback install (all on the VPS) - tweaking the ansible commands / configuration accordingly. ( <== That is what I'm trying now.)

          2. target the VPS-IP from another machine, and configure ansible for that approach.

          Which do you think is optimal?

          Also - which playbook host-patterns are integral for a direct-VPS install (all, app_server, dev, prod)?

          ^ That might be self-evident, but I'd rather confirm up front (if possible).

          I really appreciate your communication and willingness.

          2 votes
          1. Deimos
            Link Parent
            Personally, I run ansible on my local machine, which does all the necessary setup commands over SSH. You can set it up to run entirely on the VPS, but I think it's more complex. I probably...

            Personally, I run ansible on my local machine, which does all the necessary setup commands over SSH. You can set it up to run entirely on the VPS, but I think it's more complex. I probably wouldn't do it that way unless you can't do it from your local machine for some reason. The prod playbook is the one you want.

            Also, I'm not sure why I said Ubuntu 20.04 above, the OS should be Debian 10 ("buster").

            4 votes
  2. [5]
    drannex
    Link
    I have tried repeatably to launch my own instance, with varying results -- none of them entirely successful. The vagrant method has been broken for quite some time, wish I had some tips but I...

    I have tried repeatably to launch my own instance, with varying results -- none of them entirely successful. The vagrant method has been broken for quite some time, wish I had some tips but I haven't had any luck with this either.

    5 votes
    1. [4]
      Deimos
      Link Parent
      Is there something specific that's broken in the Vagrant method for you? That's how I set up my own local dev environment, so it's still working for me.

      Is there something specific that's broken in the Vagrant method for you? That's how I set up my own local dev environment, so it's still working for me.

      4 votes
      1. [3]
        talklittle
        Link Parent
        When setting up a fresh VM in a fresh Git clone, there's an Ansible problem creating the node_modules directory. The site probably functions normally, but at least some style checks requiring...

        When setting up a fresh VM in a fresh Git clone, there's an Ansible problem creating the node_modules directory. The site probably functions normally, but at least some style checks requiring Node.js break.

        This is the fix: https://gitlab.com/tildes/tildes/-/merge_requests/133

        6 votes
        1. [2]
          Deimos
          Link Parent
          Oh, thanks - I'll test that out later today and merge it. I guess that wouldn't have been affecting me because I already have the folder?

          Oh, thanks - I'll test that out later today and merge it. I guess that wouldn't have been affecting me because I already have the folder?

          5 votes
          1. talklittle
            Link Parent
            Right. On a new-from-scratch Git checkout, Ansible (via vagrant up) will skip creating node_modules, causing tests and Git hooks to fail.

            I guess that wouldn't have been affecting me because I already have the folder?

            Right. On a new-from-scratch Git checkout, Ansible (via vagrant up) will skip creating node_modules, causing tests and Git hooks to fail.

            4 votes
  3. [4]
    talklittle
    Link
    A few months back I put in some work to get Tildes unit tests and code checks running on GitHub Actions free tier. This info may inform as to what software packages need to get installed, and...

    A few months back I put in some work to get Tildes unit tests and code checks running on GitHub Actions free tier. This info may inform as to what software packages need to get installed, and folders and permissions to create.

    https://github.com/talklittle/tildes/commit/a337415ed1c91e4fd47c99d5099ec132e78f8701

    The first file, ci.yml, is where most of the good info lies.

    Notes/caveats:

    1. This is a Docker environment, as used by GitHub Actions cloud runners. Specifically I found it easier to use the Nginx Docker image instead of installing Nginx normally, but this may not apply when installing on bare metal without Docker.
    2. Uses Ubuntu instead of Debian (a limitation of GitHub Actions). This is why some of the Apt repos are turned into Ansible variables, instead of keeping the Debian repo URLs.
    3. Focuses on getting the code style checks and unit tests to run; full site may or may not run (untested).
    5 votes
    1. [3]
      SunriseMojo
      (edited )
      Link Parent
      Thank you, that is good information. I am targeting a Debian 10 VM, via SSH. Seems the challenge is aligning the ansible commands / configuration until everything is integrated (as your document...

      Thank you, that is good information.

      I am targeting a Debian 10 VM, via SSH.

      Seems the challenge is aligning the ansible commands / configuration until everything is integrated (as your document shows).

      A docker method would be beneficial. Were you attempting a Tildes install? (If so, how did it go?)

      Has anyone successfully created an instance?

      3 votes
      1. [2]
        talklittle
        Link Parent
        No, my goal was only to run the tests, to catch setup errors for newly setup development environments. That said, some of the tests, in the "webtests" folder, do encompass the core website code,...

        Were you attempting a Tildes install? (If so, how did it go?)

        No, my goal was only to run the tests, to catch setup errors for newly setup development environments.

        That said, some of the tests, in the "webtests" folder, do encompass the core website code, such as creating a user and accessing the homepage, so following my CI document should at least get you as far as loading the homepage.

        Things that I am not sure about include fetching remote content (like article word count), and background services and cron jobs. And things like gracefully restarting nginx and server components, and auto-restarting components using systemd.

        Has anyone successfully created an instance?

        As far as I know, no, but multiple people have expressed interest. If you do succeed and document how you got there, that would be fantastic. I'm sure it would get the creative juices flowing for other developers, imagining how they could build on top of the Tildes codebase.

        5 votes
        1. SunriseMojo
          Link Parent
          Okay thank you, I’ve looked through all the CI code and the webtests code. Yes that is what I’m excited about too. To see what kind of features people develop, and to experiment with new ideas....

          Okay thank you, I’ve looked through all the CI code and the webtests code.

          Yes that is what I’m excited about too. To see what kind of features people develop, and to experiment with new ideas.

          Tildes is such a solid codebase, there’s a lot of room for expansion (though it’s also great how it is).

          2 votes
  4. [2]
    weystrom
    (edited )
    Link
    I was going to send an email to @Deimos, but might as well post it here: We’re ~500 person company and we’ve been looking to spin up an internal “redditboard” (hopefully that doesn’t sound...

    I was going to send an email to @Deimos, but might as well post it here:

    We’re ~500 person company and we’ve been looking to spin up an internal “redditboard” (hopefully that doesn’t sound degrading to ~) to share the technical info/discussions/memes and what not.

    I love Tildes and decided to have a go spinning it up. Ansible playbooks are awesome, everything came up in less than 10 minutes on a fresh Debian 10 machine. I need to figure out the Postgres config to make it a bit more compact, but the base prod installation works great!

    Now it's time to figure out how to make it usable for us:

    • SSO registration/login with Google/LDAP - should be doable with Pyramid, but personally I have no idea where to start
    • allow to embed images into post for charts and screenshots - should be easy enough to tweak the templates
    • figure out the basic moderator actions and permissions - these don't seem to be documented at all

    The real question to answer - is this manageable at all in a decent amount of time, or do you think we'd be better off setting up something else? I know that open source != easy self-hosting, but I'd love to have our own tildes instance to use internally.

    3 votes
    1. Deimos
      (edited )
      Link Parent
      Posting it here is good - I haven't been posting much recently, but I'm generally around and will see it if people message/mention me. Good to hear that it was easy to get running for you! Some...

      Posting it here is good - I haven't been posting much recently, but I'm generally around and will see it if people message/mention me.

      Good to hear that it was easy to get running for you! Some thoughts about the changes you're hoping to make:

      1. Google/LDAP SSO - I have no idea about this at all either. I feel like it probably wouldn't be too difficult, but it could depend on whether there are good existing libraries for it that you can get working easily.

        Tildes being set up as invite-only is probably helpful here overall, since you won't have to worry about disabling registration or anything like that. Depending how many people you need to give access to, it may be a lot simpler to just find a way to distribute invite codes internally than implement SSO.

        This one might end up being an issue with most of the self-hostable similar platforms available too, I don't know which ones (if any) would already have built-in SSO support.

      2. Embedding images - I think there are two different parts here, and the difficulty will depend on exactly what you need:

        1. The images need to be hosted somewhere, and Tildes has no functionality at all related to uploading/hosting images. So if you need the platform itself to be able to handle the hosting, adding that to Tildes would be a significant amount of work. Other platforms with uploading/hosting functionality (which is relatively common) would likely be a lot easier.
        2. If the images are already hosted somewhere and you just need to embed them, that should actually be pretty easy. The markdown library already has support for adding images with a syntax that looks like ![label](url), and Tildes is specifically stripping that out: https://gitlab.com/tildes/tildes/-/blob/master/tildes/tildes/lib/markdown.py#L213-214

      3. Moderator-like actions/permissions - Yeah, this is definitely a rough part right now. The permissions have to be managed manually by inserting the rows directly into the database (in the user_permissions table), and there isn't really any documentation of what each of the permissions do.

        The valid permissions are listed here: https://gitlab.com/tildes/tildes/-/blob/master/tildes/tildes/enums.py#L299-311

        Most of them should probably be pretty intuitive from their names, but figuring out some of them could be a little tricky and would involve looking in the __acl__() methods that define permissions for different objects. For example, if you wanted to know what the topic.edit_by_generic_user permission does (which is probably the most confusing one), you'd search for that string in the code and find this section: https://gitlab.com/tildes/tildes/-/blob/master/tildes/tildes/models/topic/topic.py#L364-374

        That's in Topic.__acl__(), so that's adding permission to topics. In this case, someone is getting the edit permission on a topic if it's a text topic, posted by the user_id of -1 (the "generic" user that makes scheduled posts), and they have that topic.edit_by_generic_user permission.

        If you need to go a level deeper on that, you'd need to look at where the edit permission on a topic is actually checked, like:

      5 votes