12 votes

How Matrix fixed the scalability of Synapse

12 comments

  1. [6]
    admicos
    Link
    Good, they dealt with scaling up. Now deal with scaling down. Which is probably the most important piece of a server for a federated network. I recall the Matrix folk saying somewhere that their...

    Good, they dealt with scaling up.

    Now deal with scaling down. Which is probably the most important piece of a server for a federated network.

    I recall the Matrix folk saying somewhere that their eventual goal is to shut down the Matrix.org homeserver. How are you going to do that when the people that have the will to host a server (for themselves or small communities) just can't, because Synapse uses all their resources and then some.

    This is not just a Synapse/Matrix problem. See Mastodon for another example. Giant Ruby-on-Rails thing that has three separate services (web, sidekiq, streaming) you need to run and eat half of the RAM of my pretty low-end server (209m + 190m + 161m for the top three processes) for a single active account. Their web asset building step uses so much resources I just copy files over from my workstation to the server when I need to update, instead of their suggestion of building in-server (otherwise I'd have to stop literally everything on the server, allocate 2GB of extra swap, run Node.js with extra arguments telling it to use more RAM, and wait half an hour).

    At least Mastodon has reasonably competent alternatives like Pleroma (but their UX is just meh). What alternative does Synapse have? How many of the alternatives work well enough to be considered production ready?

    7 votes
    1. [3]
      tindall
      Link Parent
      There are several lower-footprint implementations of the protocol in many languages, including Rust and Go, easily found on the project's website.

      There are several lower-footprint implementations of the protocol in many languages, including Rust and Go, easily found on the project's website.

      4 votes
      1. [2]
        admicos
        Link Parent
        Synapse is the only one with the maturity of "released". Dendrite, Construct and Ligase are all somewhere in beta. Conduit and Maelstrom are all alpha. (Source:...

        There are several lower-footprint implementations of the protocol in many languages, including Rust and Go, easily found on the project's website.

        (Source: https://matrix.org/docs/projects/try-matrix-now. Project type set to "Server" and everything else except "Not actively maintained" checked)


        Dendrite is missing support for (source: the readme file):

        • Receipts
        • Push
        • Search and Context
        • User Directory
        • Presence
        • Guests

        As of October 2020 we're at around 57% CS API coverage and 81% Federation coverage,


        Construct has this note on their readme:

        Due to the breadth of the Matrix client/server protocol we can only endorse production use of Construct gradually while local user restrictions are developed. This notice applies to locally registered users connecting with clients, it does not apply to federation.

        I wasn't able to find any list of implemented or missing features.


        Ligase only has a bunch of buzzwords on their readme, and doesn't have any details on implemented stuff.


        Maelstrom has this note on their readme:

        This is a brand new project under daily active development. It is not currently in usable form yet.

        Their issue tracker is completely full of open matrix-spec issues, so I don't expect that to be a good sign


        Conduit is missing support for:

        • Appservices (Bridges and Bots)
        • Most federation features (invites, e2ee)
        • Push notifications on mobile
        • Notification settings

        So, while yes, there are alternatives, none of them seem like they could be trusted to not fall over and erase all data by accident.

        8 votes
        1. tindall
          Link Parent
          Yes, it's an in-dev protocol. I didn't post this to say, "Matrix should be used everywhere for everything!". I just think it's a cool bit of discussion about how they achieved a metric they were...

          Yes, it's an in-dev protocol. I didn't post this to say, "Matrix should be used everywhere for everything!". I just think it's a cool bit of discussion about how they achieved a metric they were working towards.

          2 votes
    2. [2]
      Happy_Shredder
      Link Parent
      I just use the docker container. Memory is cheap, and maintaining a container is pretty painless.

      I just use the docker container. Memory is cheap, and maintaining a container is pretty painless.

      1 vote
      1. admicos
        Link Parent
        The setup and maintenance isn't that big of a deal. Synapse has clear setup instructions instead of setup scripts that assume they're the only thing on the server and try to do everything by...

        I just use the docker container. [...] and maintaining a container is pretty painless.

        The setup and maintenance isn't that big of a deal. Synapse has clear setup instructions instead of setup scripts that assume they're the only thing on the server and try to do everything by themselves and force you to clean up afterwards when you just want to switch out the reverse proxy. (*cough* pihole *cough*)

        Memory is cheap

        This isn't a good assumption to make. As hardware yes, memory sticks can be pretty cheap depending on how you look at it. The worst computer in my house has 4GBs of it (originally had 2).

        But when you're talking about servers that should stay online for as long as they can, most people don't have the advantage of being able to host from their own hardware (crap ISPs, crap speeds, no datacenters close by, no spare hardware, etc etc...). So you turn to someone else to host the server for you, and you generally have to pay monthly for this. This is where stuff starts getting expensive.

        Now, I pay 5$/mo for my server, which might not be a huge deal at all for you. But remember everyone living in countries that don't use USD, EUR or other "common" currencies most places accept, and you suddenly need to know which currency you're dealing with to be able to make the claim of "memory is cheap".

        On top of that, add the financial situation or frugal-ness of the person you're talking about, and suddenly memory might not be as cheap as you'd expect.

        For larger operations like hosting Matrix.org itself, or even for smaller operations covered by donations and such, memory can be cheap compared to income and the rest of the expenses, but for someone hosting a couple things for themselves on a small server with pocket money, that is not always the case.

        5 votes
  2. [6]
    mrbig
    Link
    I’m literally watching the awful Matrix Revolutions and paused to look at the phone for a minute. I immediately thought this thread was about the movie. What a bad name for a software project.

    I’m literally watching the awful Matrix Revolutions and paused to look at the phone for a minute.

    I immediately thought this thread was about the movie.

    What a bad name for a software project.

    4 votes
    1. [5]
      Diff
      Link Parent
      The whole project suffers from that problem. More than average for open source software. The main client, I think it's named Element now, was Riot before and Vector before that.

      The whole project suffers from that problem. More than average for open source software. The main client, I think it's named Element now, was Riot before and Vector before that.

      7 votes
      1. [4]
        vaddi
        Link Parent
        All awful names for a messaging app, how are they not able to see it? The names neither convey the purpose of the app nor are easy to google, since all of them are overloaded by company names or...

        All awful names for a messaging app, how are they not able to see it? The names neither convey the purpose of the app nor are easy to google, since all of them are overloaded by company names or mathematics.

        2 votes
        1. [3]
          vord
          Link Parent
          This applies to a ton of stuff though. You just learn to append the type of program to the name until it hits sufficient searchability. "element chat" is sufficient. I go to a grocery store named...

          This applies to a ton of stuff though. You just learn to append the type of program to the name until it hits sufficient searchability. "element chat" is sufficient.

          I go to a grocery store named 'Giant'. Good luck guessing which one is the correct website.
          The word Discord has only negative connotations with chat.
          Tildes doesn't exactly reflect it's purpose in the name. I'm betting when userbase was low searchability was too.

          And that's just a tiny few off my head. Are there better names? Probably. But shedding Riot.IM was definitely a good idea because Element sounds more corporate friendly and will be easier to pull in clients to the paid offerings.

          2 votes
          1. [2]
            vaddi
            (edited )
            Link Parent
            While some people might do that, it is not everyone and even those that do it, most of the time is only on their second browsing try. This loses a lot of potential users before they even try the...

            This applies to a ton of stuff though. You just learn to append the type of program to the name until it hits sufficient searchability. "element chat" is sufficient.

            While some people might do that, it is not everyone and even those that do it, most of the time is only on their second browsing try. This loses a lot of potential users before they even try the app.

            I go to a grocery store named 'Giant'. Good luck guessing which one is the correct website.

            You have context there. People close to where you live automatically know what Giant is depending on the conversation. And Google uses your coordinates to give you what it thinks that you want.

            The word Discord has only negative connotations with chat.

            Consider other platforms:

            • Whatsapp is a pun, and people remember those with ease.
            • The names Messenger, Telegram, iMessages, Discourse, Conversations convey the act of communicating.
            • email, while not an app/platform, allows people to say "email me when you have time". You can use the name of the technology as a verb (this is also possible with Telegram).

            These are all ways of marking your brand in people's mind.

            I don't think Element offers any of those. It might be even hard for non-english speakers to say in a sentence using their native language.

            Tildes doesn't exactly reflect it's purpose in the name. I'm betting when userbase was low searchability was too.

            Tildes, while not a perfect name, has the benefit of being a word that is not common in English. People might search the word "tilde" but its plural is quite uncommon, I think. Moreover, since it is a website people might say "tildes dot net" in a conversation.

            But shedding Riot.IM was definitely a good idea

            Riot was a very bad name for a chatting app and you even have the company Riot Games that is impossible to beat at browsing results.

            1. lionirdeadman
              Link Parent
              "Messenger" aka Facebook Messenger might be what it says on the tin but it's quite generic. Same goes for "Conversations" which no one would call that in discussions. "Signal" has a similar...

              "Messenger" aka Facebook Messenger might be what it says on the tin but it's quite generic. Same goes for "Conversations" which no one would call that in discussions.

              "Signal" has a similar problem and it doesn't convey meaning.

              So while, yes, you could have a more related name, having a generic name is not such a problem imo.