Update 2023-01-31: Russ Cox of the Go team reached out to us to address this problem. After some discussion, an acceptable plan was worked out. The Go team is working on deploying an update to the “go” tool to add a -reuse flag, which should substantially reduce the traffic generated by this system for all users of Go.
In the meantime, the automated refresh traffic from proxy.golang.org was disabled for SourceHut, which the Go team assures us should have little-to-no impact on users and which reduces the burden on our system to a managable level. Following this change by the Go team, we have observed traffic from the Go module mirror reduced to an acceptable level. The Go team has decided that the automatic refresh behavior is their responsibility, not the responsibility of other operators, so any other small hosts will hopefully not be affected as the Go team will enable or disable the refresh behavior at their discretion with the burden on third-party operators in mind.
Consequently, we have cancelled our plans to disable Go traffic to git.sr.ht. No action is required by users to continue receiving service. Thanks Russ!
Drew says: The conversation on that linked issue says: I actually think he’s got some reasonable technical points in the post in general, but the framing seems dishonest as hell. Google offered an...
Drew says:
On February 24th, 2021, we reported an issue to the Go team regarding this problem. The Go team initially helped us narrow down the cause, first by setting an appropriate User-Agent to help identify this traffic, then through discussions regarding the behavior of this system. We made recommendations to Google for how to service their requirements without generating an excessive amount of redundant traffic. However, the discussion stalled and no further changes were made by Google to address the issue, and we continued to receive an excessive amount of traffic from the module mirror.
The conversation on that linked issue says:
Anyone who's receiving too much traffic from proxy.golang.org can request that they be excluded from the refresh traffic, as we did for git.lubar.me. Nobody asked for sr.ht be added to the exclusion set, so as far as it's concerned nothing has changed.
I actually think he’s got some reasonable technical points in the post in general, but the framing seems dishonest as hell. Google offered an interim solution 18 months ago, and clearly did so in earnest since they turned it around within a week for the owner of another domain, so I don’t see any reason they’d lie about not hearing anything from sr.ht.
Drew's reasons for not using the interim solution: Source: https://news.ycombinator.com/item?id=34313802
Drew's reasons for not using the interim solution:
For a number of reasons. For a start, what does disabling the cache refresh imply? Does it come with a degradation of service for Go users? If not, then why is it there at all? And if so, why should we accept a service degradation when the problem is clearly in the proxy's poor engineering and design?
Furthermore, we try to look past the tip of our own nose when it comes to these kinds of problems. We often reject solutions which are offered to SourceHut and SourceHut alone. This isn't the first time this principle has run into problems with the Go team; to this day pkg.go.dev does not work properly with SourceHut instances hosted elsewhere than git.sr.ht, or even GitLab instances like salsa.debian.org, because they hard-code the list of domains rather than looking for better solutions -- even though they were advised of several.
The proxy has caused problems for many service providers, and agreeing to have SourceHut removed from the refresh would not solve the problem for anyone else, and thus would not solve the problem. Some of these providers have been able to get in touch with the Go team and received this offer, but the process is not easily discovered and is poorly defined, and, again, comes with these implied service considerations. In the spirit of the Debian free software guidelines, we don't accept these kinds of solutions:
The rights attached to the program must not depend on the program's being part of a Debian system. If the program is extracted from Debian and used or distributed without Debian but otherwise within the terms of the program's license, all parties to whom the program is redistributed should have the same rights as those that are granted in conjunction with the Debian system.
Yes, being excluded from the refresh would reduce the traffic to our servers, likely with less impact for users. But it is clearly the wrong solution and we don't like wrong solutions. You would not be wrong to characterize this as somewhat ideologically motivated, but I think we've been very reasonable and the Go team has not -- at this point our move is, in my view, justified.
This is probably why the guy gets to me a bit: I often want the same outcomes he does, but a lot of the time I feel like his style is actively undermining the chances of that happening! Oh yeah,...
I like this part of his reasoning, and wish decision-makers (particularly ones who are less divisive / more diplomatic than Drew is) would take this approach more often.
This is probably why the guy gets to me a bit: I often want the same outcomes he does, but a lot of the time I feel like his style is actively undermining the chances of that happening!
the "solution" he was offered, and declined, is that he or anyone else can file a GitHub issue with the Golang team (at least, I think that's the process...AFAIK this isn't explicitly spelled out anywhere, other than in one of the 38 comments on a 2-year-old GitHub issue) to request exclusion
Oh yeah, it's far from being a reasonable general case fix (debate aside about whether it was ever intended as one) - but I've got to laugh at the irony of him making that important and valid point about poor discoverability and the importance of ideology... on a second-level hacker news comment in a thread that readers of the blog posts have no way of knowing about.
Yes, he’s wrong. Just not necessarily on the tech. He’s got good points about the software architecture, no question, but he isn’t primarily acting in his capacity as a programmer right now: he’s...
Yes, he’s wrong. Just not necessarily on the tech. He’s got good points about the software architecture, no question, but he isn’t primarily acting in his capacity as a programmer right now: he’s acting as lead decision maker and spokesperson for his organisation, and in that he’s simultaneously burning bridges and misleading his audience on the facts.
The cost of bearing this traffic is no longer acceptable to us, and the Go team has made no attempts to correct the issue during this time. We want to avoid causing inconvenience for Go users, but the load and cost is too high for us to continue justifying support for this feature.
Bullshit. There was an option to make that cost and load go away, and not only did he not engage with it, he didn’t even mention it in either blog post. He only addressed it when enough people asked that it was impossible to ignore. A guy as smart as he is doesn’t get to play dumb and claim that the omission was anything other than deliberate, and he doesn’t get to maintain the moral authority in the debate after being that misleading.
I think the most frustrating thing here is that if he’d just said “We’re choosing to turn go modules off entirely as a stand against their habit of making decisions we don’t like, and we don’t accept their compromise for the following reasons…” I’d probably be in this thread agreeing with him.
As it is, he spent two blog posts trying to claim it was about traffic and server load, both written after they offered to remove those two things as a concern entirely. He’s right to have technical and philosophical issues with the whole situation, and totally wrong to act as if the decision was made for dispassionate reasons of cost.
He made a subjective call specifically to put pressure on them, and that could have been fine. Just own it, be honest, and be up front about it.
It's a clunky one, but it's more seamless than his current plan. And it's also a temporary one, one whose time is almost over. Right before 1.19 released the go command got a -reuse flag pushed in...
It's a clunky one, but it's more seamless than his current plan. And it's also a temporary one, one whose time is almost over. Right before 1.19 released the go command got a -reuse flag pushed in after the feature freeze, it'll fix this problem once and for all once the proxies pick up support for this new flag. But it took time to rearchitect things. Kinda weird he's decided to resurrect this issue right as it's finally coming to an end.
That's refreshing. I don't agree with characterizing the Go team as unreasonable. These middling solutions are very common at my company and I detest them. But they are usually reasonable and...
That's refreshing. I don't agree with characterizing the Go team as unreasonable. These middling solutions are very common at my company and I detest them. But they are usually reasonable and pragmatic given the circumstances at the time. But I think it's healthy to ask them to do better.
Update: https://sourcehut.org/blog/2023-01-09-gomodulemirror/
I'm wondering what the impact is of this dispute. Are there popular Go modules on SourceHut?
I don't know how popular it is, but I do know the Gio GUI framework is hosted on SourceHut. And there are a few large enough to be packaged by Debian.
Drew says:
The conversation on that linked issue says:
I actually think he’s got some reasonable technical points in the post in general, but the framing seems dishonest as hell. Google offered an interim solution 18 months ago, and clearly did so in earnest since they turned it around within a week for the owner of another domain, so I don’t see any reason they’d lie about not hearing anything from
sr.ht.Drew's reasons for not using the interim solution:
Source: https://news.ycombinator.com/item?id=34313802
This is probably why the guy gets to me a bit: I often want the same outcomes he does, but a lot of the time I feel like his style is actively undermining the chances of that happening!
Oh yeah, it's far from being a reasonable general case fix (debate aside about whether it was ever intended as one) - but I've got to laugh at the irony of him making that important and valid point about poor discoverability and the importance of ideology... on a second-level hacker news comment in a thread that readers of the blog posts have no way of knowing about.
Seems like a very Drew situation to me: unwavering confidence that he's right and totally unwilling to compromise.
To me, this looks like an awfully clunky compromise for the reasons he stated already. So... is he wrong?
Yes, he’s wrong. Just not necessarily on the tech. He’s got good points about the software architecture, no question, but he isn’t primarily acting in his capacity as a programmer right now: he’s acting as lead decision maker and spokesperson for his organisation, and in that he’s simultaneously burning bridges and misleading his audience on the facts.
Bullshit. There was an option to make that cost and load go away, and not only did he not engage with it, he didn’t even mention it in either blog post. He only addressed it when enough people asked that it was impossible to ignore. A guy as smart as he is doesn’t get to play dumb and claim that the omission was anything other than deliberate, and he doesn’t get to maintain the moral authority in the debate after being that misleading.
I think the most frustrating thing here is that if he’d just said “We’re choosing to turn go modules off entirely as a stand against their habit of making decisions we don’t like, and we don’t accept their compromise for the following reasons…” I’d probably be in this thread agreeing with him.
As it is, he spent two blog posts trying to claim it was about traffic and server load, both written after they offered to remove those two things as a concern entirely. He’s right to have technical and philosophical issues with the whole situation, and totally wrong to act as if the decision was made for dispassionate reasons of cost.
He made a subjective call specifically to put pressure on them, and that could have been fine. Just own it, be honest, and be up front about it.
It's a clunky one, but it's more seamless than his current plan. And it's also a temporary one, one whose time is almost over. Right before 1.19 released the go command got a -reuse flag pushed in after the feature freeze, it'll fix this problem once and for all once the proxies pick up support for this new flag. But it took time to rearchitect things. Kinda weird he's decided to resurrect this issue right as it's finally coming to an end.
That's refreshing. I don't agree with characterizing the Go team as unreasonable. These middling solutions are very common at my company and I detest them. But they are usually reasonable and pragmatic given the circumstances at the time. But I think it's healthy to ask them to do better.