Careless big-time users are treating FOSS repos like content delivery networks
I'm at the Linux Foundation Members Summit, and Sonatype's CTO Brian Fox introduced me to a new open source problem. I wouldn't have thought that was possible, but here I am.
Fox, who also oversees Apache Maven, a popular Java build tool, explained that its repository site is at risk of being overwhelmed by constant Git pulls. The team has dug into this and found that 82 percent of the demand comes from less than 1 percent of IPs. Digging deeper, they discovered that many companies are using open source repositories as if they were content delivery networks (CDNs). So, for example, a single company might download the same code hundreds of thousands of times in a day, and the next day, and the next. This is unsustainable.
So Maven and other open source repositories are considering introducing a tiered payment system. Lone developers and small groups will still be able to download the code for free, but the hogs will have to pay for every download. In other words, open source software is still free as in speech, but you can forget about being "free as in beer" going forward.
How bad is it? Fox revealed that last year, major repositories handled 10 trillion downloads. That's double Google's annual search queries if you're counting from home and they're doing it on a shoestring. Fox described this as a "tragedy of the commons," where the assumption of "free and infinite" resources leads to structural waste amplified by CI/CD pipelines, security scanners, and AI-driven code generation.
...read more at theregister.com
pull down to refresh
related posts
I don't understand, what is the benefit of doing this?
pay for git pull or companies use open source repositories?
No, why would someone need to download the same code hundreds of thousands of times a day?
The download is part of some build process script and it's easier to download from the source on every build than manage a cache.
Oh, huh. I guess that makes sense? I don't quite know how these package managers work. I thought you download it once and now you have a copy on your machine. You only need to redownload if upgrading to a new version
They do, but these are likely CI builds. These builds run on containers/vms spawned from scratch for each commit, they build, validate, publish (if a release) and then self destroy.
It is a good way to avoid any possible cross contamination between builds for security and reproducibility.
Caching here, without compromises, can be quite complex depending on the underlying system, and might require a custom build script for CI, that is often undesirable, because ideally the CI build should follow the same path as a manual build whenever possible, so it can also catch bugs in the build script itself.
The people complaining about this are the hosts of Maven Central (Java's npm), and they caused this themselves by pushing for Java build centralization. You can configure a project to use any Maven repository in Java, but it is so inconvenient and looks so suspicious or hard to verify that everyone ended up publishing to Maven Central, going through all the annoying hoops they put in place. Somehow they did not consider that they would have to host the entire world?
I think we need a nostr alternative: metadata signed and curated on nostr, payloads hosted on the dev own servers.
They probably aren't using a package manager, or at least one that persists between builds, because package managers cache.
I see. Custom build script then? Coz I thought even amateurs just use package managers most of the time like
npmandpipYep custom and these build systems often don't persist the package manager's cache - preferring to start from a blank slate.
People panicking about AI has led directly to this and we have nobody to blame but ourselves.
Using docker to re-build open source wallets I guess I did this a lot, too. Not sure how to best avoid it on my end as I would really prefer to not have to wait for downloads that I did 10min ago but the tools aren't ready.
Introducing a price makes sense. This way, incentives align. If only there wasn't the inefficient payment methods and tax implications and ... Just attach some sats to the request.