A few days ago I watched a Q/A session with Linus Torvalds at Debconf 14. One of the main complaint of Linus towards Linux distribution was the way that distribution ends up using different versions of libraries than what has been used during application development. And the fact that it’s next to impossible to support properly all Linux distributions at the same time due to this kind of differences.
And now I just discovered a new proposal of the systemd team that basically tries to address this: Revisiting how we put together Linux Systems.
They suggest to make extensive use of btrfs subvolumes to host multiple variants of the /usr tree (that is supposed to contain all the invariant system code/data) that you could combine with multiple runtime/framework subvolumes thanks to filesytem namespaces and make available to individual applications.
This way of grouping libraries in “runtime subvolumes” reminds me a bit of the concepts of baserock (they are using git instead of btrfs) and while I was a bit dubious of all this (because it goes against quite a few of the principles of distribution integration) I’m beginning to believe that there’s room for both models to work together.
It would be nice if Debian could become the reference distribution that upstream developers are using to develop against Linux. This would in turn mean that when upstream distribution their application under this new form, they will provide (or reference) Debian-based subvolumes ready for use by users (even those who are not using Debian as their main OS). And those subvolumes would be managed by the Debian project (probably automatically built from our collection of .deb).
We’re still quite far from this goal but it will interesting to see this idea mature and become reality. There are plenty of challenges facing us.
This reminds of me of the cadence discussion from years ago.
http://www.markshuttleworth.com/archives/tag/cadence
Sadly, we are trying to find technical solutions to an organization problem. With cadenced releases for the main components of an OS, we would just avoid all the combinations that are tested across all distros.
I also agree with the cadence proposal by Mark. I don’t know if it would solve all the problems, but Ubuntu predictability and LTS are making sense for a lot of other people to rely on.
I was certainly in favor of cadenced releases, but the world is not a simple place and I don’t see any way to get everybody to release everything at the same time. In particular when you have many layers of dependencies. Some software need to be adjusted when other software comes out. It’s difficul to synchronize such releases when they are handled by two separate projects.
So it’s certainly worth exploring options to be able to keep some software running with older (yet still supported) versions of libraries for example. To take a concrete example, we’re going to upload Django 1.7 to Debian Jessie and this will break quite a few web applications. If they get updated in time, everything is fine, if not they will be removed from Jessie. Such removals are problematic for users… it would be nice if users could keep those app running with the help of a wheezy-based runtime.
Relying on features of a particular file system seems like a very bad idea. Shipping different versions of dependencies seems a lot like what distros like Nixos already do.
Assuming that the user will be using exactly the same libraries as on the development system seems like bad engineering practice anyway. It’s better to rely on/develop against *documented* behavior, not observed behavior.
This distribution problem only really affects those who want to distribute proprietary software anyway. That’s not really a distribution problem I have any interest in solving.
I think that once the described distribution model would be widespread even many free software projects would take benefit from it. I easily see people using LibreOffice as distributed by upstream rather than needing an entire team dedicated to repackage it for their distribution.
Likely GNOME applications would no longer need to be packaged by Debian maintainers and users could run them directly from upstream while they depend on the GNOME runtime. KDE could do the same.
Not to mention all the troubles the Debian maintainer have to go through to repackage Firefox/Iceweasel, which would go away and users would be free to use the latest and greatest directly from upstream.
James, I don’t like your negativity. You might not be interested but others are interested, even outside of the proprietary software realm. I quoted Linus Torvalds and his use case was subsurface, a free software project for a niche of users (divers).
Relying on btrfs doesn’t look like a good idea but it has interesting attributes for the specific use-case. Is might be possible to us something else at this layer, possibly based on git!
And while nixos already does some of the things that this proposal is trying to solve, I believe that the approach suggested here is much cleaner (it works with a sttandard filesystem organization).
YMMV.
How many of you “other thinkers” are there? You might not like his negativity, that’s your choice, but it doesn’t mean you are the only one right here.
And I wouldn’t rely on Linus’ word on this one. He is too “pro-proprietary” (as much as has can be in his position anyway) for that.
A similar approach uses MacOS for a long time, not subvolumes, but folders, where everything (all required libraries) is stored. This only applies to GUI like application, the rest of the core freebsd based system is still unixoidal. Therefore, the applications are simply copied to another computer and work out of the box.
I similar system for Debian or Linux in general is often applies by commercial companies, which install their stuff to /opt/ to circumvent the problems Linus discussed – also a doable approach, yet I always liked the apt-get way to installing things.
Maybe a docker based application virtualization approach could also be used, where every user can have their own application space with their own libraries.
Yep, that’s exacly what is being proposed in the “Revisiting how we put together Linux Systems” blog post linked by Raphaël: http://0pointer.net/blog/revisiting-how-we-put-together-linux-systems.html
The proposal goes beyond what Docker does as Docker is very oriented to the server usecase, taking some good lessons from it and trying to generalize its concepts.
While it has similarities with MacOS in the sense that you can bundle your libraries together with your app, it’s also very different due to the reliance on distributions-provided “runtime”. While an embedded library will never be updated by distributions, the distribution-provided runtimes will benefit from the security work of distributions and upstream developers will thus avoid much of the problems of bundling libraries while still being able to count on a specific version of a library.
Although it theoretically could avoid the bundled library security problems, it heavily increases the maintenance burden of a distribution/maintainer. Security problems are going to be rampant with old and unused libraries, leading distros to either break compatibility with apps by removing the old versions or be less secure by allowing them. The idea would not actually subtract from the maintenance burden of distributions, but instead explode it.
One example is LLVM: Julia (julialang.org) needs 3.3, but the maintainer only wants 3.4 and 3.5. For GUI applications, I have debs that are built against versions of libraries that are several **symbol bumps** away from what is in Debian Sid right now, despite being built against Ubuntu 14.04.
I think OpenSUSE’s Open Build System is a much more worthwhile idea: developers should be able to deploy one RPM builder and one DEB builder that targets multiple distros and versions with ease. With the OBS, all the craziness is on the build machine, not the end user’s computer.
OBS doesn’t help when you don’t have the required build-dependency for the target distribution because that distribution moved to the newer incompatible version of the package.
Software distribution aside, I really like the idea of having separately encrypted volumes for each user, enumerating users via an NSS module that enumerates those volumes, and authenticating users via a PAM module that decrypts those volumes.
The problem with this bundling of libraries is what happens when one ends up with 27 different versions of libssl, bundles into a load of desktop aplications, and then someone points out that most of them suffer from heartbleed. What are your chances of getting to a state where you know that you’ve fixed the problem?
Apart from that, I thought that this problem had been solved years ago, by klik ( https://en.wikipedia.org/wiki/Klik ) … apparently that’ has since become http://portablelinuxapps.org/
Phil, I think you missed the fact that libraries are provided by distributions in dedicated “runtime” subvolumes that can be (auto-)upgraded to fix security issues. Since those are not managed by application developers but by distributions, the chance of getting everything updated is much higher than if they were tightly bundled with all the applications.
Ah, I see what you mean, although the runtimes are being provided by more than just distributions, since we have the example of Gnome as one runtime producer. Also, for less popular libraries that do not make it into the runtimes, they’ll be bundled in with the aplications, so it’s better than total chaos, but it still seems to have the potential for trouble.
I suppose it does provide an environment where the better runtimes should have chance to win support from the population of application builders, thus encouraging excelence, which would be no bad thing.
> OBS doesn’t help when you don’t have the required build-dependency for the target distribution because that distribution moved to the newer incompatible version of the package.
Right, but a few core libraries like Qt and GTK have parallel installations for when API breaks. For example, I will take maya-calendar’s dependencies (http://bazaar.launchpad.net/~elementary-os/maya/deb-packaging/view/head:/debian/control):
gtk: versioned for API breaks
gee: versioned
glib: versioned
clutter: same
champlain: same
champlain-gtk: versioned
ecal: versioned
There are 5 unversioned libs: ical, granite, geocode-glib, notify, folks. The seven versioned ones are probably the most used and prone to build errors from API breaks, while the other five have simple APIs that will probably not break for a while.
So yes, the systemd proposal does have a leg up on API breaks, but it is also very costly in the form of overuse of btrfs and mounts as well as maintainership costs for older libraries.
The use of easy to set up automated build systems solves ABI breakage completely, and still allows for distros to have co-installable versions of libs (like Nix/Guix or like gtk) to provide complete or partial protection against API breaks.
I think you’re both right: OBS is a build tool, while the systemd proposal is about distribution. Using OBS to generate the packages which will compose the runtimes/application bundles seems a very sensible solution.
Note that the proposal relies on btrfs deduplication to avoid wasting space due to the same library being shipped in diffferent runtimes. This won’t solve the upgrade problem, but that’s the tradeoff of the proposal: each application shipping a library not included in a runtime will need to take care of updates, increasing the cost of maintainership a bit (pushing a security update in eg. OBS shouldn’t take much) while decreasing the complexity of the test matrix *a lot*. 🙂