Debian – Page 77

5 reasons why I still contribute to Debian after 12 years

October 11, 2010 by Raphaël Hertzog

If you’re using Debian, you know that this distribution is built entirely by volunteers that form a very diverse community. And you could be part of it. But why should you do that? I can’t tell for you but I can share my own experience. It’s been 12 years since I joined Debian and I’m going to tell you what keeps me on board.

1. Technical excellence

When facing a new challenge, Debian will strive to do the right thing. This pays off in the long term. In many cases, it means that we will take more time to implement our solution compared to other distributions out there, but this is also the reason why our packaging infrastructure allows us to offer painless upgrades and consistency across all packages.

Debian is committed to quality and builds up on his experience thanks to the Debian Policy. My time is precious, I like to spend it on something useful in the long term.

2. Inspiring goals

In its social contract, Debian has set out to create a 100% free — as in freedom — operating system. The criteria defining what constitutes a free work are listed in the Debian Free Software Guidelines (DFSG).

While the above is relatively ambitious in itself, it doesn’t inspire me much. What makes the difference to me is the emphasis given by the social contract on our users and free software. We don’t build a free operating system in the void, we build it for people.

Debian’s motto — the universal operating system — can also be interpreted in multiple ways: universal as in “for everybody on the world”, “on all kinds of computers”, or “for every possible usage”.

3. High impact work

Knowing that my work is useful to people is important, but it’s even better when I know that it will benefit to lots of people. With Ubuntu and the hundreds of other derivatives, there are nowadays literally millions of users impacted by my work. Even an insignificant one second improvement experienced by 10 millions of people represent 115 days of time saved for something else, you get the idea…

4. Working with great people

Debian has the chance to have lots of smart people on board. There’s always someone sharing valuable advice when you read Debian’s mailing lists. When I joined in 1998, I was a real newbie and I learned a lot of things by reading and interacting with people more knowledgeable than me. You can still experience the same thing nowadays but there’s one caveat: you must cope with various kinds of mailing list contributors including the “smart but uncivilized” (don’t be offended too quickly!) and the occasional troll (best ignore it, don’t feed it!).

5. Recognition of work

When you contribute to Debian, people get to know you through your contributions. It’s very rewarding to be thanked by your peers and by Debian’s users. Check out thanks.debian.net to convince you that many people are grateful for the work we put into Debian.

So that’s it for me. But what about you? What motivates you to contribute year after year, or to start contributing if you’re a prospective contributor?

Subscribe to this blog (it’s free!) by RSS, by email, or on Facebook.

The right way to remove an obsolete conffile in a Debian package

October 7, 2010 by Raphaël Hertzog

A conffile is a configuration file managed by dpkg, I’m sure you remember the introductory article about conffiles. When your package stops providing a conffile, the file stays on disk and it’s recorded as obsolete by the package manager. It’s only removed during purge. If you want the file to go away, you have to remove it yourself within your package’s configuration scripts. You will now learn how to do this right.

When is that needed?

dpkg errs on the side of safety by not removing the file until purge but in most cases it’s best to remove it sooner so as to not confuse the user. In some cases, it’s even required because keeping the file could break the software (for example if the file is in a .d configuration directory, and if it contains directives that are either no longer supported by the new version or in conflict with other new configuration files).

What’s complicated in “rm”?

So you want to remove the conffile. Adding an “rm” command in debian/postinst sounds easy. Except it’s not the right thing to do. The conffile might contain customizations made by the administrator and you don’t want to wipe those. Instead you want to keep the file around so that he can get his changes back and do whatever is required with those.

The correct action is thus to move the file away in the prerm, to ensure it doesn’t disturb the new version. At the same time, you need to verify whether the conffile has been modified by the administrator and remember it for later. In the postinst, you need to remove the file if it’s unmodified, or keep it under a different name that doesn’t interfere with the software. In many cases adding a simple .dpkg-bak suffix is enough. For instance, run-parts ignore files that contain a dot, and many other software are configured to only include files with a certain extension—say *.conf. In the postrm, you have to remove the obsolete conffiles that were kept due to local changes and you should also restore the original conffile in case the upgrade obsoleting the conffile is aborted.

Automating everything with dpkg-maintscript-helper

Phewww… that’s a lot of things to do for a seemingly simple task. Fortunately everything can be automated with dpkg-maintscript-helper. Let’s assume you want to remove /etc/foo/conf.d/bar because it’s obsolete and you’re going to prepare a new version 1.2-1 with the appropriate code to remove the file on upgrade. You just have to put this snippet in the 3 relevant scripts (preinst, postinst, postrm):

if dpkg-maintscript-helper supports rm_conffile 2>/dev/null; then
    dpkg-maintscript-helper rm_conffile /etc/foo/conf.d/bar 1.2-1 -- "$@"
fi

You can avoid the preliminary test if you pre-depend on “dpkg (>= 1.15.7.2)” or if enough time has passed to assume that everybody has a newer version anyway. You can learn all the details in dpkg-maintscript-helper’s manual page.

Debhelper integration

debhelper makes it easy to inject those commands for you. You can provide debian/*.maintscript files. See dh_installdeb’s manual page for details.

I hope you found this article helpful. You can follow me on Identi.ca, Twitter and Facebook.

Can Debian offer a Constantly Usable Testing distribution?

October 4, 2010 by Raphaël Hertzog

Debian’s “testing” distribution is where Debian developers prepare the next stable distribution. While this is still its main purpose, many users have adopted this version of Debian because it offers them a good trade-off between stability and freshness. But there are downsides to using this distribution and the “Constantly Usable Testing” (CUT) project aims to resolve those. This article will present the project and the challenges involved to make it happen.

About Debian unstable & testing

Debian unstable is the distribution where developers upload new versions of their packages. It happens frequently that some packages are not installable due to changes in other packages or due to transitions not yet completed.

Debian testing, on the contrary, is managed by a tool that ensures the consistency of the whole distribution: it picks updates from unstable only if the package has been enough tested (10 days usually), if it’s free of new release-critical bugs, if it’s available on all supported architectures, and if it doesn’t break any other package already present in testing. The Release Team (RT) controls this tool and provide “hints” to help it find a set of packages that can flow from unstable to testing.

Those rules also ensure that the packages that flow into testing are reasonably free of show-stopper bugs (like a system that doesn’t boot, or X that doesn’t work at all). This makes it very attractive to users who like to regularly get new upstream versions of their software without dealing with the biggest problems associated to them. This is all very attractive, yet several Debian developers advise people to not use testing. Why is that?

Known problems with testing

Disappearing software

The release team use this distribution to prepare the next stable release and from time to time they remove packages from it. Either because it’s needed to ensure that other packages can migrate from unstable to testing, or because they have long-standing release-critical bugs without progress towards a resolution. It also happens that they remove packages on request of the maintainers because they believe that the current version of the software cannot be supported (security-wise) for 2 years or more. The security team also regularly issues such requests.

Long delays for security and important fixes

Despite the 10-day delay in unstable, there are always some annoying bugs (and security bugs are no exceptions) that are only discovered when the package already has migrated to testing. The maintainer might be quick to upload a fixed package in unstable, and might even raise the urgency to allow the package to migrate sooner, but if the packages gets entangled in a large ongoing transition, it will not migrate before the transition is completed. Sometimes it can take weeks for that to happen.

This delay can be avoided by doing direct uploads to testing (through testing-proposed-updates) but this is almost never used, except during a freeze, where targeted bugfixes are the norm.

Not always installable

With testing evolving daily, updates sometimes break the last installation images available (in particular netboot images that get everything from the network). The debian-installer (d-i) packages are usually quickly fixed but they don’t move to testing automatically because the new combination of d-i packages has not necessarily been validated yet. Colin Watson sums up the problem:

Getting new installer code into testing takes too long, and problems remain unfixed in testing for too long. […] The problem with d-i development at the moment is more that we’re very slow at producing new d-i *releases*. […] Your choices right now are to work with stable (too old), testing (would be nice except for the way sometimes it breaks and then it tends to take a week to fix anything), unstable (breaks all the time).

CUT’s history

CUT finds its root in an old proposal by Joey Hess: it introduces the idea that the stable release is not Debian’s sole product and that testing could become — with some work — a suitable choice for end-users. Nobody took on that work and there was no visible progress in the last 3 years.

But recently Joey brought up CUT again on the debian-devel mailing list and Stefano Zacchiroli (the Debian project leader) challenged him to setup a BoF on CUT for Debconf10. It turned out to be one of the most heavily attended BoF (video recording is here), there is clearly a lot of interest in the topic.

There’s now a dedicated wiki and an Alioth project with a mailing list. The rest of this article tries to summarize the various options discussed and how they’re supposed to address the problems identified.

The ideas behind CUT

Among all the ideas, there are two main approaches that have been discussed. The first is to regularly snapshot testing at points where it is known to work reasonably well (those snapshots would be named “cut”). The second is to build an improved testing distribution tailored to the needs of users who want a working distribution with daily updates, its name would be “rolling”.

Regular snapshots of testing

There’s general agreement that regular snapshots of testing are required: it’s the only way to ensure that the generated installation media will continue to work until the next snapshot. If tests of the snapshot do not reveal any major problem, then it becomes the latest “cut”. For clarity, the official codename would be date based: e.g. “cut-2010-09” would be the cut taken during September 2010.

While the frequency has not been fixed yet, the goal is clearly to be on the aggressive side: at the very least every 6 months, but every month has been suggested as well. In order to reach a decision, many aspects have to be balanced.

One of them (and possibly the most important) is the security support. Given that the security team is already overworked, it’s difficult to put more work on their shoulders by declaring that cuts will be supported like any stable release. No official security support sounds bad but it’s not necessarily so problematic as one might imagine. Testing’s security record is generally better than stable’s one (see the security tracker) because fixes flow in naturally with new upstream versions. Stable still get fixes for very important security issues sooner than testing, but on the whole there are less known security-related problems in testing than in stable.

Since it’s only a question of time until the fixed version comes naturally from upstream, more frequent cut releases means that users get security fixes sooner. But Stefan Fritsch, who used to be involved in the Debian testing security team, has also experienced the downside for anyone who tries to contribute security updates:

The updates to testing-security usually stay useful only for a few weeks, until a fixed version migrates from unstable. In stable, the updates stay around for a few years, which gives a higher motivation to spend time on preparing them.

So if it’s difficult to form a dedicated security team, the work of providing security updates comes back to the package maintainer. They are usually quite quick to upload fixed packages in unstable but tend to not monitor whether the packages migrate to testing. They can’t be blamed because testing was created to prepare the next stable release and there is thus no urgency to get the fix in as long as it makes it before the release.

CUT can help in this regard precisely because it changes this assumption: there will be users of the testing packages and they deserve to get security fixes much like the stable users.

Another aspect to consider when picking a release frequency is the amount of associated work that comes with any official release: testing upgrades from the previous version, writing release notes and preparing installation images. It seems difficult to do this every month. With this frequency it’s also impossible to have a new major kernel release for each cut (since they tend to come out only every 2 to 3 months) and the new hardware support that it brings is something worthwhile to many users.

In summary, regular snapshots address the “not always installable” problem and changes the perception of maintainers towards testing, so that hopefully they care more of security updates in that distribution (and in cuts). But they do not solve the problem of disappearing packages. Something else is needed to fix that problem.

A new “rolling” distribution?

Lucas Nussbaum pointed out that regular snapshots of Debian is not really a new concept:

How would this differentiate from other distributions doing 6-month release cycles, and in particular Ubuntu, which can already be seen as Debian snapshots (+ added value)?

In Lucas’s eyes, CUT becomes interesting if it can provide a rolling distribution (like testing) with a “constant flux of new upstream releases”. For him, that would be “something quite unique in the Free Software world”. The snapshots would be used as starting point for the initial installation, but the installed system would point to the rolling distribution and users would then upgrade as often as they want. In this scenario, security support for the snapshots is not so important, what matters is the state of the rolling distribution.

If testing were used as the rolling distribution, the problem of “disappearing packages” would not be fixed. That’s why there have been discussions of introducing a new distribution named “rolling” that would work like testing but with adapted rules, and the cuts would then be snapshots of rolling instead of testing.

The basic proposal is to make a copy of testing and to re-add the packages which have been removed because they are not suited for a long term release while they are perfectly acceptable for a constantly updated release (the most recent example being Chromium).

Then it’s possible to go one step further: during freeze, testing is no longer automatically updated which makes it inappropriate to feed the rolling distribution. That’s why rolling would be reconfigured to grab updates from unstable (but using the same rules than testing).

Given the frequent releases, it’s likely that only a subset of architectures would be officially supported. This is not a real problem because the users who want bleeding edge software tends to be desktop users on mainly i386/amd64 (and maybe armel for tablets and similar mobile products). This choice — if made — opens up the door to even more possibilities: if rolling is configured exactly like testing but with only a subset of the architectures, it’s likely that some packages migrate to rolling before testing when non-mainstream architectures are lagging in terms of auto-building (or have toolchain problems).

While being ahead of testing can be positive for the users, it’s also problematic on several levels. First, managing rolling becomes much more complicated because the transition management work done by the release team can’t be reused as-is. Then it introduces competition between both distributions which can make it more difficult to get a stable release out, for example if maintainers stop caring of the migration to testing once the migration to rolling has been completed.

The rolling distribution is certainly a good idea but the rules governing it must be designed to avoid any conflict with the process of releasing a stable distribution. Lastly, the mere existence of rolling would finally fix the marketing problem plaguing testing: the name “rolling” does not suggest that the software is not yet ready for prime time.

Conclusion

Whether CUT will be implemented remains to be seen, but it’s off for a good start: ftpmaster Joerg Jaspert said that the new archive server can cope with a new distribution, and there’s now a proposal shaping up. The project might start quickly: there is already an implementation plan for the snapshot side of the project. The rolling distribution can always be introduced later, once it is ready. Both approaches can complement each other and provide something useful to different kind of users.

The global proposal is certainly appealing: it would address the concerns of obsolescence of Debian’s stable release by making intermediary releases. Anyone needing something more recent for hardware support can start by installing a cut and follow the subsequent releases until the next stable version. And users who always want the latest version of every software could use rolling after having installed a cut.

From a user point of view, there are similarities with the mix of usual and long term releases of Ubuntu. But from the development side, the process followed would be quite different, and the constraints imposed by having a constantly usable distribution are stronger: any wide-scale change must be designed in a way that it can happen progressively in a transparent manner for the user.

This article was first published in Linux Weekly News. If you want to see more articles like this, join Flattr and click on the flattr button below every article that you like.

How to generate different dependencies on Debian and Ubuntu with a common source package

September 27, 2010 by Raphaël Hertzog

There are situations in which a given package needs to have different dependencies in Debian and in Ubuntu. Despite this difference it’s possible to keep a single source package that will build both variants of the package. Continue reading to discover a step-by-step explanation.

1. When is that needed?

While it is possible to have different dependencies depending on the distribution on which the package is built, it should be usually avoided when possible. This infrastructure should only be used as a last resort when there are no better alternatives.

Here are some examples of when it might be needed:

Ubuntu has some packages that Debian does not have (or vice-versa), and the resulting package would benefit from having them installed.
The package names differ between Ubuntu and Debian and it’s on purpose (i.e. the difference is a justified choice and not a mistake because both distributions failed to coordinate).
The packages are built differently in both distributions, and the run time dependencies are not the same due to this. Maybe some associated patches are only applied in Ubuntu.

2. Using substitution variables in debian/control

The dependency that varies between both distributions can’t be hardcoded in debian/control, instead you should put a substitution variable (substvar) that will be replaced at build time by dpkg-gencontrol. You can name it ${dist:Depends} for example:

[...]
Depends: bzip2, ${shlibs:Depends}, ${misc:Depends}, ${dist:Depends}
[...]

Note that you typically already have other substitution variables (${shlibs:Depends} comes from dpkg-shlibdeps and ${misc:Depends} from debhelper and its dh_* scripts).

3. dpkg-gencontrol needs a -V option

The value used to replace this new variable needs to be communicated to dpkg-gencontrol. You can use the -V option for this, the syntax would be something like this:

dpkg-gencontrol [...] -Vdist:Depends="foo (>= 2), bar"

If you use debhelper, you have to pass the option to dh_gencontrol after two dashes (--):

dh_gencontrol -- -Vdist:Depends="foo (>= 2), bar"

If you use CDBS, you can set the DEB_DH_GENCONTROL_ARGS_ALL make variable:

include /usr/share/cdbs/1/rules/debhelper.mk
DEB_DH_GENCONTROL_ARGS_ALL = -- -Vdist:Depends="foo (>= 2), bar"

The value given to dpkg-gencontrol is static in all those examples, now let’s see how we can use give a different value depending on the distribution that we’re targetting.

4. Using dpkg-vendor in debian/rules

dpkg-vendor is a small tool (provided by the dpkg-dev package) that parses the /etc/dpkg/origins/default file (provided by the base-files package) to know the current distribution and its ancestry. It can be used in debian/rules to adjust the behavior depending on the current distribution. You can check its man-page to learn about the various options supported but we’re only going to use --derives-from <vendor> in this sample. With this option the script exits with zero if the current distribution is or derives from the indicated distribution, or with 1 otherwise.

Now combining all together, we can use dpkg-vendor to dynamically define the content of the substitution variable in debian/rules. Let’s suppose that you want ${dist:Depends} to be “foo (>= 2)” on Ubuntu (and its derivatives) and “bar” everywhere else. Using debhelper’s 7 tiny rules file, this example could be:

ifeq ($(shell dpkg-vendor --derives-from Ubuntu && echo yes),yes)
	SUBSTVARS = -Vdist:Depends="foo (>= 2)"
else
	SUBSTVARS = -Vdist:Depends="bar"
endif

%:
	dh $@

override_dh_gencontrol:
	dh_gencontrol -- $(SUBSTVARS)

If you use CDBS, it could be this:

include /usr/share/cdbs/1/rules/debhelper.mk
ifeq ($(shell dpkg-vendor --derives-from Ubuntu && echo yes),yes)
	DEB_DH_GENCONTROL_ARGS_ALL = -- -Vdist:Depends="foo (>= 2)"
else
	DEB_DH_GENCONTROL_ARGS_ALL = -- -Vdist:Depends="bar"
endif

Do you want to read more tutorials like this one? Click here to subscribe to my free newsletter, you can opt to receive future articles by email.