release

How to squash Debian release critical bugs

July 14, 2011 by Raphaël Hertzog

Squashing release critical (RC) bugs is one of the most helpful way to contribute to Debian. Because those are the bugs that we need to get rid of in order to release the next stable version. Let’s see how you can help.

The process is really simple:

Find a RC bug that you can fix
Prepare a fixed package
Send the patch to the BTS
Get your fixed package uploaded

I have already covered points 2 and 3 in a former article: How to prepare patches for Debian packages. So I’m not going to repeat myself in this article, instead I’ll cover how to find a release critical bug to fix and how to get your package uploaded.

Find a RC bug with UDD

The Ultimate Debian Database (UDD) provides a neat interface to query the bug tracking system: http://udd.debian.org/bugs.cgi. The form contains many options to customize the query, we’ll try explain some.

First, at the top, you can decide to list bugs based on whether or not they affect testing/stable/unstable. Indeed, thanks to version tracking the BTS knows if a bug applies to the version of the package present in a given distribution. I suggest you pick a choice that includes “sid” because that’s where you’re going to push your fixed package. If sid is not affected by the bug, you can’t do much about it (except if you want to join the release team and help ensure updates are flowing from unstable to testing).

I usually go for “wheezy and sid” because that filters out packages which are not in testing (and are thus not blocking the next release).

Then you have lots of filters to further reduce the list of bugs. It’s important to reduce the noise as you typically have several hundreds of RC bugs at a given time. I suggest you the following settings for a start, but you’re free to try other combinations depending on what you want to achieve (I’ll give some examples later):

ignore bugs tagged patch (a fix is already available);
ignore bugs tagged pending (a fix has already been committed to the VCS repository);
ignore bugs newer than 7 days (that’s the time we should leave for the maintainer to take care of the bug by himself);

I usually sort the results by source package so that I immediately see if several RC bugs are affecting the source package. And I might add “popularity contest” information to get a feel of whether the package is widely used or not.

You can see the results of the suggested query here.

Other interesting queries:

List of RC bugs blocking the migration of their package (i.e. affecting unstable but not testing, with different version in both distributions).
List of RC bugs on packages that you can adopt (i.e. bugs affecting any distribution but on orphaned packages, sorted by decreasing popularity).

Find a RC bug with rc-alert

The downside of the above method is that you get bugs on packages that you don’t know at all and that you might not care about. Some of them might even be better removed instead of fixed.

With the rc-alert command (provided by the devscripts package), you will get a list of RC bugs on packages that are installed on your computer. It should be much more restrictive. And you have command line options to further filter the list just like with the UDD form. Bonus feature: you can also filter by debtags allowing you to restrict the bugs to packages which are implemented in programming languages that you know.

$ rc-alert --include-dists TU --exclude-tags P+ --debtags implemented-in::perl,implemented-in::python
Package: gwibber
Bug:     608724
Title:   gwibber bypasses certificate checking when providing the login/password for OAuth
Flags:   [     S  ] (security)
Dists:   [STU] (stable, testing, unstable)
Debtags: implemented-in::python, interface::x11, role::program, uitoolkit::gtk, works-with::im, x11::application
[...]

Get your fixed package uploaded

Usually sending the patch to the BTS is more than enough. The maintainer or another Debian developer will pick it up at some point. But if you’re interested in becoming a Debian maintainer/developer, it might be worth to push your changes more quickly and get credited by a proper upload with your name.

Instead of just preparing a patch, you should also prepare a fixed package following the rules of a Non-Maintainer Upload. Put this package somewhere online (for example on mentors.debian.net).

When you send your patch to the BTS, include the link to the updated package and express your wish to get this update sponsored. If you haven’t heard anything from the maintainer after 2-3 days, try to find a sponsor on debian-mentors@lists.debian.org. NMUs are easier to review than completely new packages and many Debian developers like to kill RC bugs, so there’s a good chance that you’ll find someone.

Now happy RC bug-squashing! And if you really enjoy it, you can subscribe to debian-bugs-rc (~3000 mails per month)… 🙂

If you want to read more articles like this one, click here to subscribe to my free newsletter. You can also follow me on Identi.ca, Twitter and Facebook.

People behind Debian: Philipp Kern, Stable Release Manager

June 10, 2011 by Raphaël Hertzog

Philipp is a Debian developer since 2005 and a member of the release team since 2008. Since he took the responsibility of Stable Release Manager, the process has evolved for the best. I asked him to explain how the release team decides what’s fit for stable or not.

His work on the buildd infrastructure is also admirable but I’ll let him describe that. My questions are in bold, the rest is by Philipp.

Who are you?

I’m a 24 year old Computer Science student, living in Karlsruhe, Germany. As a student assistant I’m currently in two jobs: one is taking care of getting our university IPv6 network in shape, or rather generally getting fringe technologies into the network. The second is taking care of a s390x machine in the basement which my faculty got sponsored recently. In my spare time I tend my Debian duties and I’m active in the student council (Fachschaft) as a sys-admin, software developer and until recently as treasurer.

I got accepted as a Debian Developer in 2005. I’m only really active since I was invited to join the Release Team in early 2008 after I contributed rewrites of some scripts that got lost in a disk crash of ftp-master. In late 2008 I also took on wanna-build/buildd duties.

What’s your notable achievement within Debian?

For one I worked on the deployment of a single buildd/sbuild combination over all of our buildds and I rebased it on the sbuild in the archive several times already. All buildds basically look the same on all architectures, with only few variations. The chroots are now also created in a mostly predictable manner.

Then we finally got build autosigning after years of constant poking. However the policy decision to allow it was made by the ftp-masters anyway, for which I’m grateful, as it eases the workload of the buildd maintainers, the Security Team and the Release Team quite a bit.

On the release side there’s the integration of volatile into the proper structures of Debian. But there I’m guilty for choosing the bad name squeeze-updates is, in comparision with security’s squeeze/updates. Let’s see if we can improve that for wheezy.

The policy for stable updates have changed over time. Can you summarize what kind of updates are allowed nowadays?

All the changes that were made to the policy were driven by the aim to keep stable (and to some lesser degree oldstable) usable throughout its lifetime.

I have to admit that we got slightly more liberal in what we accept since I took over. The previous stable update policy included the fixes of critical bugs that break the system in interesting ways and security fixes. We also opened up the possibility of including important fixes for annoying bugs on a case-by-case basis.

The whole “don’t update that much” part of the stable release management is rooted in “let’s don’t change the behaviour of a running system” and “let’s have as few regressions as possible”. We currently try to counter that with a review process that only allows self-contained fixes that were tested in unstable first, if applicable.

In the future I’d also like to have a feedback process that includes the users of the packages to report problems more early. In fact that’s also why we now send calls for testing to debian-stable-announce one week in advance (see this mail as an example).

So stable is updated through point releases that happen roughly every two months for stable and roughly every four for oldstable. They are accompanied by announcements and new CD/DVD sets.

To be able to push some updates with less delay to our users and to avoid that they have to pick them from proposed-updates, there is stable-updates. The current policy for this suite is available in this post on debian-devel-announce. It boils down to everything urgent like tzdata, regression fixes and volatile packages. We noticed that some packages in stable just weren’t useful anymore when they got updated through volatile. Hence those are folded into stable, too. We also relaxed the rules a bit so that leaf packages like clive, that rely on external websites not to change their URLs, can be updated too.

If somebody wonders what happened to the “and a half” releases… Those were intended to mark the inclusion of new hardware support (mostly by including a co-installable new kernel version). This kind of flag point release is thankfully no longer needed because our marvellous kernel team now backports certain hardware drivers to the kernel version in stable. They have some trouble soliciting test feedback for them, though. It would be cool if more people using stable could respond to their calls for testing.

What are your plans for Debian Wheezy?

Luckily the wanna-build/buildd side isn’t much dependent on release cycles. The stable release management also just takes care of what gets dumped on them by the testing RMs. 😉

The near-term goals are certainly:

getting d-i to use some development process in the archive that’s compatible with the normal flow of packages into testing and
hopefully getting wanna-build to use a sane SQL layout and hence better performance

Apart from that my work is mainly squashing the bugs on buildd.debian.org for the wanna-build part, so if you want something fixed, you need to report it, I’m afraid.

You have several roles within Debian, in particular stable release manager and member of the wanna-build team. Which role do you enjoy more and why?

So the regular duties of the SRM, apart from setting up policies, are reviewing requests for uploads to proposed-updates, accepting them, chasing builds and scheduling point releases. It’s mainly steady work. As for the Release Team it’s working in a fabulous team where everyone’s doing an awesome job.

For me the wanna-build role is holding the things together and fixing up stuff that breaks while the world continues to evolve. And occassionally taking a stab at the ftp-masters. So rather… different.

My heart is probably siding with the buildd side because it’s infrastructure but release work is fun, too.

What’s the biggest problem of Debian?

Mainly that we’re not quick in picking up new technologies and that we got slow on innovation. Distro-wide changes are usually a very big effort to coordinate and to finally complete. I guess people should be able to push fixes for release goals more eagerly into the archive.

Also it seems to me that many DDs are actually disengaging from the release process, especially during the freeze but also before, which makes me a bit sad. However I’m still not sure how we could counter that. As we’re all volunteers you can’t suddenly make others love the idea of releasing something together, I guess.

Do you have wishes for Debian Wheezy?

It’s getting old, but there’s certainly multiarch. Also a usable Python 3 is something I’d really appreciate.

Is there someone in Debian that you admire for their contributions?

Peter Palfrader did an awesome job of getting a nicely working team together for DSA and got many tasks as automated as they are now. It’s a pleasure to work with them and setting up a new buildd is, for example, a real breeze. He also developed a sane archive snapshot service, kudos for that.

Thank you to Philipp for the time spent answering my questions. I hope you enjoyed reading his answers as I did. Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Twitter and Facebook.

Do we need project-wide support for Debian rolling?

April 29, 2011 by Raphaël Hertzog

The discussion about Debian rolling started sooner than expected on debian-devel (see the thread here). I initially wanted to iron out the biggest problems through discussions on my blog and try to submit a somewhat polished proposal… instead we ended up discussing the same things both on -devel and on my blog.

That said it’s not that bad (except for the time I lost to have similar conversations in both places) because the debian-devel discussions included members of the release team and it looks like they are not fundamentally opposed to the idea.

Despite this, the introduction of a “frozen” suite branched off from “testing/rolling” is not really consensual (yet?). But the idea of officially supporting testing on a best-effort basis appears to have almost no opposition.

While some will undoubtedly believe that this is a useless exercise, I believe it would help if the project stated this in a somewhat official manner. My answer to the question in the title is thus:

YES

I am thus considering to submit a general resolution to that effect. My current draft is below.

Title: Debian endorses usage of testing by end-users, and renames it to rolling

The Debian project recognizes that the Debian testing distribution—initially created to make it easier to prepare and test the next stable release—has become a useful product of its own. It satisfies the needs of users who are looking for the latest stable versions of software and who can cope (or even appreciate) a system that’s constantly evolving.

The Debian project decides to endorse this usage and will strive to provide a good experience to users of “testing”. To better communicate this policy change to our users, “testing” will be renamed “rolling”.

While we believe that this is a good move, we would like to remind our users that Debian is a volunteer project and that our resources are not infinite. Package maintainers are contributing to Debian on a best-effort basis. This means that they might not be able to properly support their package(s) in all distributions. In that case, the project recommends that maintainers apply the following priorities:

Support in stable (security updates, release critical bugs)

Preparation of the next stable release

Support in rolling

Note that this general resolution could have amendments with s/rolling/current/ and that would solve the bikeshedding over the name that started several times already.

I deliberately separated “Preparation of the next stable release” and “Support in rolling” so that priorities are clear even if we decide to not freeze “rolling” and to introduce a “frozen” distribution to finalize the next stable release.

I also did not go into too much details on the implications that it might have, it’s best to leave that up to each contributor/team/etc.

I hesitated to add a paragraph stating that we want to try to gradually improve the usability of testing but in the end I think it would be somewhat redundant. We’re always trying to do our best when we decide to take on something.

All comments welcome (even if you just agree and would be willing to second such a GR).

Update: I will tweak the draft included in this article when I get good suggestions. Thanks to Lucas Nussbaum for the first one.

No freeze of Debian’s development, what does it entail?

April 28, 2011 by Raphaël Hertzog

The main feature of rolling is that it would never freeze. This is not without consequences.

Possible consequences

It can divert developers from working on the release

No freeze means developers are free to continue their work as usual in unstable. Will it be more difficult to release because some people will spend their time working on a new upstream version instead of fixing RC bugs in the version that is frozen? Would we lose the work of the people who do lots of NMU to help with the release?

It makes it more difficult to cherry-pick updates from unstable

No freeze also means that unstable is going to diverge sooner from testing and it will be more difficult to cherry-pick updates from unstable into testing. And the release team likes to cherry-pick updates that have been tested in unstable because updates that comes through testing-proposed-updates have often not been tested and need thus a more careful review.

Frozen earth

My responses to the objections

Those are the two major objections that we’ll have to respond to. Let’s try to analyze them a bit more.

It’s not testing vs rolling

On the first objection I would like to respond that we must not put “testing” and “rolling/unstable” in opposition. The fact that a contributor can’t do its work as usual in unstable does not mean that he will instead choose to work on fixing RC bugs in testing. Probably that some do, but in my experience we simply spend our time differently, either working more on non-Debian stuff or doing mostly hidden work that is then released in big batches at the start of the next cycle (which tends to create problems of its own).

I would also like to argue that by giving more exposure to rolling and encouraging developers to properly support their packages in rolling, it probably means that the overall state of rolling should become gradually better compared to what we’re currently used to with testing.

The objection that rolling would divert resources from getting testing in a releasable shape is difficult to prove and/or disprove. The best way to have some objective data would be to setup a questionnaire and to ask all maintainers. Any volunteer for that?

Unstable as a test-bed for RC bugfixes?

It’s true that unstable will quickly diverge from testing and that it will be more difficult to cherry-pick updates from unstable into testing. This cannot be refuted, it’s a downside given the current workflow of the release team.

But I wonder if the importance of this workflow is not overdone. The reason why they like to cherry-pick from unstable is because it gives them some confidence that the update has not caused other regressions and ensures that testing is improving.

But if they’re considering to cherry-pick an update, it’s because the current package in testing is plagued by an RC bug. Supposing that the updated package has introduced a regression, is it really better to keep the current RC bug compared to trading it for a new regression? It sure depends on the precise bugs involved so that’s why they prefer to know up-front about the regression instead of making a blind bet.

Given this, I think we should use testing-proposed-updates (tpu) as a test-bed for RC bug fixes. We should ask beta-testers to activate this repository and to file RC bugs for any regression. And instead of requiring a full review by a release manager for all uploads to testing-proposed-updates, uploads should be auto-accepted provided that they do not change the upstream version and that they do not add/remove binary packages. Other uploads would still need manual approval by the release managers.

On top of this, we can also add an infrastructure to encourage peer-reviews of t-p-u uploads so that reviews become more opportunistic instead of systematic. Positive reviews would help reduce the aging required in t-p-u before being accepted into testing.

This changes the balance by giving a bit more freedom to maintainers but still keeps the safety net that release managers need to have. It should also reduce the overall amount of work that the release team has to do.

Comments welcome

Do you see other important objections beside the two that I mentioned?

Do you have other ideas to overcome those objections?

What do you think of my responses? Does your experience infirm or confirm my point of view?