CUT – Page 2

People behind Debian: Adam D. Barratt, release manager

April 7, 2011 by Raphaël Hertzog

Adam D. Barratt is a Debian developer since 2008, in just a few years he got heavily involved to the point of being now “Release manager”, a high responsibility role within the community. He worked hard with the other members of the release team to make Squeeze happen.

You could expect the release managers to have some rest after a big release, but it’s not really the case. With the long freeze, loads of “transitions” have accumulated and they are now busy to get all those updated packages in the new testing (wheezy). Despite this Adam took some time to answer my questions.

He shares with us his impression on the Squeeze release, his opinion on time-based freezes (regular/predictable freeze) and much more. Read on. My questions are in bold, the rest is by Adam.

Who are you?

I’m a 31 year old software developer and part-time sysadmin for a software and IT services company based in the south of England. I have no children, no pets and a long-suffering partner who puts up with me spending far too much time tinkering with things and people making fun of her Macbook during Debconf.

As well as being on the release team, I’m a member of the maintainer teams for devscripts and lintian.

Can you describe your journey in Debian and in the release team?

I was introduced to Debian as part of an infrastructure upgrade at work, moving from a set of Red Hat and Solaris-based systems. As part of that, we submitted some bugs for issues we found during the upgrade and for small patches we included in some software to add extra functionality we wanted. From that starting point I became more interested in Debian in general and began following some of the mailing lists and IRC channels.

When Julian Gilbey asked for help with the maintenance of devscripts, I submitted some patches for some of the outstanding bug reports and was invited to join the team which was being created to handle maintenance for the package. One of the then Release Managers was also on the team and asked if I’d be interested in working on a couple of updates they wanted to the scripts which generate the proposed-updates overview pages. I added the new functionality which was merged in to the live scripts and a little while later I was invited to join the team, shortly before Debconf 9.

As most readers will be aware, we unfortunately reached a point during last year where we didn’t have anyone filling the Release Manager role. During that period, I became more active in handling transitions and requests for updates to stable and as time went on more people started to suggest that I should put myself forward for the position, or refer to me as already being RM. I procrastinated over the decision for some time but after discussions during Debconf 10 I came round to the idea that we should have the RM role filled again and agreed to take it on, together with Neil. The rest, as they say…

How much time do you usually spend working for the release team ?

I’ve been trying to work out how to usefully answer this question. My initial answer was “approximately two hours each day”, but the longer I thought about it the more I started debating exactly what I should include under the umbrella of release work; after some to-and-fro I’ve decided to stick with my initial answer.

During periods when Debian is frozen and particularly in the lead up to the release that time commitment increases significantly, particularly over weekends. I’m reliably informed that at that point the correct answer to the question is “too much time”. 🙂

What’s your own retrospective of the Squeeze release? What went well and what needs to be improved?

Overall, I believe the release went well and that we should all be proud of the Squeeze release. The parts of the release cycle which highlighted the need for improvement all share, imo, a single root cause – communication, particularly around freeze-related plans. We worked hard during the freeze itself to improve our communication with the rest of the project and want to continue in that vein during the Wheezy cycle.

One thing that I personally found quite difficult at times before the freeze was keeping track of the transitions which were still waiting for a place in the queue; it’s also something that we could improve on at this early stage of the Wheezy cycle. In order to help us keep a clear overview of requests for transitions, stable updates and binNMUs, it would be helpful if they could be filed as appropriately user-tagged bugs. This not only allows us to easily get an overview of the status of requests from the BTS but also aids transparency by allowing anyone else to do so; as a useful additional feature, it means that we can use the BTS’s blocking functionality to indicate reasons why a request cannot be fulfilled right now.

Are you in favor of time based freeze?

I think there’s merit in having a time frame that we can work towards in order to achieve the goals which we set ourselves for the release, as individual maintainers, maintenance teams and a project. I do have concerns that even with such a time frame in place there will still be uploads made very close to the proposed freeze point and transitions which may be unfinished, for example because of an unforeseen entanglement with or reliance on the transition of another package.

One thing I’m interested in is how exact and specific that time frame should be and the balance between predictability and being able to achieve everything we want for a great release; this is something we can cover in the debate on this subject which I know many people have strong opinions about.

What are your plans for Debian Wheezy?

The Wheezy to-do list I started before the final Squeeze release begins “multiarch, multiarch, multiarch”. It looks like we’re finally going to get that achieved during this release cycle, thanks to a great deal of hard work from various people. I’m also interested in seeing the C.UTF-8 locale standardised throughout Debian and continuing to work on our tools and processes to make tracking of transitions and stable updates simpler (or at least appearing to be so 🙂 and more transparent.

With my package maintenance hats on, I’d like to help ensure that both devscripts and lintian are able to keep pace with changes in the development landscape in Debian (e.g. more useful package diffing for source format v3 packages) and continue to be tools that are an integral part of package development in Debian.

Some people (including me) would like a rolling distribution constantly usable by end-users. Do you think that the release process currently geared towards producing “stable” can be accommodated to support this?

I’m not yet convinced that the concept of a rolling, “constantly usable” distribution can be easily integrated in to the workflow that exists around preparing stable releases in Debian. The “testing” distribution was created as, and continues to be used as, a tool to enable the release team to create the next stable release – that it happens to be something that people can use every day for much of the time is mostly a happy side-effect of the fact that we don’t gratuitously break it, but is by no means guaranteed to be the case early in the release cycle or during large, disruptive, transitions.

It’s been suggested that “testing” and “rolling” could be basically the same for most of the cycle, with “rolling” then continuing to be updated when testing is frozen. This would essentially mean an extra suite which is only used for a few months every couple of years or so, which is one of the things that “testing” was intended to avoid (i.e. the old “frozen” suite) and seems like a lot of overhead to introduce in order to reduce disruption to some users during the freeze. The early part of the release cycle also tends to include a number of larger transitions which often require packages to either be removed from testing or broken as part of migrating the transition, if they are not able to be successfully updated in time.

What’s the biggest problem of Debian?

The thing that I’ve been noticing myself becoming frustrated by recently is a tendency to debate the minor details of proposals, rather than concentrating on getting the key points right to begin with. Clearly for some projects such as multiarch the details may be as important as the big picture, but in most cases the people working on a development should be allowed to look after the smaller details themselves.

That’s not meant to imply that feedback from other parts of the project should not be welcomed, simply that if we consider Debian to be a “do-ocracy” then we need to permit people the freedom to “do”.

Is there someone in Debian that you admire for their contributions?

All previous release managers, for making the job look much easier than it seems when you’re in the “hot seat”. 🙂

Outside of the release team, Joey Hess for his contributions to various parts of the Debian development environment over the years, such as debhelper and debian-installer, and Colin Watson for his enviable willingness to tackle a wide variety of different projects within Debian.

Thank you to Adam for the time spent answering my questions. I hope you enjoyed reading his answers as I did. Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Twitter and Facebook.

People behind Debian: Mehdi Dogguy, release assistant

December 23, 2010 by Raphaël Hertzog

Mehdi is a Debian developer for a bit more than a year, and he’s already part of the Debian Release Team. His story is quite typical in that he started there by trying to help while observing the team do its work. That’s a recurrent pattern for people who get co-opted in free software teams.

Read on for more info about the release team, and Mehdi’s opinion on many topics. My questions are in bold, the rest is by Mehdi (except for the additional information that I inserted in italics).

Who are you?

I’m 27 years old. I grew up in Ariana in northern Tunisia, but have been living in Paris, France, since 2002.

I’m a PhD Student at the PPS laboratory where I study synchronous concurrent process calculi.

I became interested in Debian when I saw one of my colleagues, Samuel Mimram (first sponsor and advocate) trying to resolve #440469, which is a bug reported against a program I wrote. We have never been able to resolve it but my intent to contribute was born there. Since then, I started to maintain some packages and help where I can.

What’s your biggest achievement within Debian?

I don’t think I had time to accomplish a lot yet 🙂 I’ve been mostly active in the OCaml team where we designed a tool to compute automatically the dependencies between OCaml packages, called dh-ocaml. This was a joint work with Stéphane Glondu, Sylvain Le Gall and Stefano Zacchiroli. I really appreciated the time spent with them while developing dh-ocaml. Some of the bits included in dh-ocaml have been included upstream in their latest release.

I’ve also tried to give a second life to the Buildd Status Pages because they were (kind of) abandoned. I intend to keep them alive and add new features to them.

If you had a wand and could change one thing in Debian, what would that be?

Make OCaml part of a default Debian installation 😀

But, since I’m not a magician yet, I’d stick to more realistic plans:

A lot of desktop users fear Debian. I think that the Desktop installation offered by Debian today is very user-friendly and we should be able to attract more and more desktop users. Still, there is some work to be done in various places to make it even more attractive. The idea is trying to enhance the usability and integration of various tools together. Each fix could be easy or trivial but the final result would be an improved Desktop experience for our users. Our packaged software run well. So, each person can participate since the most difficult part is to find the broken scenarios. Fixes could be found together with maintainers, upstream or other interested people.
I’ll try to come up with a plan, a list of things that need polishing or fixes and gather a group of people to work on it. I’d definitely be interested in participating in such a project and I hope that I’ll find other people to help. If the plan is clear enough and has well described objectives and criteria, it could be proposed to the Release Team to consider it as a Release Goal for Wheezy.
NMUs are a great way to make things move forward. But, sometimes, an NMU could break things or have some undesirable effects. For now, NMUers have to manually track the package’s status for some time to be sure that everything is alright. It could be a good idea to be auto-subscribed to the bugs notifications of NMUed packages for some period of time (let’s say for a month) to be aware of any new issues and try to fix them. NMUing a package is not just applying a patch and hitting enter after dput. It’s also about making sure that the changes are correct and that no regressions have been introduced, etc…
Orphaned packages: It could be considered as too strict and not desired, but what about not keeping orphaned and buggy packages in Testing? What about removing them from the archive if they are buggy and still unmaintained for some period? Our ftp archive is growing. It could make sense to do some (more strict) housekeeping. I believe that this question can be raised during the next QA meeting. We should think about what we want to do with those packages before they rot in the archive.

[Raphael Hertzog: I would like to point out that pts-subscribe provided by devscripts makes it easy to temporarily subscribe to bug notifications after an Non-Maintainer Upload (NMU).]

You’re a Debian developer since August 2009 and you’re already an assistant within the Release Management team. How did that happen and what is this about?

In the OCaml team, we have to start a transition each time we upload a new version of the OCaml compiler (actually, for each package). So, some coordination with the Release Team is needed to make the transition happen.

When we are ready to upload a new version of the compiler, we ask the Release Team for permission and wait for their ack. Sometimes, their reply is fast (e.g. if their is no conflicting transition running), but it’s not always the case. While waiting for an ack, I used to check what was happening on debian-release@l.d.o. It made me more and more interested in the activities of the Release Team.

Then (before getting my Debian account), I had the chance to participate in DebConf9 where I met Luk and Phil. It was a good occasion to see more about the tools used by the Release Team. During April 2010, I had some spare time and was able to implement a little tool called Jamie to inspect the relations between transitions. It helps us to quickly see which transitions can run in parallel, or what should wait. And one day (in May 2010, IIRC), I got offered by Adam to join the team.

As members of the Release Team, we have multiple areas to work on:

Taking care of transitions during the development cycle, which means making sure that some set of packages are correctly (re-)built or fixed against a specific (to each transition) set of packages, and finding a way to tell Britney that those packages can migrate and it would be great if she also shared the same opinion. [Raphael Hertzog: britney is the name of the software that controls the content of the Testing distribution.]
Paying attention to what is happening in the archive (uploads, reported RC bugs, etc…). The idea is to try to detect unexpected transitions, blocked packages, make sure that RC bug fixes reach Testing in a reasonable period of time, etc…
During a freeze, making sure that unblock requests and freeze exceptions are not forgotten and try to make the RC bug count decrease.

There are other tasks that I’ll let you discover by joining the game.

Deciding what goes (or not) in the next stable release is a big responsibility and can be incredibly difficult at times. You have to make judgement calls all the time. What are your own criteria?

That’s a very hard to answer question (at least, for me). It really depends on the “case”. I try to follow the criteria that we publish in each release update. Sometimes, an unblock request doesn’t match those criteria and we have to decide what to accept from the set of proposed changes. Generally, new features and non-fixes (read new upstream versions) changes are not the kind of changes that we would accept during the freeze. Some of them could be accepted if they are not intrusive, easy and well defended. When, I’m not sure I try to ask other members of the Release Team to see if they share my opinion or if I missed something important during the review. The key point is to have a clear idea on what’s the benefit of the proposed update, and compare it to the current situation. For example, accepting a new upstream release (even if it fixes some critical bugs) is taking a risk to break other features and that’s why we (usually) ask for a backported fix.

It’s also worth noticing that (most of the time) we don’t decide what goes in, but (more specifically) what version of a given package goes in and try to give to the contributors an idea on what kind of changes are acceptable during the freeze. There are some exceptions though. Most of them are to fix a critical package or feature.

Do you have plans to improve the release process for Debian Wheezy?

We do have plans to improve every bit in Debian. Wheezy will be the best release ever. We just don’t know the details yet 🙂

During our last meeting in Paris last October, the Release Team agreed to organize a meeting after Squeeze’s release to discuss (among other questions) Wheezy’s cycle. But the details of the meeting are not fixed yet (we still have plenty of time to organize it… and other more important tasks to care about). We would like to be able to announce a clear roadmap for Wheezy and enhance our communication with the rest of the project. We certainly want to avoid what happened for Squeeze. Making things a bit more predictable for developers is one of our goals.

Do you think the Constantly Usable Testing project will help?

The original idea by Joey Hess is great because it allows d-i developers to work with a “stable” version of the archive. It allows them to focus on the new features they want to implement or the parts they want to fix (AIUI). It also allows to have constantly available and working installation images.

Then, there is the idea of having a constantly usable Testing for users. The idea seems nice. People tend to like the idea behind CUT because they miss some software disappearing from Testing and because of the long delays for security fixes to reach Testing.

If the Release Team has decided to remove a package from Testing, I think that there must be a reason for that. It either means that the software is broken, has unfixed security holes or was asked for the removal by its maintainer. I think that we should better try to spend some time to fix those packages, instead of throwing a broken version in a new suite. It could be argued that one could add experimental’s version in CUT (or sid’s) but every user is free to cherry-pick packages from the relevant suite when needed while still following Testing as a default branch.

Besides, it’s quite easy to see what was removed recently by checking the archive of debian-testing-changes or by querying UDD. IMO, It would be more useful to provide a better interface of that archive for our users. We could even imagine a program that alerts the user about installed software that got recently removed from Testing, to keep the user constantly aware any issue that could affect his machine. About the security or important updates, one has to recall the existence of Testing-security and testing-proposed-updates that are used specifically to let fixes reach Testing as soon as possible when it’s not possible to go through Unstable. I’m sure that the security team would appreciate some help to deal with security updates for Testing. We also have ways to speed migrate packages from Unstable to Testing.

I have to admit that I’m not convinced yet by the benefits brought by CUT for our users.

Thank you to Mehdi for the time spent answering my questions. I hope you enjoyed reading his answers as I did. Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Twitter and Facebook.

Can Debian offer a Constantly Usable Testing distribution?

October 4, 2010 by Raphaël Hertzog

Debian’s “testing” distribution is where Debian developers prepare the next stable distribution. While this is still its main purpose, many users have adopted this version of Debian because it offers them a good trade-off between stability and freshness. But there are downsides to using this distribution and the “Constantly Usable Testing” (CUT) project aims to resolve those. This article will present the project and the challenges involved to make it happen.

About Debian unstable & testing

Debian unstable is the distribution where developers upload new versions of their packages. It happens frequently that some packages are not installable due to changes in other packages or due to transitions not yet completed.

Debian testing, on the contrary, is managed by a tool that ensures the consistency of the whole distribution: it picks updates from unstable only if the package has been enough tested (10 days usually), if it’s free of new release-critical bugs, if it’s available on all supported architectures, and if it doesn’t break any other package already present in testing. The Release Team (RT) controls this tool and provide “hints” to help it find a set of packages that can flow from unstable to testing.

Those rules also ensure that the packages that flow into testing are reasonably free of show-stopper bugs (like a system that doesn’t boot, or X that doesn’t work at all). This makes it very attractive to users who like to regularly get new upstream versions of their software without dealing with the biggest problems associated to them. This is all very attractive, yet several Debian developers advise people to not use testing. Why is that?

Known problems with testing

Disappearing software

The release team use this distribution to prepare the next stable release and from time to time they remove packages from it. Either because it’s needed to ensure that other packages can migrate from unstable to testing, or because they have long-standing release-critical bugs without progress towards a resolution. It also happens that they remove packages on request of the maintainers because they believe that the current version of the software cannot be supported (security-wise) for 2 years or more. The security team also regularly issues such requests.

Long delays for security and important fixes

Despite the 10-day delay in unstable, there are always some annoying bugs (and security bugs are no exceptions) that are only discovered when the package already has migrated to testing. The maintainer might be quick to upload a fixed package in unstable, and might even raise the urgency to allow the package to migrate sooner, but if the packages gets entangled in a large ongoing transition, it will not migrate before the transition is completed. Sometimes it can take weeks for that to happen.

This delay can be avoided by doing direct uploads to testing (through testing-proposed-updates) but this is almost never used, except during a freeze, where targeted bugfixes are the norm.

Not always installable

With testing evolving daily, updates sometimes break the last installation images available (in particular netboot images that get everything from the network). The debian-installer (d-i) packages are usually quickly fixed but they don’t move to testing automatically because the new combination of d-i packages has not necessarily been validated yet. Colin Watson sums up the problem:

Getting new installer code into testing takes too long, and problems remain unfixed in testing for too long. […] The problem with d-i development at the moment is more that we’re very slow at producing new d-i *releases*. […] Your choices right now are to work with stable (too old), testing (would be nice except for the way sometimes it breaks and then it tends to take a week to fix anything), unstable (breaks all the time).

CUT’s history

CUT finds its root in an old proposal by Joey Hess: it introduces the idea that the stable release is not Debian’s sole product and that testing could become — with some work — a suitable choice for end-users. Nobody took on that work and there was no visible progress in the last 3 years.

But recently Joey brought up CUT again on the debian-devel mailing list and Stefano Zacchiroli (the Debian project leader) challenged him to setup a BoF on CUT for Debconf10. It turned out to be one of the most heavily attended BoF (video recording is here), there is clearly a lot of interest in the topic.

There’s now a dedicated wiki and an Alioth project with a mailing list. The rest of this article tries to summarize the various options discussed and how they’re supposed to address the problems identified.

The ideas behind CUT

Among all the ideas, there are two main approaches that have been discussed. The first is to regularly snapshot testing at points where it is known to work reasonably well (those snapshots would be named “cut”). The second is to build an improved testing distribution tailored to the needs of users who want a working distribution with daily updates, its name would be “rolling”.

Regular snapshots of testing

There’s general agreement that regular snapshots of testing are required: it’s the only way to ensure that the generated installation media will continue to work until the next snapshot. If tests of the snapshot do not reveal any major problem, then it becomes the latest “cut”. For clarity, the official codename would be date based: e.g. “cut-2010-09” would be the cut taken during September 2010.

While the frequency has not been fixed yet, the goal is clearly to be on the aggressive side: at the very least every 6 months, but every month has been suggested as well. In order to reach a decision, many aspects have to be balanced.

One of them (and possibly the most important) is the security support. Given that the security team is already overworked, it’s difficult to put more work on their shoulders by declaring that cuts will be supported like any stable release. No official security support sounds bad but it’s not necessarily so problematic as one might imagine. Testing’s security record is generally better than stable’s one (see the security tracker) because fixes flow in naturally with new upstream versions. Stable still get fixes for very important security issues sooner than testing, but on the whole there are less known security-related problems in testing than in stable.

Since it’s only a question of time until the fixed version comes naturally from upstream, more frequent cut releases means that users get security fixes sooner. But Stefan Fritsch, who used to be involved in the Debian testing security team, has also experienced the downside for anyone who tries to contribute security updates:

The updates to testing-security usually stay useful only for a few weeks, until a fixed version migrates from unstable. In stable, the updates stay around for a few years, which gives a higher motivation to spend time on preparing them.

So if it’s difficult to form a dedicated security team, the work of providing security updates comes back to the package maintainer. They are usually quite quick to upload fixed packages in unstable but tend to not monitor whether the packages migrate to testing. They can’t be blamed because testing was created to prepare the next stable release and there is thus no urgency to get the fix in as long as it makes it before the release.

CUT can help in this regard precisely because it changes this assumption: there will be users of the testing packages and they deserve to get security fixes much like the stable users.

Another aspect to consider when picking a release frequency is the amount of associated work that comes with any official release: testing upgrades from the previous version, writing release notes and preparing installation images. It seems difficult to do this every month. With this frequency it’s also impossible to have a new major kernel release for each cut (since they tend to come out only every 2 to 3 months) and the new hardware support that it brings is something worthwhile to many users.

In summary, regular snapshots address the “not always installable” problem and changes the perception of maintainers towards testing, so that hopefully they care more of security updates in that distribution (and in cuts). But they do not solve the problem of disappearing packages. Something else is needed to fix that problem.

A new “rolling” distribution?

Lucas Nussbaum pointed out that regular snapshots of Debian is not really a new concept:

How would this differentiate from other distributions doing 6-month release cycles, and in particular Ubuntu, which can already be seen as Debian snapshots (+ added value)?

In Lucas’s eyes, CUT becomes interesting if it can provide a rolling distribution (like testing) with a “constant flux of new upstream releases”. For him, that would be “something quite unique in the Free Software world”. The snapshots would be used as starting point for the initial installation, but the installed system would point to the rolling distribution and users would then upgrade as often as they want. In this scenario, security support for the snapshots is not so important, what matters is the state of the rolling distribution.

If testing were used as the rolling distribution, the problem of “disappearing packages” would not be fixed. That’s why there have been discussions of introducing a new distribution named “rolling” that would work like testing but with adapted rules, and the cuts would then be snapshots of rolling instead of testing.

The basic proposal is to make a copy of testing and to re-add the packages which have been removed because they are not suited for a long term release while they are perfectly acceptable for a constantly updated release (the most recent example being Chromium).

Then it’s possible to go one step further: during freeze, testing is no longer automatically updated which makes it inappropriate to feed the rolling distribution. That’s why rolling would be reconfigured to grab updates from unstable (but using the same rules than testing).

Given the frequent releases, it’s likely that only a subset of architectures would be officially supported. This is not a real problem because the users who want bleeding edge software tends to be desktop users on mainly i386/amd64 (and maybe armel for tablets and similar mobile products). This choice — if made — opens up the door to even more possibilities: if rolling is configured exactly like testing but with only a subset of the architectures, it’s likely that some packages migrate to rolling before testing when non-mainstream architectures are lagging in terms of auto-building (or have toolchain problems).

While being ahead of testing can be positive for the users, it’s also problematic on several levels. First, managing rolling becomes much more complicated because the transition management work done by the release team can’t be reused as-is. Then it introduces competition between both distributions which can make it more difficult to get a stable release out, for example if maintainers stop caring of the migration to testing once the migration to rolling has been completed.

The rolling distribution is certainly a good idea but the rules governing it must be designed to avoid any conflict with the process of releasing a stable distribution. Lastly, the mere existence of rolling would finally fix the marketing problem plaguing testing: the name “rolling” does not suggest that the software is not yet ready for prime time.

Conclusion

Whether CUT will be implemented remains to be seen, but it’s off for a good start: ftpmaster Joerg Jaspert said that the new archive server can cope with a new distribution, and there’s now a proposal shaping up. The project might start quickly: there is already an implementation plan for the snapshot side of the project. The rolling distribution can always be introduced later, once it is ready. Both approaches can complement each other and provide something useful to different kind of users.

The global proposal is certainly appealing: it would address the concerns of obsolescence of Debian’s stable release by making intermediary releases. Anyone needing something more recent for hardware support can start by installing a cut and follow the subsequent releases until the next stable version. And users who always want the latest version of every software could use rolling after having installed a cut.

From a user point of view, there are similarities with the mix of usual and long term releases of Ubuntu. But from the development side, the process followed would be quite different, and the constraints imposed by having a constantly usable distribution are stronger: any wide-scale change must be designed in a way that it can happen progressively in a transparent manner for the user.

This article was first published in Linux Weekly News. If you want to see more articles like this, join Flattr and click on the flattr button below every article that you like.