Debian – Page 68

Best practices when sponsoring Debian packages

February 10, 2011 by Raphaël Hertzog

Sponsoring a package means uploading a package for someone else (usually a new contributor starting out as package maintainer). This is an activity reserved to Debian Developer who are supposed to be knowledgeable about packaging. This article tries to document the process to ensure the sponsor is doing a reasonably good job according to Debian’s standards.

Sponsoring a package in the Debian archive is not a trivial matter. It means that you verified the packaging and that it is of the level of quality that Debian strives to have. Let’s have a look to what you can and should do when you’re sponsoring a package.

Sponsoring the initial upload

Sponsoring a brand new package into Debian requires a thorough review of the Debian packaging. Building the package and testing the software is definitely not enough! You should open every file in the debian directory and look out for potential problems. Here’s a checklist that you can use to perform the audit:

Verify that the upstream tarball provided is the same that has been distributed by the upstream author (when the sources are repackaged for Debian, generate the modified tarball yourself).
Run lintian. It will catch many common problems. Be sure to verify that any lintian overrides setup by the maintainer is fully justified.
Run licensecheck and verify that debian/copyright seems correct and complete. Look for license problems (like files with “All rights reserved” headers, or with a non-DFSG compliant license).
Build the package with pbuilder (or any similar tool) to ensure that the build-dependencies are complete.
Proofread debian/control: does it follow the best practices? are the dependencies complete?
Proofread debian/rules: does it follow the best practices? do you see some possible improvements?
Proofread the maintainer scripts (preinst, postinst, prerm, postrm, config): will the preinst/postrm work when the dependencies are not installed? are all the scripts idempotent (i.e. can you run them multiple times without consequences)?
Review any change to upstream files (either in .diff.gz, or in debian/patches/ or directly embedded in the debian tarball for binary files). Are they justified? Are they properly documented (with DEP-3 for patches)?
For every file, ask yourself why the file is there and whether it’s the right way to achieve the desired result. Is the maintainer following the best packaging practices described by the Developers Reference?
Build and install the packages, try the software. Ensure you can remove and purge the packages. Maybe test the packages with piuparts.

If the audit did not reveal any problem, you can upload the package. But remember that even if you’re not the maintainer, the sponsor is still responsible of what he uploaded to Debian. That’s why you’re encouraged to keep up with the package through the Package Tracking System.

Sponsoring an update of an existing package

You will usually assume that the package has already gone through a full review. So instead of doing it again, you will carefully analyze the difference between the current version and the new version prepared by the maintainer. If you have not done the initial review yourself, you might still want to have a more deeper look just in case the initial reviewer was sloppy.

To be able to analyze the difference you need both versions. Download the current version of the source package (with apt-get source) and rebuild it (or download the current binary packages with aptitude download). Download the source package to sponsor (usually with dget).

Read the new changelog entry, it should tell you what to expect during the review. The main tool you will use is debdiff, you can run it with two source packages (.dsc files), or two binary packages, or two .changes files (it will then compare all the binary packages listed in the .changes).

If you compare the source packages (excluding upstream files in the case of a new upstream version, for example by filtering the output of debdiff with filterdiff -i '*/debian/*'), you must understand all the changes you see and they should be properly documented in the Debian changelog.

If everything is fine, build the package and compare the binary packages to verify that the changes on the source package have no unexpected consequences (like some files dropped by mistake, missing dependencies, etc.).

You might want to check out the Package Tracking System to verify if the maintainer has not missed something important. Maybe there are translations updates sitting in the BTS that could have been integrated. Maybe the package has been NMUed and the maintainer forgot to integrate the changes from the NMU in his package. Maybe there’s a release critical bug that he has left unhandled and that’s blocking migration to testing. Whatever. If you find something that she could have done (better), it’s time to tell her so that she can improve for next time. And so that she has a better understanding of her responsibilities.

If you have found no problem, upload the new version. Otherwise ask the maintainer to provide you a fixed version.

This article will be repurposed to enhance the Debian Developers Reference, hopefully leading to a fix for the wishlist bug #453313. Click here and help me fix more of those.

You’re also welcome to suggest improvements in the comments. Are there other checks that you’re always doing? Do you have some handy tip to make it easier to review a package?

Debian Cleanup Tip #2: Get rid of obsolete packages

February 7, 2011 by Raphaël Hertzog

Last week, we learned to remove useless configuration files. This week, we’re going to take care of obsolete packages.

An obsolete package is a package who is no longer provided by any of the APT repositories listed in /etc/apt/source.lists (and /etc/apt/sources.list.d/). There can be multiple reasons why a package is no longer available in the repository (or at least not under the same name) :

the upstream author stopped maintaining the software a long time ago, nobody else took over and the Debian maintainer preferred to remove the package from Debian. Usually there are alternatives in the Debian archive.
the package was orphaned in Debian since a long time, nobody took over and it had very few users. The Debian QA team might have asked its removal.
the latest version of the software might have been packaged under a new package name. Either because the amount of changes was so important that it was preferred to not upgrade automatically to the latest version (it has been the case with request-tracker and nagios, they both embed a version number in their package names), or simply because the maintainer wants to let the user install several versions at the same time (that’s the case for example with the Linux kernel, the python interpreter and many libraries).
the software has been renamed, the maintainer renamed the packages and kept transitional packages under the old name for one release. Then the transitional packages have been removed.

In any case, it’s never a good idea to keep obsolete packages around: they do not benefit from security updates and they might cause problems during upgrades if they depend on other packages that should be removed to complete the upgrade.

You could blindly remove them with aptitude purge ~o (or aptitude purge ?obsolete) but you might want to first verify what those package are. There might be some packages that you have manually installed, that are not part of any current APT repository, and that you want to keep around nevertheless (I have skype, dropbox and a few personal packages for example). You can get the list with aptitude search ?obsolete…

With the graphical package manager (Synaptic), you can find the list of obsolete packages by clicking on the “Status” button and selecting “Installed (local or obsolete)”. You can then go through the list and decide for each package whether you want to keep it or not.

Follow me on Identi.ca, Twitter and Facebook. Or subscribe to this blog by RSS or by email.

Debian 6.0 is out, Wheezy kicks off

February 6, 2011 by Raphaël Hertzog

As you probably already know, Debian released Squeeze aka Debian 6.0 this week-end. It was a really great week-end.

I saw quite a few release already, but none with so much online social activity. Alexander and Meike Schmehl live commented the release on the Debian identi.ca account and Joey Hess held his Debian Party Line.

On top of this, the team in charge of the website rolled out a new design on quite a few online services during the week-end including www.debian.org, wiki.debian.org, planet.debian.org and more.

Congratulations to all the people who made this happen (and the release team in particular). It’s great to see Debian achieve all of this.

Do you know that it’s the third time in a row that we manage to release in the 18-24 months timeframe? 3.1: June 2005 → 4.0: April 2007 → 5.0: February 2009 → 6.0: February 2011.

Yes, despite our size and the fact that we are all volunteers, we have managed to stick to a reasonable schedule for a stable distribution that is deployed on a large scale.

Wheezy kicks off

The best part of the release for us — the developers — is that wheezy is now open for development and we can work on new features for the next release. 😉

And it started quickly: according to UDD, wheezy already features 488 new source packages that are not in squeeze, 1713 updated source packages and among those 1246 are new upstream versions.

I really look forward to the upcoming projects and related discussions.

Click here to subscribe to my free newsletter and get my monthly analysis on what’s going on in Debian and Ubuntu. Or just follow along via the RSS feed, Identi.ca, Twitter or Facebook.

For the curious, here are the UDD queries I used:

# Updated packages in wheezy
select count(source) from sources_uniq as su where (select version from sources_uniq where release='squeeze' and distribution='debian' and source = su.source) < su.version and release='wheezy' and distribution='debian';
# Updated with a new upstream version
select count(source) from sources_uniq as su where debversion(regexp_replace((select version from sources_uniq where release='squeeze' and distribution='debian' and source = su.source), '-.*', '')) < debversion(regexp_replace(su.version, '-.*', '')) and release='wheezy' and distribution='debian';
# New package in wheezy
select count(source) from sources_uniq as su where source not in (select source from sources_uniq where release='squeeze' and distribution='debian' and source = su.source) and release='wheezy' and distribution='debian';

People behind Debian: Mike Hommey, Firefox/Iceweasel maintainer

February 3, 2011 by Raphaël Hertzog

For as long as I have known Mike, he’s been maintaining one of the largest (and most widely used) package: Iceweasel, Debian’s default web browser. It’s effectively Mozilla’s Firefox although it has been renamed to avoid some rules enforced by Mozilla’s trademark policy. But we might see Firefox back in Debian… read on to know Mike’s plans.

His story is also very interesting in that he went from simple user to Iceweasel maintainer, and now he’s working for Mozilla Corporation. But you already knew that contributing to free software is a good way to grow skills and gain experience that are valuable on the job market, right? 😉

My questions are in bold, the rest is by Mike.

Who are you?

I am Mike Hommey. I am French, even if my name doesn’t sound so.

I have been a GNU/Linux user since 1995, using Debian since 1999, and started packaging in 2003. After a long time in the NM queue, I became a DD in 2005.

The main reason I started being actively involved is quite selfish: I wanted some things done, and I just did them. My first packages were extensions for the Mozilla Suite, the old one that directly inherited from Netscape Communicator. Later on, I wanted some of these extensions to work with the new kid on the block: Phoenix; or maybe it was already Firebird. By then, integrating extensions was a hassle, and the mozilla-browser maintainer had come up with a script that would make it easier for extension packages. So I ported that script for use with Phoenix/Firebird and sent it to the BTS.

Then Phoenix/Firebird became Firefox, and it grew a real extension manager. Except that is was entirely not suited for extensions installed by the packaging system. So I made it work better. And made a few more changes to make Firefox work better. All through patches or NMUs.

Then I became co-maintainer, then Firefox couldn’t keep its name and became Iceweasel, and, fast-forward until today, I’m now taking care of Iceweasel/Xulrunner and Iceape in stable, unstable, experimental and an external repository.

This 8 years-long story (wow, I hadn’t even realized it had been that long until I wrote it) tells you something: newcomers shouldn’t be afraid to stick their hands anywhere, even in big packages. I was only a Mozilla and Phoenix end-user when I started.

Apart from Mozilla-related packaging, I also happen to (co-)maintain some other packages in Debian, most notably webkit, though I haven’t been very active in the pkg-webkit team recently. There are several reasons for that, the main one being that there are other team members getting the job done. And I can’t say that much about the pkg-mozilla team, since there are effectively no other team members.

What’s your biggest achievement within Debian?

I am quite proud that Squeeze will be the first Debian release where Iceweasel works on all supported architectures. Previously, we only knew the package compiled on all these architectures. Starting with Squeeze, we know it passes several of the test suites coming with the source code. This ensures most of the browser is functional. It was actually far from being the case before, as it appeared when the test suites were first enabled.

I think it is very important to have properly working packages on all our supported architectures. The current evolution of hardware shows how important it still is. Who would have thought a few years back that we would see ARM or MIPS based netbooks today ?

What are your plans for Debian Wheezy?

For Debian Wheezy, I would like to improve Iceweasel/Icedove/Iceape integration with the packaging system. Something that would allow to use the apt database to find extensions or plugins, on top of the current behaviour. I also want to improve desktop integration, most notably for KDE.

Another thing I’d like to see happen is to package google breakpad independently (its source code is currently embedded in chromium and iceweasel, though not built, as far as I know), and have our Mozilla products builds be able to send their crash reports upstream. It is very important that upstream gets direct visibility over what crashes on GNU/Linux systems. It helps making them aware of issues that would otherwise probably remain unnoticed, as there are much more users using distributions packages than upstream tarballs. And not all Debian users report crashes to the BTS. I know at least Fedora and SuSE are already doing it.

More importantly, I want to avoid the sad situation we got with Squeeze, where we’re going to release a 18 months old Iceweasel. We got there for a combination of reasons, one being that I was preparing for a freeze in December 2009 and focusing on getting 3.5 in shape instead of the then upcoming 3.6.

The result was that we didn’t get 3.6 packages before upstream released 3.6.3 in April 2010. By then it was too late, considering the uncertainty of the release schedule, to update reverse dependencies and get 3.6 ready for Squeeze.

Things are different for 4.0. I’m already getting ahead of things, and while reverse dependencies haven’t been taken care of, all beta releases have been made available through http://mozilla.debian.net/, usually within hours after upstream release (sometimes even a few hours before official announcement, but always after availability on ftp.mozilla.org). A great share of our patches were also pushed upstream.

Several things made that possible:

I’m now closely following upstream trunk
I’m starting to prepare packages when candidate builds are distributed upstream, which usually happens a few days ahead for betas, and more than a week ahead for stable releases.
I automated part of the package building process, and have a beefy build machine. I’m getting near being able to provide nightly builds of upstream trunk.

There will be a couple major upstream releases until Wheezy comes along, and I want to make them available early in unstable, even if that means breaking reverse dependencies until they are fixed.

Finally, I would like to get other people involved, starting with the Icedove maintainer. Volunteers are very much welcome, for any kind of involvement: bug triaging, documentation (which we are deeply lacking), testing, code… People interested can contact the pkg-mozilla team list. As the Debian packaging gets less time consuming, I should be able to spend more time mentoring, which should help.

If you could spend all your time on Debian, what would you work on?

I actually have been working almost full time on Debian for a few months during a more-or-less purposeful unemployment period. That’s when I attempted to add an alert popup when Iceweasel or extensions are upgraded (which was eventually removed because not working very effectively). That’s when I worked on Iceweasel 3.6 and the first 4.0 betas. That’s when I pushed more than 70 patches upstream, and got upstream commit access.

I know for a fact that keeping up with upstream and answering bug reports is already time consuming, and time comes in finite quantity. Working full time on Debian allows to do more than that, as I’ve been able to experience first hand. Unfortunately, that doesn’t help pay the bills, so it would be best if time could be shared with other persons. Like, actual members in the pkg-mozilla team. 😉

Anyways, if I were to work full time on Debian again, I’d like to do more than packaging, and help improve some of our infrastructure, most notably the buildd network, to grow some automated way to get packages built on all or some of our architectures. Call that PPA if you want, though I don’t think we need something exactly like PPA.

We sure do have porter boxes, but there is a huge overhead to get packages built there: you need to first try, then see you lack build dependencies, then wait for admins to install them, then come back and launch your build, and finally come back again much later to see how it went. And that’s when it goes smoothly. Repeat for each architecture.

It’s already cumbersome when it’s a small package, so imagine when it’s Iceweasel. I’d like to be able to push a (signed) .dsc somewhere, say I want it built on that and that architecture, and get binary packages and logs back. It could then be up to developers to distribute them, or not. Sometimes, you’re just interested in test suite results.

Another related wish would be a way to access build trees of failed builds, possibly on the buildd where it failed.

What’s the biggest problem of Debian?

Lack of contributors, lack of contributors, and… lack of contributors.

It can sound weird to say that in a community of 1000+ people. But the sad reality is that we’ve seen, repeatedly, calls for help in various places from various understaffed teams (most often of effectively one member) for years. So there is obviously a problem getting people to invest time in these teams either on the long term, or at all.

I know this is not a very significant metric, but it already should say a lot: I wouldn’t be surprised if more than 2/3 of the number of source lines of code in Debian come from less than 5% of the number of packages in the archive, and see less than 5% of the DD/DM population involved. If someone feels like doing the actual math, I’m curious to know how far I’m from reality, and in what direction.

We also lack manpower to be able to make our release development smoother. In a perfect world where manpower is not a scarce resource, we should work on $release+1 when $release is being frozen. Said otherwise, we should have been able to start working on Wheezy in August 2010, or even earlier.

You’ve been working for Mozilla Corporation for a few months. How does the work look like and what are you working on?

I am actually not working full time for MoCo; I am only contracting for 4 days a week. That was a request I made to keep having enough spare time for Debian or other activities. I am working from home, and am pretty much free about when I work, as long as I work long enough. I take advantage of this freedom and the saved commuting time to live a healthier life by doing some sports in the daytime (which also happens to be cheaper).

I mostly work with Taras Glek and other people on various aspects of Firefox startup performance. This involves pretty interesting low-level stuff, in which I actually got involved before being contracted. This work has nothing to do with packaging Iceweasel, though I can have a few opportunities to do things somehow related.

You told in your blog that you were discussing with Mozilla how Debian could use the Firefox trademark. Can you tell us a bit about the progress you made?

I have got an informal approval of the patches we are applying to the Iceweasel 4.0 beta packages. There are a few concerns about using the system cairo library, and the embedded copy of libpng needs to be used to have support for the animated PNGs that are used in the user interface and that are required in Firefox’ feature-set. Other than that, there are pretty much no objection on the technical side.

We managed to get to some common ground as to how to manage stable updates and more generally, changes to our packages, but I’m still waiting for an actual agreement draft. While I, as the maintainer, am fine with the way it should look according to my discussions with the people involved at Mozilla, I’ll seek review from my fellow Debian Developers when I’ll have something to show.

I haven’t decided yet how this will reflect on the Debian packages, though. For example, would Iceweasel stay? But I do hope Wheezy will get Firefox back.

Is there someone in Debian that you admire for their contributions?

Not someone, but all past, present and future release team members. This is a team that receives a lot of critics (and I indirectly gave one a few paragraphs ago), yet manages to keep doing an amazing job and its members are very committed to their task. I admire their dedication.

Thank you to Mike for the time spent answering my questions. I hope you enjoyed reading his answers as I did. Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Twitter and Facebook.

Sponsoring the initial upload

Sponsoring an update of an existing package

Wheezy kicks off

Tags

Recent Posts