Stories of life

Journey of a new GNOME 3 Debian packager

April 11, 2011 by Raphaël Hertzog

With all the buzz around GNOME 3, I really wanted to try it out for real on my main laptop. It usually runs Debian Unstable but that’s not enough in this case, GNOME 3 is not fully packaged yet and it’s only in experimental for now.

I asked Josselin Mouette (of the pkg-gnome team) when he expected it to be available and he could not really answer because there’s lots of work left. Instead Roland Mas gently answered me “Sooner if you help”. 🙂

First steps as a GNOME packager

This is pretty common in free software and for once I followed the advice, I spent most of sunday helping out with GNOME 3 packaging. I have no prior experience with GNOME packaging but I’m fairly proficient in Debian packaging in general so when I showed up on #debian-gnome (irc.debian.org) on sunday morning, Josselin quickly added me to the team on alioth.debian.org.

Still being a pkg-gnome rookie, I started by reading the documentation on pkg-gnome.alioth.debian.org. This is enough to know where to find the code in the SVN repository, and how to do releases, but it doesn’t contain much information about what you need to know to be a good GNOME packager. It would have been great to have some words on introspection and what it changes in terms of packaging for instance.

Josselin suggested me to start with one of the modules that was not yet updated at all (most packages have a pre-release version—usually 2.91—in experimental, but some are still at 2.30).

Packages updated and problems encountered

(You can skip this section if you’re not into GNOME packaging)

So I picked up totem. I quickly updated totem-pl-parser as a required build-dependency and made my first mistake by uploading it to unstable (it turns out it’s not a problem for this specific package). Totem itself was more complicated even if some preliminary work was already in the subversion repository. It introduces a new library which required a new package and I spent a long time debugging why the package would not build in a minimalistic build environment.

Indeed while the package was building fine in my experimental chroot, I took care to build my test packages like the auto-builders would do with sbuild (in sid environment + the required build-dependencies from experimental) and there it was failing. In fact it turns out pkg-config was failing because libquvi-dev was missing (and it was required by totem-pl-parser.pc) but this did not leave any error message in config.log.

Next, I decided to take care of gnome-screensaver as it was not working for me (I could not unlock the screen once it was activated). When built in my experimental chroot, it was fine but when built in the minimalistic environment it was failing. Turns out /usr/lib/gnome-screensaver/gnome-screensaver-dialog was loading both libgtk2 and libgtk3 at the same time and was crashing. It’s not linked against libgtk2 but it was linked against the unstable version of libgnomekbdui which is still using libgtk2. Bumping the build-dependency on libgnomekbd-dev fixed the problem.

In the evening, I took care of mutter and gnome-shell, and did some preliminary work on gnome-menus.

Help is still welcome

There’s still lots of work to do, you’re welcome to do like me and join to help. Come on #debian-gnome on irc.debian.org, read the documentation and try to update a package (and ask questions when you don’t know).

Installation of GNOME 3 from Debian experimental

You can also try GNOME 3 on your Debian machine, but at this point I would advise to do it only if you’re ready to invest some time in understanding the remaining problems. It’s difficult to cherry-pick just the required packages from experimental, I tried it and at the start I ended up with a bad user experience (important packages like gnome-themes-standard or gnome-icon-theme not installed/updated and similar issues).

To help you out with this, here’s a file that you can put in /etc/apt/preferences.d/gnome to allow APT to upgrade the most important GNOME 3 packages from experimental:

Package: gnome gnome-desktop-environment gnome-core alacarte brasero cheese ekiga empathy gdm3 gcalctool gconf-editor gnome-backgrounds gnome-bluetooth gnome-media gnome-netstatus-applet gnome-nettool gnome-system-monitor gnome-system-tools gnome-user-share baobab gnome-dictionary gnome-screenshot gnome-search-tool gnome-system-log gstreamer0.10-tools gucharmap gvfs-bin hamster-applet nautilus-sendto seahorse seahorse-plugins sound-juicer totem-plugins remmina vino gksu xdg-user-dirs-gtk gnome-shell gnome-panel dmz-cursor-theme eog epiphany-browser evince evolution evolution-data-server file-roller gedit gnome-about gnome-applets gnome-control-center gnome-disk-utility gnome-icon-theme gnome-keyring gnome-menus gnome-panel gnome-power-manager gnome-screensaver gnome-session gnome-settings-daemon gnome-terminal gnome-themes gnome-user-guide gvfs gvfs-backends metacity mutter nautilus policykit-1-gnome totem yelp gnome-themes-extras gnome-games libpam-gnome-keyring rhythmbox-plugins banshee rhythmbox-plugin-cdrecorder system-config-printer totem-mozilla epiphany-extensions gedit-plugins evolution-plugins evolution-exchange evolution-webcal gnome-codec-install transmission-gtk avahi-daemon tomboy network-manager-gnome gnome-games-extra-data gnome-office update-notifier shotwell liferea epiphany-browser-data empathy-common nautilus-sendto-empathy brasero-common
Pin: release experimental
Pin-Priority: 500

Package: *
Pin: release experimental
Pin-Priority: 150

The list might not be exhaustive and sometimes you will have to give supplementary hints to apt for the upgrade to succeed, but it’s better than nothing.

I hope you find this useful. I’m enjoying my shiny new GNOME 3 desktop and it’s off for a good start. My main complaint is that hamster-applet (time tracker) has not yet been integrated in the shell.

Review of my Debian related goals for 2010

January 17, 2011 by Raphaël Hertzog

Last year I shared my “Debian related goals for 2010”. I announced that I would not be able to complete them all and indeed I have not, but I have still done more than what I expected. Let’s have a look.

Translate my Debian book into English and get it published: NOT DONE

Or rather not yet done. It’s still an important project of mine and I will do it this year. When I wrote this last year, I expected to find a publisher that would take care of everything but we failed to find a suitable one so we’re going to do it ourselves.

Cleanup the dpkg perl API and create libdpkg-perl: DONE

libdpkg-perl has been introduced with dpkg 1.15.6.

Create dpkg-buildflags: DONE

dpkg-buildflags has been introduced with dpkg 1.15.7. It’s not widely used yet and it won’t really be until debhelper 7 supports it (see #544844). It would be nice to see progress on this front this year.

Ensure the new source formats continue to gain acceptance: DONE

The adoption rate has been steady, it clearly slowed down since the freeze though. I have implemented quite a few features to satisfy the needs of users, like the possibility to unapply the patches after the build, or the possibility to fail in case of unwanted upstream changes.

Design a generic vcs-buildpackage infrastructure to be integrated in dpkg-dev: NOT DONE

I believe it’s something important on the long term but it never made it to my short-term TODO list and it’s unlikely to change any time soon.

Continue fixing dpkg bugs faster than they are reported: PARTLY DONE

We have dealt with many bugs over the year, but we still have 20 more bugs than at the start of last year (370 vs 350). We’re not doing bad compared to many other Debian teams but we can still benefit from some help. Start here if you’re interested.

Enhance our infrastructure to ease interaction between contributors and to have a better view of how each package is maintained: NOT DONE

I am convinced that we need something to have a clearer idea of the commitments made by each contributor. I don’t put the same amount of care in maintaining smarty-gettext that I do on dpkg. If we had a database of the stuff that we know we don’t do well enough, it’s easier to point new contributors towards those.

Anyway, this project is still unlikely to come to the top of my priorities any time soon.

Work on the developers-reference: NOT DONE

We have switched the Maintainer field to debian-policy@lists.debian.org to have more review of the changes suggested through the bug tracking system but that has not changed much on the global situation.

I still hope to become more active on it sometimes this year. Maybe by trying to make it more fun and creating the text for some of the wishlist bugs as blog articles first.

Rewrite in C the last Perl scripts provided by the dpkg binary package: DONE

Dpkg 1.15.8 was the first version working without Perl, I announced it in July.

Integrate the 3-way merge tool for Debian changelogs in dpkg-dev: DONE

dpkg-mergechangelogs is part of dpkg-dev since 1.15.7.

I enjoy it regularly. Unfortunately it doesn’t work well for cherry-picks. Would be nice to see this fixed, anyone up to the task? 🙂

Click here to subscribe to my free newsletter and get my monthly analysis on what’s going on in Debian and Ubuntu. Or just follow along via the RSS feed, Identi.ca, Twitter or Facebook.

The secret plan behind the “3.0 (quilt)” Debian source package format

October 21, 2010 by Raphaël Hertzog

New source package formats do wonders While I have spent countless hours working on the new source format known as “3.0 (quilt)”, I’ve just realized that I have never blogged about its features and the reasons that lead me to work on it. Let’s fix this.

The good old “1.0” format

Up to 2008, dpkg-source was only able to cope with a single source format (now named “1.0”). That format was used since the inception of the project. While it worked fine for most cases, it suffered from a number of limitations—mainly because it stored the Debian packaging files as a patch to apply on top of the upstream source tarball.

This patch can have two functions: creating the required files in the debian sub-directory and applying changes to the upstream sources. Over time, if the maintainer made several modifications to the upstream source code, they would end up entangled (and undocumented) in this single patch. In order to solve this problem, patch systems were created (dpatch, quilt, simple-patchsys, dbs, …) and many maintainers started using them. Each implementation is slightly different but the basic principle is always the same: store the upstream changes as multiple patches in the debian/patches/ directory and apply them at build-time (and remove them during cleanup).

Design goals for the new formats

When I started working on the new source package format, I set out to get rid of all the known limitations and to integrate a patch system in dpkg-source. I wanted to clear up the situation so that learning packaging only requires to learn one patch system and would not require modifying debian/rules to use it. I picked quilt because it was popular, came with a large set of features, and was not suffering from NIH syndrome. This lead to the “3.0 (quilt)” source format.

I also created “3.0 (native)” as a distinct format. “1.0” was able to generate two types of source packages (native and non-native) but I did not want to continue with this mistake of mixing both in a single format. The KISS principle dictated that the user should pick the format of his choice, put it in debian/source/format and be done with it. Now the build can rightfully fail when the requirements are not met instead of doing something unexpected as a fallback.

Features of “3.0 (quilt)”

This is the format that replaces the non-native variant of the 1.0 source format. The features below are specific to the new format and differentiate it from its ancestor:

Supports compression formats other than gzip: bzip2, lzma, xz.
Can use multiple upstream tarballs.
Can include binary files in the debian packaging.
Automatically replaces the “debian” directory present in the upstream tarball (no repacking required).
Creates a new quilt-managed patch in debian/patches/ when it finds changes to the upstream files.

Features of “3.0 (native)”

This format is very similar to the native variant of the 1.0 source format except for two things:

it supports compression formats other than gzip: bzip2, lzma, xz.
it excludes by default a bunch of files that should usually not be part of the tarball (VCS specific files, vim backup files, etc.)

Timeline

Looking back at the history is interesting. This project already spans multiple years and is not really over until a majority of packages have switched to the new formats.

January 2008: the discussion how to cope with patches sanely rages on debian-devel@lists.debian.org. My initial decisions are the result of this discussion.
March 2008: I have implemented the new formats and I request feedback. dpkg 1.14.17 (uploaded to experimental) is the first release supporting them.
April 2008: I ask ftpmasters to support the new source packages in #457345.
June 2008: Lenny freeze. dpkg is not supposed to change anymore. Several changes concerning the new source formats are still accepted in the following months given that this code is not yet used in production and must only be present so that lenny can cope with new source packages once squeeze starts using them.
February 2009: Lenny release.
March 2009: Work on squeeze has started, ftpmasters have done nothing to support new source formats, I submit a patch in #457345 to speed things up. I start a wiki page to track the project’s progress and to answer common questions of maintainers.
November 2009: After an ftpmaster sprint, it’s now possible to upload new source packages in unstable. This draws massive attention to the new format and some people start complaining about some design decisions. The implementation of “3.0 (quilt)” changes a lot during this month. dpkg in lenny is even updated to keep up with those changes.
March 2010: Up to now, I was planning to let dpkg-source build new source packages by default at some point in the future. After several rounds of discussions, I agree that it’s not the best course of action and decides instead to make debian/source/format mandatory. The maintainer must be explicit about the source format that s/he wants to use.
October 2010: The new source formats are relatively popular, a third of the source packages have already switched: see the graph. The squeeze freeze in August clearly stopped the trend, hopefully it will continue once squeeze is released.
June 2013: Project is finished?

As you can see this project is not over yet, although the most difficult part is already behind me. For my part, the biggest lesson is that you won’t ever get enough review until your work is used within unstable. So if you have a Debian project that impacts a lot of people, make sure to organize an official review process from the start. And specifying your project through a Debian Enhancement Proposal is probably the best way to achieve this.

If you appreciate the work that I put into this project, feel free to join Flattr and to flattr dpkg from time to time. Or check out my page “Support my work“.

5 years of Freexian

January 5, 2010 by Raphaël Hertzog

5 years ago I founded my own company Freexian SARL with the goal to make a living out of my free software experience. I marketed the company as being specialized on the Debian distribution in the hope to combine my Debian work and my professional work.

Given that Freexian is still alive I think I met the first goal. My free software experience allowed me to complete many projects: a large number of development projects for embedded devices running a custom Linux distribution (usually built with debian udebs), the development of a Debian derivative (SLIS) and some recurring tasks of remote system administration.

However, even if I use Debian daily for all my work, very few of the projects that I complete for customers have direct results in terms of improvements for Debian (except some bugreports and some related fixes). And even when I’m able to contribute something back to Debian, it’s usually not in areas that I care about.

My focus within Debian is on the technical and organizational infrastructure of the project: as a dpkg/dpkg-dev maintainer I try to improve the packaging infrastructure, as a QA member I maintain the Package Tracking System to ease collaboration, as an Alioth admin I ensure all DD can host VCS repositories for their Debian related projects, as a developers-reference co-maintainer I try to share good packaging practices, etc. Given this bias, it’s difficult to find customer projects that would let me contribute in those areas. Thus I think I need to try another approach: the simplest solution would be to find sponsors for some of my own Debian-related projects (if you have something else to suggest, please leave a comment — either in the blog or by mail).

That said finding sponsors looks like a difficult task in itself. While I can imagine (for example) a company using Debian on embedded devices that would like to sponsor the rewrite of update-alternatives in C in order to get rid of the perl dependency in the dpkg package (if you know such a company, get in touch with me!), I don’t see who would have an interest in sponsoring the time that I need to contribute new sections to the developers-reference manual. But who knows… maybe I should just try and publicly solicit sponsorship for some of the projects that I care about. In any case, suggestions and comments are welcome!