apt-get install debian-wizard

March 2011 wrap up

April 3, 2011 by Raphaël Hertzog

Since I’m soliciting donations to support my Debian work, the least I can do is explain what I do. You can thus expect to see an article like this one every month.

Multi-Arch work

I updated the code to use another layout for the control files stored in /var/lib/dpkg/info/. Instead of using a sub-directory per architecture (arch/package.type), we decided to use package:arch.type but only for packages which are Multi-Arch: same. dpkg is taking care to rename the files the first time it is executed with write rights and then updates /var/lib/dpkg/info/format to remember that the upgrade has been done and that we can rely on the new structure.

I filed a few bugs on packages that are improperly accessing those internal files instead of using the appropriate dpkg-query interface. I sent a heads-up mail on -devel to make other people aware of those problems in the hope to discover most of them as early as possible.

After that, the work stalled because Guillem went away for 2 weeks and thus stopped his review of my work. I hope he will quickly resume the review and that we will get something final this month.

With the arrival of dpkg 1.16.0, it’s now possible to start converting libraries to multi-arch even if full multi-arch support has not yet landed in dpkg proper. See http://wiki.debian.org/Multiarch/Bootstrapping for the detailed plan.

If you’re curious about Multi-Arch, you might want to read this article of Steve Langasek as well.

Bug triage for dpkg in launchpad

At the start of the month, there was close to 500 bugs reported against the dpkg package in Launchpad. Unfortunately most of it is noise… many of the reported bugs are misfiled, they show an upgrade problem of a random package and that upgrade problem confuses update-manager which tries to configure an already configured package. This generates a second error that apport attributes to dpkg and the resulting bug report is thus filed on dpkg. There are literally hundreds of those that have to be reclassified.

Michael Vogt and Brian Murray did some triaging, and I also spend quite some hours on this task. It’s a bit frustrating as I tend to mark many reports “Incomplete” because there’s no way they can be acted upon and many of them are so old that the reporter is unlikely to be able to provide supplementary information.

But in the middle of this noise, there are some useful bug reports, like LP#739179 which enabled me to fix a regression even before it reached Debian Unstable (because Ubuntu runs a snapshot of dpkg with multiarch support).

I subscribed to the Launchpad bugs for dpkg via the Debian Package Tracking System (thanks to the derivatives-bugs keyword) and will try to keep up with the incoming reports.

Misc dpkg work

The ftpmasters came up with a request for a new field (see 619131) in source packages. After a quick discussion and a round of review on debian-policy@l.d.o, I implemented the new Package-List field. This should allow the ftpmasters to save some time in NEW processing, but we deferred the change for the next dpkg version (1.16.1) to ponder a bit more on the design of the field.

I also fixed a bunch of bugs (#619541, #605719, #598922, #616096) and merged a patch of Mark Hymers to recognize the new Built-Using field.

Developers-reference work

The review process for changes to the developers-reference is not working as it should. And I suffered from it while trying to integrate the patch I wrote for the “Developer duties” chapter (see #548867).

We purposely changed the maintainer field from debian-doc to debian-policy in the hope to have more reviews of suggested changes and to seek some sort of consensus before committing anything. But we don’t get more reviews… and deciding to commit a patch is now even harder than it was (except for trivial stuff where personal opinions can’t interfere).

In my case, I only got the feedback of Charles Plessy which was very mixed to say the least. I tried to improve my patch based on what he expressed but I also clearly disagreed with some of his assertions and was convinced that my wording was in line with the dominant point of view within Debian.

We tried to involve the release team in the discussion because most of what I documented was about helping making stable release happen, but nobody of the team answered.

Instead of letting the situation (and my patch) rot, I solicited feedback from the DPL and from another developers-reference editor to see whether my patch was an improvement or not. After some more time, I went ahead and committed it.

It was not pleasant for anyone.

I don’t know how we can improve this. Contrary to the policy, the developers-reference is a document that is not normative, I believe the result is better when we put some “soul” into it. But it’s a real challenge when you seek a consensus and that the interest in reviewing changes is so low.

DVD shop listed on debian.org

In February, I launched a DVD shop whose benefits are used to fund my Debian work. Shortly after the launch I used the official form to be added to the official listing of Debian CD vendors and offered a few suggestions to deal with vendors who are selling unofficial images (with firmware in my case).

A few weeks later, I got no answers: neither for my request nor for my suggestions, I mailed the cdvendors@debian.org team directly asking for a status update and quickly got an answer suggesting that Simon Paillard usually does the work and can’t process the backlog due to some injury. At this point no concerns had been raised about adding me to the list. To save some time and some work for the team, I added myself to the list since I had commit rights and I informed them that I did it, so that they can review it.

Shortly after I did that, Martin Zobel Helas objected to my addition. I cleared some misunderstandings but the discussion also lead to some changes to please everybody: the listing now indicates that some images are unofficial and I have prepared a special landing page for people coming from the Debian website through this listing.

Debian column on OMG! Ubuntu

I have always been a firm believer that it’s important for Debian to reach out to the widest public with its message of freedom. Thus when Benjamin Humphrey contacted the debian-publicity team to find volunteers to write a Debian column on OMG! Ubuntu, I immediately jumped in.

I wrote 4 articles over there. The tone is very different from my articles on my blog and I like that duality. Check out Debian is dying! Oh my word!, Debian or Ubuntu, which is the best place to contribute?, Are you contributing your share? and Ubuntu’s CTO reveals DEX: an effort to close the gap with Debian.

It’s a great win-win situation, OMG! Ubuntu benefits from my articles, Debian’s values are relayed further, and OMG! Ubuntu’s large audience also helps me develop my own blog.

Work on my book

I had lots of paperwork to do this month (annual accounting stuff for my company) and I did not have as much time as I hoped for my book. Still I have a updated a few more chapters of my French book and I certainly hope to complete the update during April.

This means that the work on the English translation could start in may.

Work on my blog

Just like for my book, it has been relatively difficult for me to cope with my policy of two articles every week. But I still managed to get quite some good stuff out.

I interviewed Christian Perrier (Debian’s translation coordinator) and also Bdale Garbee (chair of Debian’s technical committee).

I finished my series of “Debian Cleanup Tips” with 2 supplementary articles:

The removal of firmware is causing troubles to quite some users so I wrote an article explaining how to deal with the problem. A regular reader also asked me to write an article about Jigdo, I executed myself because it was a good idea and that he has been very nice with me: Download ISO images of Debian CD/DVD at light speed with Jigdo.

Last but not least, I shared my package maintainer pledge which inspired my developers-reference patch (see discussion above).

Thanks

Many thanks to all the people who showed their appreciation of my work. The 324.37 EUR that you gave me in February represented 2 days and a half of my time that I have spent working on the above projects.

See you next month for a new summary of my activities.

Free and Open Source Software Projects to Flattr

April 1, 2011 by Raphaël Hertzog

Flattr recently launched FlattrChattr, a blog featuring news about what’s happening in the Flattr ecosystem. I hear that they might do an article about Flattr FOSS… and it’s not an April fool’s joke 🙂

Anyway, here are 5 suggestions of free software related “things” that you can support with Flattr.

Libre Office via The Document Foundation (Flattr link). This is the fork of the famous OpenOffice.org that all major distributions (including Debian) are using. The foundation runs a fundraising challenge to cover the expected costs in terms of marketing, hardware, infrastructure and so on.
Greenshot (Flattr link) is a nice Windows application to take screenshots.
Debian Packages that Need Lovin’ (Flattr link) is a web interface where you can easily browse the list of Debian packages that are in need of some help, or a new maintainer. It’s much more usable than the corresponding entry in the bug tracker (see wnpp) or even than the corresponding page on the official website (see here).
Patchage (Flattr link) is a modular patch bay for Jack audio and Alsa Midi. Patchage provides a graphical interface to connect jack and midi inputs and outputs. Each application is represented as one box with inputs on the left and output on the right. Boxes can be moved around and arranged to have a clear display of the current setup. (Description copied from the corresponding Debian package because I’m not familiar at all with any of this stuff)
Seif Lotfy (Flattr link) is one of the leading developers behind the Zeitgeist framework used by modern desktops to provide more “contextual awareness”. The framework logs your activity, establishes relationships between your documents based on many criteria, and makes all the information available to applications. They can use it to make it easier to find your documents, to provide improved contextual suggestions, etc.

This article is part of the Flattr FOSS project.

People behind Debian: Bdale Garbee, chair of the technical committee

March 28, 2011 by Raphaël Hertzog

Bdale is a long-time Free Software believer, he has been contributing even before Debian existed… in the prehistoric era of free software. 🙂

Anyone who went to a big Free Software conference has seen one of his colorful t-shirts. Or maybe you have heard the story where he got his beard shaved by Linus Torvalds to raise funds to protect the Tasmanian Devil.

More seriously Bdale has played and continue to play a number of important roles in the Debian community. He also represents one of the biggest corporate sponsors (both for DebConf and for the servers that Debian owns): Hewlett Packard.

My questions are in bold, the rest is by Bdale.

Who are you?

I made my first personal contribution of source code to what we now call Free Software in 1979. I started with HP in 1986 and for nearly a decade have served the company as Chief Technologist for Open Source & Linux. I am president of Software in the Public Interest, which is the “umbrella organization” providing legal and financial existence for Debian in the USA. I also represent users, developers, and Debian interests on a number of boards including at the Linux Foundation and the Freedombox Foundation.

I’m happily married with two children. Many people in Debian have met some or all of my family. They all joined me for Debconf in Edinburgh, and my daughter Elizabeth also attended in Caceres and New York.

I joined Debian in 1994. I’ve been responsible for a number of packages essential to our base system continuously since that time. But I’ve also contributed to the project in many other ways over the years. I ran the first server that was fully dedicated to Debian. Ideas of mine influenced the development of project infrastructure, from the early design of our mirror network to structuring the archive around a ‘package pool’. I started or made significant early contributions to 5 ports of Debian to non-i386 architectures. I served as Debian Project Leader (DPL) in 2002-2003, was acting Secretary for a while, and have served on the Technical Committee for a number of years.

Over the years, I’ve also had some interesting hobbies. I helped design, build, and program pieces of various amateur radio satellites. I enjoy making physical things, and have many tools for working in wood and metals. My son and I are very active in the world of high power model rockets. And with my partner (and fellow Debian developer!) Keith Packard I’m now running a small business making and selling open hardware and open source avionics for hobby rockets. You can read more about that at http://altusmetrum.org.

You’re the chair of the Debian technical committee. Can you quickly explain the role of this committee?

I think many people assume the Technical Committee has a larger role in Debian than it really does. Section 6 of Debian’s constitution defines the official role of the Technical Committee. Most importantly, the committee exists as a last resort place to resolve technical conflicts between Debian developers that they are unable to resolve by themselves. Most of the power in Debian is left in the hands of individual developers, who are usually able to collaborate with each other to make good technical decisions. So the Technical Committee’s resolution process has only rarely been needed, which I think is a very good thing.

From my point of view, the technical committee is not working. In many cases, the committee does not take any (timely) decision and just waits until the underlying situation has evolved to a point where the intervention of the committee is no longer needed. Do you agree with this and how can you explain it?

I think it’s very important for all of us to remember that everyone working on Debian does so voluntarily, and people who volunteer their time generally deserve a measure of respect and appreciation for their efforts.

No issue is brought to the Technical Committee unless resolving it is expected to be really difficult, or at least contentious. And often, the issues brought to the committee have been as much or more about personality than technology. That makes some of them really hard to solve.

So I do not agree that the technical committee is not working. It seems to me that the decisions that bog down and take a long time are the ones where arguments start out or become emotional instead of technical. In this context, if committee members can help lead public and private discussions in a way that causes a situation to evolve to the point where a decision is no longer needed, that may be healthier for the project in the long term than a quick vote that satisfies some contributors at the expense of others.

The last important change that was made to try to revive the committee was the addition of two new members (Don Armstrong and Russ Allbery). Is there anything else that could be tried?

The biggest improvement I could personally wish for is something people sending issues to the committee can help with. As the ultimate technical decision making body for a project whose output is mostly software, the more a request can be put in terms of a decision about source code, the easier it will be for us to make a decision. That won’t always be possible, but when we’re forced to try and dream up alternatives and then figure out whether anyone would actually be willing to write the code to implement those alternatives, the process takes a lot longer than choosing between competing patch sets or deciding whether a patch should be included.

Besides your role in the technical committee, you have held the role of mediator/facilitator/advisor on numerous occasions. Because you’re an old wise bearded guy who travels a lot and knows many Debian contributors… I would like to thank you for all this work that few people notice. Are there been times where this has been a real burden for you?

Thank you for mentioning this. I’ve put a lot of my heart into Debian over the years, largely because it’s a project and a community that continues to amaze and inspire me.

I feel fortunate to have been able to meet and work on Debian with so many outstanding people from around the world. Many are now my friends, with all the silly and serious things being a friend implies. I’ve been asked for and have given advice many times. I’ve helped celebrate birthdays, marriages, new jobs, and the arrival of children. Sadly, I have also found myself having to try and find the right words to mark the loss of some of these friends…

The only time any of this feels like a burden is when there’s some important problem that many people care about, that I’m working “behind the scenes” to help fix, but can’t really talk about publicly without causing more harm than good. It’s distressing to have people think you don’t care or aren’t helping, when really you’re doing everything you possibly can… just not in a publicly visible way. Of course I understand that this is an impossible situation. If you can’t see what’s happening, there’s no way to know if something is happening or not. That’s why I advocate doing as much as possible in Debian, and SPI, and everywhere else I contribute in as open a way as possible.

You have been Debian Project Leader and you promoted the vision of Debian as the Universal Operating System. What does “universal” mean for you?

The biggest thing to me at the time was the idea that Debian could be anything. Those who chose to work on Debian would ultimately determine what Debian became. I also wanted to make sure we thought about as broad a set of potential users and collaborators as possible.

But this vision provided a framework for pursuing a whole range of worthwhile increases in Debian’s scope of utility, some of which I articulated in my DPL platforms, some others put forward. Internationalization, porting to more supported architectures, our inclusive and evolving approaches to accepting new developers and new packages, and so forth.

I think this vision has served us well, and it pleases me that it has stayed a part of our collective thinking for so long.

We’re again in Debian’s electoral period, what do you think of the work done by the current DPL?

I’m very happy with what I’ve observed of Stefano’s activities during his first year as DPL. He has an obvious enthusiasm for Debian, communicates well both in one to one interactions and in front of a crowd, and I think represents Debian very well.

It is interesting that he’s running unopposed for re-election this year. I choose to interpret that as evidence he’s doing a good job, the project is running well, and nobody feels the need to try and take the job away From him. I’m glad he’s willing to continue in this role for another year.

What’s the most important thing that Debian should achieve in the wheezy timeframe?

I don’t yet have a very crisp personal wish-list for wheezy. But I would certainly like to see multiarch support finally completed! I’m also very interested to see what comes from the CUT work.

You have been an early supporter of “multiarch”, a project to allow easy installation of foreign architecture packages. It’s on good track for Wheezy. Do you think it’s an important milestone?

My original motivation for requesting multiarch support was to enable support for 32-bit x86 binaries on ia64 “Itanium” systems, in the time leading up to the “sarge” release. I ended up creating the ia32-libs package, which I’m not proud of. The emergence of 64-bit extensions to x86 (the amd64 architecture) made this a much broader issue. Today, I run a 64 bit kernel and a 32 bit user space on my notebook. There are problems with just moving entirely to 64 bit… but I would like to be able to run some applications that work with large data sets in full 64 bit mode!

Thank you to Bdale Garbee for the time spent answering my questions. I hope you enjoyed reading his answers as I did. Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Twitter and Facebook.

Download ISO images of Debian CD/DVD at light speed with Jigdo

March 22, 2011 by Raphaël Hertzog

Debian CD/DVD ISO images are huge files (4.7 GB for a DVD). It can take a long time to download if your connection with the server is not very fast. Jigdo can help you download them much more quickly if you have a high-speed connection to a normal Debian mirror.

If your download rate from your usual Debian mirror is not better than the download rate from the closest Debian CD mirror, then you should download your images straight from the Debian CD mirror. Otherwise read on and learn how you can save lots of time.

Jigdo is based on a simple idea: the CD images contain mainly thousands of .deb files (and source files too) that are already available on all Debian mirrors. So instead of downloading the ISO image in its entirety, you could reconstruct it with all the files that it contains, that you would download from your usual Debian mirror.

Thus when you download a .jigdo file, you only download a little text file (compressed with gzip) that contains enough information to reconstruct the complete image:

it contains a list of the individual files that are needed to rebuild the image;
it indicates a mirror where those files can be downloaded (it’s mainly a fallback URL in case our preferred mirror doesn’t have them anymore);
it contains an URL to a .template file which describes the layout of all files within the ISO image. It also contains the data which can’t be downloaded from the mirror.

The tool that we will use to regenerate the ISO images is called jigdo-lite, it’s available on Windows if you don’t have any Debian system yet. Otherwise just install the jigdo-file Debian package.

We just need the URL of a .jigdo file (get one from the Debian CD page) and we pass it to jidgo-lite on the command line. It asks for a path of “Files to scan”: if we have an older version of the same CD image, we can indicate a path where it is mounted and it will avoid redownloading the files that we already have, otherwise just press Enter. Then it asks for the Debian mirror to use and by default suggests the one used in /etc/apt/sources.list. Again pressing Enter is usually enough. You can even use --noask to avoid the prompts.

$ jigdo-lite http://cdimage.debian.org/cdimage/release/6.0.1/multi-arch/jigdo-dvd/debian-6.0.1-i386-amd64-source-DVD-1.jigdo

Jigsaw Download "lite"
Copyright (C) 2001-2005  |  jigdo@
Richard Atterer          |  atterer.net
Getting mirror information from /etc/apt/sources.list

Downloading .jigdo file
--2011-03-22 09:08:11--  http://[2001:6b0:e:2018::163]/cdimage/release/6.0.1/multi-arch/jigdo-dvd/debian-6.0.1-i386-amd64-source-DVD-1.jigdo
[...]

-----------------------------------------------------------------
Images offered by `http://cdimage.debian.org/cdimage/release/6.0.1/multi-arch/jigdo-dvd/debian-6.0.1-i386-amd64-source-DVD-1.jigdo':
  1: 'Debian GNU/Linux 6.0.1 "Squeeze" - Official Multi-architecture i386/amd64/source DVD #1 20110319-14:55 (20110319)' (debian-6.0.1-i386-amd64-source-DVD-1.iso)

Further information about `debian-6.0.1-i386-amd64-source-DVD-1.iso':
Generated on Sat, 19 Mar 2011 15:11:27 +0000

-----------------------------------------------------------------
If you already have a previous version of the CD you are
downloading, jigdo can re-use files on the old CD that are also
present in the new image, and you do not need to download them
again. Mount the old CD ROM and enter the path it is mounted under
(e.g. `/mnt/cdrom').
Alternatively, just press enter if you want to start downloading
the remaining files.
Files to scan: 

-----------------------------------------------------------------
The jigdo file refers to files stored on Debian mirrors. Please
choose a Debian mirror as follows: Either enter a complete URL
pointing to a mirror (in the form
`ftp://ftp.debian.org/debian/'), or enter any regular expression
for searching through the list of mirrors: Try a two-letter
country code such as `de', or a country name like `United
States', or a server name like `sunsite'.
Debian mirror [http://ftp.fr.debian.org/debian]: 

Downloading .template file
--2011-03-22 09:12:41--  http://cdimage.debian.org/cdimage/release/6.0.1/multi-arch/jigdo-dvd/debian-6.0.1-i386-amd64-source-DVD-1.template
[...]
Successfully created `debian-6.0.1-i386-amd64-source-DVD-1.iso'

-----------------------------------------------------------------
Finished!
The fact that you got this far is a strong indication that `debian-6.0.1-i386-amd64-source-DVD-1.iso'
was generated correctly. I will perform an additional, final check,
which you can interrupt safely with Ctrl-C if you do not want to wait.

OK: Checksums match, image is good!

And you’re done, you have successfully downloaded an official Debian DVD with jigdo-lite. You can head over to the Debian CD page and find out the URL of the .jigdo files for the latest stable release.

One last thing, Bluray images are only available with Jigdo because they are so huge that keeping complete ISO images would be a waste of disk resources of the CD images server (and its mirrors).

Of course, if downloading ISO images and burning CD/DVD is not for you, you can always order some professionally printed CD/DVD. You will even help Debian by doing this.

If you want to read more articles like this one, click here to subscribe to my free newsletter. You can also follow me on Identi.ca, Twitter and Facebook.