Review of my Debian related goals for 2010

Last year I shared my “Debian related goals for 2010”. I announced that I would not be able to complete them all and indeed I have not, but I have still done more than what I expected. Let’s have a look.

Translate my Debian book into English and get it published: NOT DONE

Or rather not yet done. It’s still an important project of mine and I will do it this year. When I wrote this last year, I expected to find a publisher that would take care of everything but we failed to find a suitable one so we’re going to do it ourselves.

Cleanup the dpkg perl API and create libdpkg-perl: DONE

libdpkg-perl has been introduced with dpkg 1.15.6.

Create dpkg-buildflags: DONE

dpkg-buildflags has been introduced with dpkg 1.15.7. It’s not widely used yet and it won’t really be until debhelper 7 supports it (see #544844). It would be nice to see progress on this front this year.

Ensure the new source formats continue to gain acceptance: DONE

The adoption rate has been steady, it clearly slowed down since the freeze though. I have implemented quite a few features to satisfy the needs of users, like the possibility to unapply the patches after the build, or the possibility to fail in case of unwanted upstream changes.

Design a generic vcs-buildpackage infrastructure to be integrated in dpkg-dev: NOT DONE

I believe it’s something important on the long term but it never made it to my short-term TODO list and it’s unlikely to change any time soon.

Continue fixing dpkg bugs faster than they are reported: PARTLY DONE

We have dealt with many bugs over the year, but we still have 20 more bugs than at the start of last year (370 vs 350). We’re not doing bad compared to many other Debian teams but we can still benefit from some help. Start here if you’re interested.

Enhance our infrastructure to ease interaction between contributors and to have a better view of how each package is maintained: NOT DONE

I am convinced that we need something to have a clearer idea of the commitments made by each contributor. I don’t put the same amount of care in maintaining smarty-gettext that I do on dpkg. If we had a database of the stuff that we know we don’t do well enough, it’s easier to point new contributors towards those.

Anyway, this project is still unlikely to come to the top of my priorities any time soon.

Work on the developers-reference: NOT DONE

We have switched the Maintainer field to debian-policy@lists.debian.org to have more review of the changes suggested through the bug tracking system but that has not changed much on the global situation.

I still hope to become more active on it sometimes this year. Maybe by trying to make it more fun and creating the text for some of the wishlist bugs as blog articles first.

Rewrite in C the last Perl scripts provided by the dpkg binary package: DONE

Dpkg 1.15.8 was the first version working without Perl, I announced it in July.

Integrate the 3-way merge tool for Debian changelogs in dpkg-dev: DONE

dpkg-mergechangelogs is part of dpkg-dev since 1.15.7.

I enjoy it regularly. Unfortunately it doesn’t work well for cherry-picks. Would be nice to see this fixed, anyone up to the task? :-)

Click here to subscribe to my free newsletter and get my monthly analysis on what’s going on in Debian and Ubuntu. Or just follow along via the RSS feed, Identi.ca, Twitter or Facebook.

How to create custom RSS feeds with WordPress

WordPress has many alternate built-in feeds: per category, per tag, per author, per search-keyword. But in some cases, you want feeds built with some more advanced logic. Let’s look at the available options.

WordPress advanced built-in feeds

You can create feeds for “unions” or “intersections” of tags, you just have to use a URL like /tag/foo,bar/feed/ (all articles tagged with foo or bar) or /tag/foo+bar/feed/ (all articles tagged with foo and bar).

You can also have feeds excluding a category, although that requires you to know the category identifier (and hardcode it in the URL like this: /?feed=rss2&cat=-123 where 123 is the category id that you want to exclude).

But there’s no simple way to have a feed that excludes articles with a given tag. The best solution found involves creating a custom feed. I’ll show you a variation of this below.

Creating a custom feed

  1. First of, install the Feed Wrangler plugin, it will take care of registering our custom feeds with wordpress.
  2. Go to “Settings > Feed Wrangler” in your WordPress administrative interface and create a new feed, let’s call it “myfeed”.
  3. You should now create a “feed-myfeed.php” file and put it in your current theme’s directory. The initial content of that file should be this:
    <?php
    include('wp-includes/feed-rss2.php');
    ?>
  4. At this point, you already have a new feed that you can access at /feed/myfeed/ (or /?feed=myfeed). It’s a complete feed like the main one.

Now, we’re going to look at ways to customize this feed. We’re going to do this by changing/overriding the default query that feed-rss2.php’s loop will use.

A feed excluding articles with a tag

If you want to create a feed that excludes the tag “foo”, you could use this:

<?php
global $wp_query;
$tag = get_term_by("slug", "foo", "post_tag");
$args = array_merge(
        $wp_query->query,
        array('tag__not_in' => array($tag->term_id))
);
query_posts($args);
include('wp-includes/feed-rss2.php');
?>

That was relatively easy, thanks to the “tag__not_in” query parameter. Now you can further customize the feed by adding supplementary query parameters to the $args array. The documentation of query_posts details the various parameters that you can use.

A feed excluding articles with a custom field (meta-data)

I went further because I did not want to use a tag to exclude some posts: that tag would have been public even if it was only meaningful to me. So I decided to use a custom field to mark the posts to exclude from my new feed. I named the field “no_syndication” and I always give it the value “1″.

This time it’s not so easy because we have no query parameter that can be used to exclude posts based on custom fields. We’re going to use the “post__not_in” parameter that can be used to exclude a list of posts. But we must first generate the list of posts that we want to exclude. Here we go:

<?php
global $wp_query;
$excluded = array();
$args_excluded = array(
    'numberposts'     => -1,
    'meta_key'        => 'no_syndication',
    'meta_value'      => 1,
    'post_type'       => 'post',
    'post_status'     => 'published'
);
foreach (get_posts($args_excluded) as $item) {
        $excluded[] = $item->ID;
}
$args = array_merge(
        $wp_query->query,
        array('post__not_in' => $excluded)
);
query_posts($args);
include('wp-includes/feed-rss2.php');
?>

A feed with modified content

You might want to add a footer to the articles that are syndicated. I use the Ozh’ Better Feed plugin for this but it applies to all your feeds.

You could do that sort of transformation only in your customized feed by using the WordPress filter named the_content_feed.

Here’s a simple example:

<?php
function myfeed_add_footer($content) {
        return $content . "<hr/>My footer here";
}
add_filter('the_content_feed', 'myfeed_add_footer');
include('wp-includes/feed-rss2.php');
?>

I’ll stop here but obviously you have lots of options and many ways to tweak all the snippets above. They have been tested with WordPress 3.0.4.

Note that in all those examples, I took care to not duplicate the code from feed-rss2.php, instead I used include() to execute it. That way my custom feeds will automatically benefit from all the future enhancements and fixes made by the WordPress developers.

But if you have to modify the XML structure of your custom feeds, you can paste the content of feed-rss2.php in your file and change it like you want…

Librement: a new way to help people who want to contribute to free software

Find your way in the free software worldI have this project in my head, I want to work on it but I always lack the time. In order to go forward, I thought I could write about it, at least it would let me clarify my ideas and the core goals. So here I am, I will present you Librement (I have registered the alioth project but it’s empty).

The core goal is to make it easy for every user to contribute to free software in some way. I will now present the main features that I envision.

Defining skills and interests

In order to propose tasks that the user can do, we must have an idea of his skills. So on the first run (and later through a preferences menu) the user will be invited to define his skills:

  • his native languages (multiple allowed)
  • other languages he can understand
  • programming languages he knows
  • version control systems he can use
  • markup language he knows (HTML, DocBook, Wiki-like formats, etc.)
  • etc.

Maybe we can also ask which skills he would like to learn. Because contributing to free software is a nice opportunity to learn new skills!

We should also find out what the user is interested in. What are his favorite free software projects? What kind of contributions would he like to do (documentation, translation, coding, bug fixing, bug triaging, creating artwork, donations, etc.)?

Choose activities and pick concrete tasks

Based on the user’s skills and his interests, the software shows a list of possible activities. The user can then sort that list, from the most interesting one to those that he doesn’t want to do.

Each activity can generate concrete tasks. For example, the activity “Do translation for Debian” could generate a task “Translate strings in debconf/fr.po” or “Review translations in partman/fr.po”.

Work on tasks

When the user decides to work on a task, a step-by-step assistant helps him/her. It can automate some steps and provide explanations for the remaining ones, for example in the case of a translation for Debian:

  • grab the PO file (from a VCS, from an HTTP URL, from a translation server, etc.);
  • select and install a software to work with PO file (if not already done);
  • edit the PO file with the preferred program;
  • check the PO file (is it complete? is there no mistakes like missing substitutions?);
  • send back the completed PO file in a mail to the Debian bugtracking system.

If the tasks is not completed in one go, the user can resume it the next time.

Each free software project must provide some meta-information describing the various workflows involved for contributing to the different parts of the project. If necessary the project can also provide new plugins to support new operations that are not available in the default library.

Setting goals

In order to keep the user motivated, the software could track how much time he spent contributing to free software and it could verify if the user reached the goals he picked up for himself. Maybe it can also hook into the OMG Trophy Awarding System.

The sky is the limit

I hope that you now have a clearer idea of what this desktop application is supposed to be. There are literally hundreds of ways to contribute to free software and I like the idea that we can streamline the process for most users.

All the plugins implementing activities can use local information (list of packages installed with their versions, configuration settings, etc.) to propose tasks targetted to the user and highly beneficial for the corresponding free software projects. For example, a bug tagged unreproducible might benefit from a few more users trying to reproduce it. The software could direct the user to this bug report if it detects that he/she runs the same version on the same architecture and that this software is regularly run on the system.

Many projects have created “operations” or “events” to encourage people to contribute, they could all be implemented as dedicated activities in Librement. I’m thinking of stuff like Gnome Love, Ubuntu’s 5-a-day, Ubuntu’s 100 papercuts, etc.

Even for people who have no time to contribute, the application can still be useful by referencing the various ways to donate money (or material) to projects that they are using.

Feedback welcome

I’m excited by the potential of such an application, but it’s normal since it’s my idea. Do you believe it can be useful and popular? Do you have ideas of exciting activities that such a framework can offer?

PS: If you wonder how I came up with the name “Librement”, here’s the explanation. It’s a French word which means “freely”. And users who want to give back are trying to live up to the principles of free software, which I sum up by “they are trying to live freely”.

Follow me on Identi.ca, Twitter and Facebook. Or subscribe to this blog by RSS or by email.

The secret plan behind the “3.0 (quilt)” Debian source package format

New source package formats do wondersWhile I have spent countless hours working on the new source format known as “3.0 (quilt)”, I’ve just realized that I have never blogged about its features and the reasons that lead me to work on it. Let’s fix this.

The good old “1.0″ format

Up to 2008, dpkg-source was only able to cope with a single source format (now named “1.0″). That format was used since the inception of the project. While it worked fine for most cases, it suffered from a number of limitations—mainly because it stored the Debian packaging files as a patch to apply on top of the upstream source tarball.

This patch can have two functions: creating the required files in the debian sub-directory and applying changes to the upstream sources. Over time, if the maintainer made several modifications to the upstream source code, they would end up entangled (and undocumented) in this single patch. In order to solve this problem, patch systems were created (dpatch, quilt, simple-patchsys, dbs, …) and many maintainers started using them. Each implementation is slightly different but the basic principle is always the same: store the upstream changes as multiple patches in the debian/patches/ directory and apply them at build-time (and remove them during cleanup).

Design goals for the new formats

When I started working on the new source package format, I set out to get rid of all the known limitations and to integrate a patch system in dpkg-source. I wanted to clear up the situation so that learning packaging only requires to learn one patch system and would not require modifying debian/rules to use it. I picked quilt because it was popular, came with a large set of features, and was not suffering from NIH syndrome. This lead to the “3.0 (quilt)” source format.

I also created “3.0 (native)” as a distinct format. “1.0″ was able to generate two types of source packages (native and non-native) but I did not want to continue with this mistake of mixing both in a single format. The KISS principle dictated that the user should pick the format of his choice, put it in debian/source/format and be done with it. Now the build can rightfully fail when the requirements are not met instead of doing something unexpected as a fallback.

Features of “3.0 (quilt)”

This is the format that replaces the non-native variant of the 1.0 source format. The features below are specific to the new format and differentiate it from its ancestor:

  • Supports compression formats other than gzip: bzip2, lzma, xz.
  • Can use multiple upstream tarballs.
  • Can include binary files in the debian packaging.
  • Automatically replaces the “debian” directory present in the upstream tarball (no repacking required).
  • Creates a new quilt-managed patch in debian/patches/ when it finds changes to the upstream files.

Features of “3.0 (native)”

This format is very similar to the native variant of the 1.0 source format except for two things:

  • it supports compression formats other than gzip: bzip2, lzma, xz.
  • it excludes by default a bunch of files that should usually not be part of the tarball (VCS specific files, vim backup files, etc.)

Timeline

Looking back at the history is interesting. This project already spans multiple years and is not really over until a majority of packages have switched to the new formats.

  • January 2008: the discussion how to cope with patches sanely rages on debian-devel@lists.debian.org. My initial decisions are the result of this discussion.
  • March 2008: I have implemented the new formats and I request feedback. dpkg 1.14.17 (uploaded to experimental) is the first release supporting them.
  • April 2008: I ask ftpmasters to support the new source packages in #457345.
  • June 2008: Lenny freeze. dpkg is not supposed to change anymore. Several changes concerning the new source formats are still accepted in the following months given that this code is not yet used in production and must only be present so that lenny can cope with new source packages once squeeze starts using them.
  • February 2009: Lenny release.
  • March 2009: Work on squeeze has started, ftpmasters have done nothing to support new source formats, I submit a patch in #457345 to speed things up. I start a wiki page to track the project’s progress and to answer common questions of maintainers.
  • November 2009: After an ftpmaster sprint, it’s now possible to upload new source packages in unstable. This draws massive attention to the new format and some people start complaining about some design decisions. The implementation of “3.0 (quilt)” changes a lot during this month. dpkg in lenny is even updated to keep up with those changes.
  • March 2010: Up to now, I was planning to let dpkg-source build new source packages by default at some point in the future. After several rounds of discussions, I agree that it’s not the best course of action and decides instead to make debian/source/format mandatory. The maintainer must be explicit about the source format that s/he wants to use.
  • October 2010: The new source formats are relatively popular, a third of the source packages have already switched: see the graph. The squeeze freeze in August clearly stopped the trend, hopefully it will continue once squeeze is released.
  • June 2013: Project is finished?

As you can see this project is not over yet, although the most difficult part is already behind me. For my part, the biggest lesson is that you won’t ever get enough review until your work is used within unstable. So if you have a Debian project that impacts a lot of people, make sure to organize an official review process from the start. And specifying your project through a Debian Enhancement Proposal is probably the best way to achieve this.

If you appreciate the work that I put into this project, feel free to join Flattr and to flattr dpkg from time to time. Or check out my page “Support my work“.

5 reasons why I still contribute to Debian after 12 years

Debian is packaging the free sofware world.If you’re using Debian, you know that this distribution is built entirely by volunteers that form a very diverse community. And you could be part of it. But why should you do that? I can’t tell for you but I can share my own experience. It’s been 12 years since I joined Debian and I’m going to tell you what keeps me on board.

1. Technical excellence

When facing a new challenge, Debian will strive to do the right thing. This pays off in the long term. In many cases, it means that we will take more time to implement our solution compared to other distributions out there, but this is also the reason why our packaging infrastructure allows us to offer painless upgrades and consistency across all packages.

Debian is committed to quality and builds up on his experience thanks to the Debian Policy. My time is precious, I like to spend it on something useful in the long term.

2. Inspiring goals

In its social contract, Debian has set out to create a 100% free — as in freedom — operating system. The criteria defining what constitutes a free work are listed in the Debian Free Software Guidelines (DFSG).

While the above is relatively ambitious in itself, it doesn’t inspire me much. What makes the difference to me is the emphasis given by the social contract on our users and free software. We don’t build a free operating system in the void, we build it for people.

Debian’s motto — the universal operating system — can also be interpreted in multiple ways: universal as in “for everybody on the world”, “on all kinds of computers”, or “for every possible usage”.

3. High impact work

Knowing that my work is useful to people is important, but it’s even better when I know that it will benefit to lots of people. With Ubuntu and the hundreds of other derivatives, there are nowadays literally millions of users impacted by my work. Even an insignificant one second improvement experienced by 10 millions of people represent 115 days of time saved for something else, you get the idea…

4. Working with great people

Debian has the chance to have lots of smart people on board. There’s always someone sharing valuable advice when you read Debian’s mailing lists. When I joined in 1998, I was a real newbie and I learned a lot of things by reading and interacting with people more knowledgeable than me. You can still experience the same thing nowadays but there’s one caveat: you must cope with various kinds of mailing list contributors including the “smart but uncivilized” (don’t be offended too quickly!) and the occasional troll (best ignore it, don’t feed it!).

5. Recognition of work

When you contribute to Debian, people get to know you through your contributions. It’s very rewarding to be thanked by your peers and by Debian’s users. Check out thanks.debian.net to convince you that many people are grateful for the work we put into Debian.

So that’s it for me. But what about you? What motivates you to contribute year after year, or to start contributing if you’re a prospective contributor?

Subscribe to this blog (it’s free!) by RSS, by email, or on Facebook.

Can Debian achieve world domination without being on Facebook?

Debian Developers who like Debian on FacebookFacebook is not very popular among free software hackers. When I announced my Facebook page on identi.ca (see here) I got a few replies suggesting it was odd for me to use Facebook.

Indeed there are many good reasons why Facebook should be avoided: it is a centralized and proprietary service that is not very privacy-friendly. But the truth is that lots of people are using it (even Debian developers, can you recognize them on the picture?) and some are using their Facebook news feed as their main source of news.

You might wonder how many persons that represents, so here are the figures: the Debian Facebook page has 48,361 fans and the Ubuntu one has 247,932 fans. That’s right, an announce put on the Debian facebook page would reach more persons than the most popular announce list that Debian is currently running (debian-announce@lists.debian.org has about 29,000 subscribers). Unfortunately that Debian Facebook page is empty and I don’t even know who the administrator is. The Ubuntu page on the contrary is properly configured to relay news from The Fridge and Jono Bacon is posting some custom updates from time to time.

Debian contributors regularly mention “world domination” as the ultimate goal of the universal operating system (I even feature this in my blog header banner! :-)). But working towards world domination means — in my opinion — that we should communicate our ideals of freedom to as many people as possible, even if they are using a service that we don’t want to promote.

The Debian social contract acknowledges that some users have to use non-free software and we provide the non-free section for them. In the same spirit, I believe we must have a presence on Facebook. That does not mean that we endorse Facebook, and we should surely promote Joindiaspora once it’s usable. But in the mean time we should reach out to Facebook users and allow them to follow us with the platform of their choice.

That’s the choice I recently made when I decided to setup a public facebook page featuring my free software work, my blog and my book. So if you are a Facebook user, click here to visit my page and click on “Like” if you want to follow this blog in your Facebook news feed. I also share interesting Debian or Ubuntu related articles that I discover while reading my RSS subscriptions.

Below are the usual facebook widgets for the 3 pages quoted in this article (they might not show up if you read this article through an RSS feed):


About the Debian Community Poll

While I find the idea interesting, several of the questions can’t be correctly answered because the proposed choices are not realistic or too limited.

On the question of the usage of money, I believe we should spend money to fund important projects but I don’t want to fund “people having important positions in Debian and doing important work”. What should I reply? (Granted, there’s the other item but that doesn’t help getting a clear picture of the answers)

On the question “Do you prefer time based releases instead of the «it’s ready when it’s ready» releases?”, it is putting two concepts in opposition when the release managers recently proposed a third way that combines both: “time based freezes and release when it’s ready”. This is what I want and I can’t adequately express it either in the current poll.

Open money and Debian

I recently read an introductory article in an IT magazine on the increased usage of new currencies. It briefly mentioned the work of a researcher called Arthur Brock in using those currencies within open source communities. While I did not find more details on the research, it led me to think again about Debian’s relation with money.

The reason why we have troubles using money to pay the time spent by our developers is that money is scarce and it’s thus next to impossible to be fair in the way money is spent among us. So why not invent a new currency that is not scarce and that would encourage the kind of work that we really value? Apparently there is free software out there to build new currencies: see metacurrency.org or openmoney.org. Let’s call the new currency “swirly/swirlies” for the sake of the examples below.

There are then multiple ways to create a small economy within the community and/or even create bridges with the national currencies:

  • we could have auctions (priced in swirlies) to redistribute goods among us (say I have this unused laptop and I want to give it away to someone who could make use of it within Debian)
  • Debian sponsors/partners could offer discounts codes on their products and Debian would exchange them against the Debian-specific currency (inspired by community way);
  • swirlies could be used to get funding to attend debconf and/or other meetings;
  • we could donate our swirlies as bounties on important projects that we want to see implemented, we could even grant swirlies to release managers to help them drive the project towards a release (they would not end up in their accounts but they could use them to motivate people to work on release blockers);

I’m sure we can come up with many similar ideas. Feel free to share yours in the comments.

Debian related goals for 2010

Here’s stuff that I’d like to do this year, more or less by decreasing order of importance:

  • translate my Debian book into English and get it published;
  • finish the cleanup of the perl API in dpkg-dev in order to create libdpkg-perl;
  • create dpkg-buildflags to export default build flags that packages should use (and get rid of the code setting those environment variables in dpkg-buildpackage), needed to properly fix #489771;
  • ensure the new source formats continue to gain acceptance by improving whatever is needed;
  • design a generic vcs-buildpackage infrastructure to be integrated in dpkg-dev. This design will probably happen through a DEP (Debian Enhancement Proposal) to ensure we have had proper discussion before someone gets to the implementation;
  • continue fixing dpkg bugs faster than they are reported;
  • enhance our infrastructure to ease interaction between contributors and to have a better view of how each package is maintained (see my last blog entry on this topic);
  • update the developers-reference where needed and fix some of the numerous wishlist bugs;
  • rewrite in C the last perl scripts provided by the dpkg binary package (update-alternatives/mksplit mainly, for dpkg-divert there’s a preliminary patch available already) so that it’s easier to build a minimal system without perl-base;
  • integrate the 3-way merge tool for Debian changelogs in dpkg-dev;

All of this probably doesn’t fit in my free time (being a father since last month does not help increasing my free time :-)), so if you’re interested in seeing one or more of those projects completed, and if you know some person/company that could sponsor them, get in touch with me!

5 years of Freexian

5 years ago I founded my own company Freexian SARL with the goal to make a living out of my free software experience. I marketed the company as being specialized on the Debian distribution in the hope to combine my Debian work and my professional work.

Given that Freexian is still alive I think I met the first goal. My free software experience allowed me to complete many projects: a large number of development projects for embedded devices running a custom Linux distribution (usually built with debian udebs), the development of a Debian derivative (SLIS) and some recurring tasks of remote system administration.

However, even if I use Debian daily for all my work, very few of the projects that I complete for customers have direct results in terms of improvements for Debian (except some bugreports and some related fixes). And even when I’m able to contribute something back to Debian, it’s usually not in areas that I care about.

My focus within Debian is on the technical and organizational infrastructure of the project: as a dpkg/dpkg-dev maintainer I try to improve the packaging infrastructure, as a QA member I maintain the Package Tracking System to ease collaboration, as an Alioth admin I ensure all DD can host VCS repositories for their Debian related projects, as a developers-reference co-maintainer I try to share good packaging practices, etc. Given this bias, it’s difficult to find customer projects that would let me contribute in those areas. Thus I think I need to try another approach: the simplest solution would be to find sponsors for some of my own Debian-related projects (if you have something else to suggest, please leave a comment — either in the blog or by mail).

That said finding sponsors looks like a difficult task in itself. While I can imagine (for example) a company using Debian on embedded devices that would like to sponsor the rewrite of update-alternatives in C in order to get rid of the perl dependency in the dpkg package (if you know such a company, get in touch with me!), I don’t see who would have an interest in sponsoring the time that I need to contribute new sections to the developers-reference manual. But who knows… maybe I should just try and publicly solicit sponsorship for some of the projects that I care about. In any case, suggestions and comments are welcome!