Contributing

Looking back at 16 years of dpkg history with some figures

August 13, 2012 by Raphaël Hertzog

With Debian’s 19th anniversary approaching, I thought it would be nice to look back at dpkg’s history. After all, it’s one of the key components of any Debian system.

The figures in this article are all based on dpkg’s git repository (as of today, commit 9a06920). While the git repository doesn’t have all the history, we tried to integrate as much as possible when we created it in 2007. We have data going back to April 1996…

In this period between April 1996 and August 2012:

146 persons contributed to dpkg (result of git log --pretty='%aN'|sort -u|wc -l)
6948 commits have been made (result of git log --oneline | wc -l)
3133612 lines have been written/modified (result of git log --stat|perl -ne 'END { print $c } $c += $1 if /(\d+) insertions/;')

Currently the dpkg source tree contains 28303 lines of C, 14956 lines of Perl and 6984 lines of shell (figures generated by David A. Wheeler’s ‘SLOCCount’) and is translated in 40 languages (but very few languages managed to translate everything, with all the manual pages there are 3997 strings to translate).

The top 5 contributors of all times (in number of commits) is the following (result of git log --pretty='%aN'|sort| uniq -c|sort -k1 -n -r|head -n 5):

Guillem Jover with 2663 commits
Raphaël Hertzog with 993 commits
Wichert Akkerman with 682 commits
Christian Perrier with 368 commits
Adam Heath with 342 commits

I would like to point out that those statistics are not entirely representative as people like Ian Jackson (the original author of dpkg’s C reimplementation) or Scott James Remnant were important contributors in parts of the history that were recreated by importing tarballs. Each tarball counts for a single commit but usually bundles much more than one change. Also each contributor has its own habits in terms of crafting a work in multiple commits.

Last but not least, I have generated this 3 minutes gource visualization of dpkg git’s history (I used Planet’s head pictures for dpkg maintainers where I could find it).

Watching this video made me realize that I have been contributing to dpkg for 5 years already. I’m looking forward to the next 5 years 🙂

And what about you? You could be the 147th contributor… see this wiki page to learn more about the team and to start contributing.

Contributing to the translation of Debian

January 31, 2012 by Raphaël Hertzog

If you’re not into packaging and if you asked how you could help Debian, someone probably suggested that you help to translate it.

It’s true that translating Debian is essential if we want to make Debian available to everybody on the world. There are many persons who are stuck as soon as they get a message in English, so it’s important to aim for 100% coverage in terms of localization.

Some vocabulary: localization vs internationalization

Internationalization (i18n) is the work that makes it possible to translate messages in a given application.

Localization (l10n) is the work of translating messages of said application. So as a translator, you’ll be doing “localization” but some knowledge of “internationalization” is still useful… because it will define how you’re supposed to provide the translations. We’ll come back to that later.

Join your localization team

Usually the translation work is shared among multiple translators within a localization team. Check out the Debian International page on www.debian.org to find out instructions for translators for each language.

Many teams have a debian-l10n-*@lists.debian.org mailing list used for coordination, feel free to ask questions on those lists when you start (but make sure that you have read the relevant documentation before).

Each team has its own workflow, so observe for a while to get used to what’s happening before asking your first questions.

What is there to translate?

The translation of most of the software provided by Debian is not handled by Debian. The Debian translation teams “only” handle the translation of:

the software that are specific to Debian (debian-installer, dpkg, APT, etc.) (*);
the Debconf prompts in all Debian packages (*);
the Debian documentation (*);
the Debian website;
the Debian wiki;
the descriptions of packages.

Now before contributing to your first translation, I have to come back to internationalization to teach you a few things. In the above list, the projects marked with “(*)” do use PO files for their translation and the next sections will explain you how to work with those files.

Introduction to Gettext

The free software community has mostly standardized on a single internationalization infrastructure known as Gettext. With this tool, you’re provided a “POT file” which contains all the translatable strings. It looks like this:

# SOME DESCRIPTIVE TITLE.
# Copyright (C) YEAR Software in the Public Interest, Inc.
# This file is distributed under the same license as the PACKAGE package.
# FIRST AUTHOR , YEAR.
#
#, fuzzy
msgid ""
msgstr ""
"Project-Id-Version: dpkg 1.16.1\n"
"Report-Msgid-Bugs-To: debian-dpkg@lists.debian.org\n"
"POT-Creation-Date: 2011-09-23 03:37+0200\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: FULL NAME \n"
"Language-Team: LANGUAGE \n"
"Language: \n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Plural-Forms: nplurals=INTEGER; plural=EXPRESSION;\n"

#: lib/dpkg/ar.c:66
#, c-format
msgid "invalid character '%c' in archive '%.250s' member '%.16s' size"
msgstr ""

#: lib/dpkg/ar.c:81 lib/dpkg/ar.c:97 lib/dpkg/ar.c:108 lib/dpkg/ar.c:112
#: lib/dpkg/ar.c:134 utils/update-alternatives.c:1154
#, c-format
msgid "unable to write file '%s'"
msgstr ""

[…]

The lines starting with “#:” are comments that indicate the source files where the (English) string is used. This can be useful if you want check the source to have more information about how the string is used.

The lines starting with “#,” contain flags that can be important. If the “fuzzy” flag is set, the translated string is not used because it must be updated (or at least verified) since the original string evolved. The “c-format” flags indicates that the string must be a C format string, this has some implications in what’s allowed in the string (in particular when it embeds conversion specifier for arguments submitted to printf-like functions).

Another thing to note is that the translation of the empty string is used to store some meta-information about the translation itself.

Contributing a translation as a PO file

When you start a new translation, you copy that POT file to create a “PO file” for your own language (eg. fr.po for the French language). You replace some template values (identified with the upper case words in the POT file) and you replace all the empty strings on “msgstr” lines with the translation of the string that appears in the previous “msgid” line.

The result could be something like this:

# translation of fr.po to French
# Messages français pour dpkg (Linux-GNU Debian).
msgid ""
msgstr ""
"Project-Id-Version: fr\n"
"Report-Msgid-Bugs-To: debian-dpkg@lists.debian.org\n"
"POT-Creation-Date: 2011-09-23 03:37+0200\n"
"PO-Revision-Date: 2012-01-16 07:57+0100\n"
"Last-Translator: Christian Perrier \n"
"Language-Team: French \n"
"Language: fr\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Plural-Forms: Plural-Forms: nplurals=2; plural=n>1;\n"
"X-Generator: Lokalize 1.2\n"

#: lib/dpkg/ar.c:66
#, c-format
msgid "invalid character '%c' in archive '%.250s' member '%.16s' size"
msgstr "caractère invalide « %1$c » dans la taille du membre « %3$.16s » de l'archive « %2$.250s »"

#: lib/dpkg/ar.c:81 lib/dpkg/ar.c:97 lib/dpkg/ar.c:108 lib/dpkg/ar.c:112
#: lib/dpkg/ar.c:134 utils/update-alternatives.c:1154
#, c-format
msgid "unable to write file '%s'"
msgstr "impossible d'écrire le fichier « %s »"

[…]

If there’s already a “PO file” for your language, there might still work to do: there might be strings that have not yet been translated and there might be “fuzzy” strings which have to be updated — strings which were already translated but where the original string has been modified.

There are software that can assist you to edit PO files: poedit, virtaal, lokalize, gtranslator. There are also special extensions for vim (packaged in vim-scripts) and for Emacs.

Submit the translation for inclusion

Once you have a complete PO file, you should submit it for inclusion. Sometimes you will have been granted commit rights to the source code repository so that you can include your translation by yourself. In the other cases, you should submit your translation with a bug report tagged “l10n” and someone else will include your work in the next release.

Depending on the team, the workflow might require a review before the submission. In that case, you usually have to send a call for review on the coordination mailing list.

Go ahead!

Hopefully those explanations will be enough to get you started. There are many other things to learn¹ but it’s good to learn while practicing…

¹ For example, can you find out why the French translation above changed “%c” in “%1$c”?

Do you want to read more tutorials like this one? Click here to subscribe to my free newsletter, you can opt to receive future articles by email.

How to prepare patches for Debian packages

July 4, 2011 by Raphaël Hertzog

You want to start contributing to Debian and/or Ubuntu, you decided to help a package maintainer and you’re now looking for how to change a source package and how to submit your changes.

1. Retrieve the source package and install build-dependencies

The first step is to retrieve the latest version of the source package and to install the required build-dependencies. I already covered how to do this with apt-get in the article explaining how to rebuild a source package.

If you prefer, you can use dget (from the devscripts package) to directly grab the source package. You can find the URL of the .dsc files in the Package Tracking System for example.

Sometimes apt-get will warn you that the source package is maintained in a VCS repository, like this:

$ apt-get source wordpress
[...]
NOTICE: 'wordpress' packaging is maintained in the 'Git' version control system at:
git://git.debian.org/git/collab-maint/wordpress.git
[...]

In that case, you can use debcheckout to retrieve the VCS repository instead (provided that you have the corresponding VCS installed):

$ debcheckout wordpress
declared git repository at git://git.debian.org/git/collab-maint/wordpress.git
git clone git://git.debian.org/git/collab-maint/wordpress.git wordpress ...
Cloning into wordpress...

Note however that some maintainers use their VCS in a way that’s not really compatible with the explanations that I will give below.

It’s also a good idea to install the package “packaging-dev”. It’s a meta-package depending on the most common tools that are used for Debian packaging work.

2. Do the changes

Execute dch --nmu to record the fact that you’re working on an update prepared by someone who is not the maintainer (NMU means Non Maintainer Upload). This also ensures that if we build the package, we won’t overwrite the original source package that we downloaded, thus making it possible to generate a “diff” between both versions.

2.1. Modify Debian packaging files

Now fire your text editor and do the required changes in the “debian” sub-directory. You will probably run dch -a multiple times to document each subsequent change.

2.2. Modify upstream files

If you have to modify upstream files, the proper way to do it depends on the source package format (“1.0” vs “3.0 (quilt)” vs “3.0 (native)”, see the debian/source/format file) and on the presence or not of a patch system (the what-patch can help you identify it). In this explanation, I’ll assume that the package is using the recommended format: “3.0 (quilt)”. (It also works for “1.0” if quilt is used and if you configured ~/.quiltrc as recommended by /usr/share/doc/quilt/README.source).

First you should ensure that all patches have been applied with quilt push -a. If there’s no patch yet, you want to create the debian/patches directory (with mkdir debian/patches). Note that you should better invoke quilt from the root of the source package (and the examples below assume this).

2.2.1. Import a patch

If the upstream changes are already in a patch file (say /tmp/patch that you downloaded from the upstream VCS repository) you can import that patch like this:

$ quilt import -P fix-foobar.patch /tmp/patch
Importing patch /tmp/patch (stored as fix-foobar.patch)
$ quilt push
Applying patch fix-foobar.patch
[...]
Now at patch fix-foobar.patch

The -P option allows to select the name of the patch file created in debian/patches/. As you see, the new patch file is recorded in debian/patches/series but not applied by default, we’re thus doing it with quilt push.

2.2.1. Create a new patch

If the upstream changes that you want to make are not in a patch yet, you should tell quilt that you’re going to create one:

$ quilt new fix-foobar.patch
Patch fix-foobar.patch is now on top

Then you must record every file that you’re going to modify with a quilt add invocation. quilt then makes a backup of those files so that it can generate the patch later on. If you’re going to modify the files with your text editor you can just do quilt edit file-to-modify, it’s the same than quilt add file-to-modify followed by sensible-editor file-to-modify.

$ quilt edit foobar.c
File foobar.c added to patch fix-foobar.patch

The last step is tell quilt to generate the patch:

$ quilt refresh
Refreshed patch fix-foobar.patch

3. Test your changes

You should build your modified package with “debuild -us -uc”. You can easily install the resulting package with “debi”. Verify that everything works as expected. If not, continue your modifications until you’re satisfied with the result.

4. Generate a patch and mail it

If you followed the instructions, you should have two .dsc files in the parent directory, like this:

$ cd ..
$ ls wordpress_*.dsc
../wordpress_3.0.5+dfsg-1.1.dsc
../wordpress_3.0.5+dfsg-1.dsc

Generating the patch to send to the maintainer is then just a matter of running debdiff:

$ debdiff wordpress_3.0.5+dfsg-1.dsc wordpress_3.0.5+dfsg-1.1.dsc >/tmp/wp-debdiff

You can send the /tmp/wp-debdiff file to the wordpress maintainer. Usually you send it via the bugreport that your update is fixing and you add the “patch” tag to the report.

This can be automated with the nmudiff utility. By default it assumes that you’re using mutt but it can also directly feed the resulting mail to sendmail. The default text that nmudiff proposes assumes that you’re actually performing an NMU and that the result has been uploaded. If that’s not the case, you should edit the text and make it clear that you’re just sending a patch.

If you have been working in a VCS repository, instead of using debdiff you can simply use the diff feature integrated in your VCS (git diff, svn diff, etc.). But note that with a distributed VCS (like git/bzr/mercurial, unlike svn) you should probably have committed all individual changes in separate changesets. And instead of sending a single patch, you’re probably going to send a series of patches (though it might be easier to just upload your branch in a public repository and give the corresponding URL to the maintainer).

Found it useful? Be sure to not miss other packaging tips (or lessons), click here to subscribe to my free newsletter and get new articles by email.

How to start contributing to Debian?

June 30, 2011 by Raphaël Hertzog

I often get requests of persons who would like to contribute to Debian but who don’t know where to start. Let’s try to answer this question properly so that I can give out this URL the next time that I am asked.

The Debian website has a page explaining how to help Debian. While it provides no less than 10 suggestions in a daunting text-only list, it’s difficult to know what to do next once you picked up something that you could do.

I will try to fix this by providing concrete information for each cases in upcoming articles but in the mean time I propose you another approach to start with. Before answering your question (“what can I do for Debian”), we need to know some information about you.

What motivates you?

You’re a volunteer, you’re not doing stuff for Debian because someone told you so. You must have some intrinsic motivation and the ultimate motivation is usually that you’re enjoying what you’re doing.

So what are you enjoying and/or what are your motivations ?

Is there something that you would like to learn? A new programming language? Packaging? Coding? System administration? A specific software?
Do you want to interact with smart people?
Do you like to help users?
Do you like to fix software just so that it works for you?
Do you like to build something remarkable and useful for millions of people?

On the opposite, make sure to know what you hate and what you want to stay far away from. Maybe you dislike a programming language so much that you don’t want to be involved in a project where you would have to use it, etc.

Write down the answers to the questions, you might need them later when you’ll ask other Debian contributors how you can help.

What are your skills?

If you’re not interested in learning new skills, then you must obviously select a task where your current skills are sufficient. Again make a list of your skills and in particular of skills that you’d like to practice! Here’s a non-exhaustive list of skills to consider:

What languages are you fluent with? Are you confident to write documentation or translate documentation in those languages?
Are you a programmer? If yes, which languages do you know?
Can you diagnose problems? Can you debug problems with strace and/or gdb?
Can you triage bugs?
Do you know some Debian packaging?
Are you an artist and/or a web designer?
Do you know how to work with VCS (subversion, git, bzr, …)?

How much time can you spend on Debian?

This is the last important information that you need to communicate whenever you’re asking someone else what you could do for Debian. There’s no point giving you a big task if you can only spend 30 minutes every week. On the opposite if you can work on Debian full time during a week (because you’re between 2 contracts or because you’re in vacation), it’s equally important to know.

In general contributing to Debian requires time, you should be ready to spend at least several hours per week and possibly more at the start while you’re learning everything.

Find something to do

At this point you have a generic idea of what you’d like to do but you’re still missing a concrete objective. Let’s try to find one, we’ll explore several ways to do this.

Scratch your itch

The best objectives are those that satisfy your own needs. Here are some examples:

Did you notice a missing feature? Try to implement it.
Have you been annoyed by a bug? Try to fix it.
Did you lose too much time on something because there was no documentation? Write the missing documentation and submit it where appropriate.
File bug reports for the things that you can’t fix yourself. Even wishlist bug reports for new features.
Do you use software that are not packaged for Debian? Create the package(s) and maintain it/them.
Do you need a newer version of a package compared to what’s in Debian unstable? Contact the maintainer and propose your help to update the package.
Do you need a newer version of a package compared to what’s in Debian stable? Contact the maintainer and propose your help to create a backport.

If you’re a good Debian citizen, you have already filed bugs for issues that bugged you. Then you can browse http://bugs.debian.org/from:hertzog@debian.org to find out some ideas of stuff to do (obviously replace hertzog@debian.org by your own email address).

Join a team

If you don’t have a specific itch to scratch, you might want to focus your work on a specific team. Head over to wiki.debian.org/Teams and browse the list of teams.

You’ll surely find one that works in an area that you like. If you select a packaging team, pick one that works on packages that you’re actually using.

Some of the teams have instructions for newcomers, follow them when that is the case. Otherwise join the mailing list and the IRC channel, and get a feel of how the team works. See if it suits you, you can follow several teams at the same time and pick the one that you prefer after a few days/weeks.

Once you have lurked a bit, if you still don’t know how you could help, then just ask on the mailing list. Include all the answers you have collected to the 3 questions on your motivations, your skills and your time available.

Focus on a specific package

You can concentrate your work on a specific package even if you’re in a team, it’s often a good idea. But I list it separately because not all packages are team maintained and you might want to help maintain a package where there’s a single maintainer currently. I leave it up to you to find a way to select a package that interests you…

Then head over to the package tracking system: http://packages.qa.debian.org/dpkg (replace dpkg by the name of the package that interests you)

Fill in the form in the bottom-left corner with your email and select “opts” in the drop-down list, then click “go”. You get a new form where you can select the information that you’ll receive, I recommend you to keep everything except “upload-binary” and to validate the form.

From now on, you get the same mails than the maintainer (and a bit more actually), and it’s a good idea to inform the maintainer that you subscribed and that you’re going to help a bit. Maybe he’s willing to grant you commit rights immediately, maybe he will ask you to send patches for a start. The important thing is to create a good relationship. In any case (even if you did not get any answer from the maintainer), you should be free to help triage existing bugs and to help deal with the flow of incoming bugs (including forwarding bugs when appropriate).

Help a Debian developer

Paul Tagliamonte once blogged Hey, DDs, need help?. He offered his help to “overworked Debian developers”. His post was missing all the information required (cf motivations/skills/time) but the approach is a good one.

The best way to start contributing is to work with existing Debian developers. Even if you “just” want to be sponsored for your own pet package, you should consider that mentoring is a burden for many Debian developers and that you’re more likely to get a sponsor if you have an existing relationship with a Debian developer that you helped. Pick a developer that works in an area that is of interest, and offer him your help.

To simplify things even further, I have created a wiki page where you can find out how you can help me and thus build a relationship with me.

That’s it for now, I hope this article will help you to start contributing. Don’t forget to subscribe to my newsletter to not miss future articles for new contributors. Note that you can refer to this article with the following URL: https://raphaelhertzog.com/go/contributing/ (easier to type and remember).

PS: You might want to also check my Contributing to Debian page.