My Debian Activities in February 2012

This is my monthly summary of my Debian related activities. If you’re among the people who made a donation to support my work (384.14 €, thanks everybody!), then you can learn how I spent your money. Otherwise it’s just an interesting status update on my various projects.

Dpkg and multiarch

The month started with a decision of the technical committee which allowed me to proceed with an upload of a multiarch dpkg even if Guillem had not yet finished his review (and related changes). Given this decision, Guillem made the experimental upload himself.

I announced the availability of this test version and invited people to test it. This lead to new discussions on debian-devel.

We learned in those discussions that Guillem changed his mind about the possibility of sharing (identical) files between multiple Multi-Arch: same packages, and that he dropped that feature. But if this point of the multiarch design had been reverted, it would mean that we had to update again all library packages which had already been updated for multi-arch. The discussions mostly stalled at this point with a final note of Guillem explaining that there was a tension between convenience and doing the right things every time that we discuss far-reaching changes.

After a few weeks (and a helpful summary from Russ Allbery), Guillem said that he remained unconvinced but that he put back the feature. He also announced that he’s close to having completed the work and that he would push the remaining parts of the multiarch branch to master this week (with the 1.16.2 upload planned next week).

That’s it for the summary. Obviously I participated in the discussions but I didn’t do much besides this… I have a “mandate” to upload a multiarch dpkg to sid but I did not want to make use of it while those discussions remained pretty unconclusive. Also Guillem made it pretty clear that the multiarch implementation was “buggy”, “not right” and “not finished” and that he had reworked code fixing at least some of the issues… since he never shared that work in progress, I also had no way to help even just by reviewing what he’s doing.

We also got a few multiarch bug reports, but I couldn’t care to get them fixed since Guillem clearly held a lock on the codebase having done many private changes… it’s not quite like this that I expect to collaborate on a free software project but life is full of surprises!

I’ll be relieved once this story is over. In the mean time, I have added one new thing on my TODO list since I made a proposal to handle bin-nmu changelogs and it’s something that could also fix #440094.

Misc dpkg stuff

After a discussion with Guillem, we agreed that copyright notices should only appear in the sources and not in manual pages or --version output, both of which are translated and cause useless work to translators when updated. Guillem already had some code to do it for --version strings, and I took care of the changes for the manual pages.

I merged some minor documentation updates, fixed a bug with a missing manpage. Later I discovered that some recent changes lead to the loss of all the translated manual pages. I suggested an improvement to dh_installman to fix this (and even prepared a patch). In the end, Guillem opted for another way of installing translated manual pages.

Triggered by a discussion on debian-devel, I added a new entry to my TODO list: implementing dpkg-maintscript-helper rm_conffile_if_owner to deal with the case where a conffile is taken over by another package which might (or might not) be installed.

Misc packaging

At the start of the month, I packaged quilt 0.51. The number of Debian specific patches is slowly getting down. With version 0.51, we dropped 5 patches and introduced a new one. Later in the month I submitted 4 supplementary patches upstream which have been accepted for version 0.60.

This new version (just released, I will package it soon) is an important milestone since it’s the first version without any C code (Debian had this for a long time but we were carrying an intrusive patch for this). Upstream developer Jean Delvare worked on this and based his work on our patch, but he went further to make it much more efficient.

Besides quilt, I also uploaded dh-linktree 0.2 (minor doc update), sql-ledger 2.8.36 (new upstream version), logidee-tools 1.2.12 (minor fixes) and publican 2.8-2 (to fix release critical bug #660795).

Debian Consultants

The Debian Project Leader is working on federating Debian Companies. As the owner of Freexian SARL, I was highly interested in it since Freexian “contributes to Debian, offers support for Debian and has a strategic interest in Debian”. There’s only one problem, you need to have at least 2 Debian developers on staff but I have no employees (it’s me only). I tried to argue that I have already worked with multiple Debian developers (as contractors) when projects were too big for me alone (or when I did not have enough time). Alas this argument was not accepted.

Instead, and since our fearless leader is never afraid to propose compromises, he suggested me (and MJ Ray who argued something similar than me) to try to bring life to the Debian Consultants list which (in his mind) would be more appropriate for one-man companies like mine. I accepted to help “animate” the list, and on his side, he’s going to promote both the “Debian Companies” and the “Debian Consultants” lists.

In any case, the list has seen some traffic lately and you’re encouraged to join if you’re a freelancer offering services around Debian. The most promising thing is that James Bromberger offered to implement a real database of consultants instead of the current static page.

Book update

We made quite some progress this month. There’s only one chapter left to translate. I thus decided to start with proofreading. I made a call for volunteers and I submitted one (different) chapter to 5 proofreaders.

The liberation campaign made a nice leap forwards thanks to good coverage on barrapunto.com. We have reached 80% while we were only at 72% at the start of the month (thanks to the 113 new supporters!). There’s thus less than 5000 EUR to raise before the book gets published under a free license.

Looking at the progression in the past months, this is unlikely to be completed on time for the release of the book in April. It would be nice though… so please share the news around you.

Speaking of the book’s release, I’m slowly preparing it. Translating docbook files is not enough, I must be able to generate HTML, ePub and PDF versions of the book. I’m using Publican for most formats, but for the PDF version Publican is moving away of fop and the replacement (webkit-based) is far from being satisfactory to generate a book ready for print. So I plan to use dblatex and get Publican to support dblatex as a backend.

I have hired Benoît Guillon, the upstream author of dblatex, to fix some annoying bugs and to improve it to suit my needs for the book (some results are already in the upstream CVS repository). I’m also working with a professional book designer to get a nice design.

I have also started to look for a Python Django developer to build the website that I will use to commercialize the book. The website will have a larger goal than just this though (“helping to fund free software developers”) but in free software it’s always good to start with your own case. :-)

Hopefully everything will be ready in April. I’m working hard to meet that deadline (you might have noticed that my blog has been relatively quiet in the last month…).

Thanks

See you next month for a new summary of my activities.

My Debian Activities in January 2012

This is my monthly summary of my Debian related activities. If you’re among the people who made a donation to support my work (213.68 €, thanks everybody!), then you can learn how I spent your money. Otherwise it’s just an interesting status update on my various projects.

Dpkg

The “biggest change” I made is a small patch that brings to an end years and years of recurring discussions about the build-arch and build-indep targets of debian/rules (see #229357). Last year the technical committee took this issue in its hands (see #629385) but it failed to take any resolution. Fortunately thanks to this we got some concrete numbers on the colateral damages inflicted on the archive for each possible approach. In the end, Guillem and I managed to agree on the way forward.

The remaining of what I did as dpkg maintainer has not much to do with coding. I reviewed the work of Gianluca Ciccarelli on dpkg-maintscript-helper who is trying to provide helper functions to handle migration between directories and symlinks. I also reviewed a 2000-lines patch from Patrick Schoenfeld who’s trying to provide a perl API to parse dpkg log files and extract meaningful data out of them.

I updated the dpkg-architecture manual page to document the Makefile snippet /usr/share/dpkg/architecture.mk and to drop information that’s no longer releveant nowadays.

I reviewed a huge patch prepared by Russ Allbery to update the Debian policy and document the usage of symbols files for libraries. As the author of dpkg-gensymbols, I was keen to see it properly documented at the policy level.

I brought up for discussion a detail that was annoying me for quite some time: some copyright notices were embedded in translatable strings and updating them resulted in useless work for translators. In the end we decided to drop those notices and to keep them only at the source level.

I updated my multiarch branch on top of Guillem’s branch several times, all the fixes that were in my branch have been integrated (often in a modified form).

Unfortunately even if the code works quite well, Guillem doesn’t want to release anything to Debian until he has finished to review everything… and many people are annoyed by the unreasonable delay that it imposes. Cyril Brulebois tried to release a snapshot of the current multiarch branch to experimental but Guillem has been prompt to revert this upload.

I’m somewhat at a loss in this situation. I offered my help to Guillem multiple times but he keeps doing his work in private, he doesn’t share many details of his review except some comments in commit logs or when it affects the public interface. I complained once more of this sad situation.

Debian Package Maintenance Hub

That’s the codename I use for a new infrastructure that I would like to develop to replace the Package Tracking System and the DDPO and several other services. I started to draft a Debian Enhancement Proposal (DEP), see DEP-2, and requested some comments within the QA team.

For now, it looks like that nobody had major objections on the driving idea behind this project. Those who commented were rather enthusiastic. I will continue to improve this DEP within the QA team and at some point I will bring the discussion to a larger audience like debian-devel@lists.debian.org.

Package Tracking System

Even if I started to design its replacement, the PTS will still be used for quite some time so I implemented two new features that I deemed important: displaying a TODO notice when there is (at least) one open bug related to a release goal, displaying a notice when the package is involved in an ongoing or upcoming transition.

Misc packaging tasks

I created and uploaded the dh-linktree package which is a debhelper addon to create symlink trees (useful to replace embedded copies of PHP/JavaScript libraries by symlinks to packaged copies of those files).

I packaged quilt 0.50. I helped the upstream authors to merge a Debian patch that had been forwarded by Martin Quinson (a quilt’s co-maintainer). I packaged a security release of WordPress (3.3.1) and a new upstream release of feed2omb and gnome-shell-timer.

I prepared a new Debian release of python-django with a patch cherry-picked from the upstream SVN repository to fix the RC bug #655666.

Book update

We’re again making decent progress in the translation of the Debian Administrator’s Handbook, about 12 chapters are already translated.

The liberation campaign is also (slowly) going forward. We’re at 72% now (thanks to 63 new supporters!) while we were only at 67% at the start of January.

Thanks

See you next month for a new summary of my activities.

Debian related goals for 2012

Like last year, here’s a list of Debian related goals that I’d like to achieve this year. I might not have the time to implement all the projects, but I like to write them down to keep me motivated. And maybe it can inspire other people to implement some of them (or to help me).

  1. Finish the translation of the Debian Administrator’s Handbook

    The target is to have the book available in April. It would be nice to complete the liberation fund until then so that the book is immediately made available under a DFSG-free license.

  2. Update the Debian Administrator’s Handbook for Wheezy
  3. Translating the book in English is only the start of the journey. The real challenge is to keep the book up-to-date with each subsequent release of Debian. And Wheezy should hopefully be released in 2012 since the freeze is in June.

  4. Design and implement the Debian Package Maintenance Hub

    It’s an ambitious project that aims to merge and replace the PTS, the DDPO and their respective mail variants. It should also standardize the flow of information directed towards package maintainers. I’m going to use the DEP process to drive this project.

    This could easily take most of the year, but hopefully I’ll motivate other people to chime in and help.

  5. Implement dpkg --check-db and dpkg --repair-db

    While dpkg is fairly reliable, it’s not exempt of bugs and more annoyingly, harddrives/filesystems are not 100% reliable either, thus it happens that some internal database files get corrupted. Given that most files are text based, advanced users can manually fix them but many less skilled users are just left with a broken system that they tend to reinstall.

    To avoid this, we could provide a command that would try to automatically bring back the internal database to a sane state by looking for a working backup to restore (while at the same time marking some packages as requiring re-installation since we have some indications that they were present).

  6. Implement storage of dpkg’s internal files in Git

    This would be an extension of the former idea. Installing a package dpkg-db-history (any idea for a better name?) would setup dpkg hooks that would record every database change in a git repository. This repository could then be used to restore the last working version of the database.

Besides those concrete projects, I want to do better than last year on the topic of funding my Debian work. I will thus reiterate some objectives:

  1. Write useful articles for Debian users and Debian contributors.

    They should complete the pages Mastering Debian, Contributing to Debian 101, Debian Packaging Tutorials, and help me increase the audience of this blog.

  2. Write at least one Debian-related ebook (different from the Debian Admin Handbook) and sell it.

    It could be an ebook targetting testing users since I believe that many more users could benefit from it if they had some better knowledge of the limitations and of the way to mitigate the problems that arise from time to time.

    Or maybe it could be an ebook for people who want to start contributing to Debian, it could even be bundled with a few hours of mentoring.

  3. By the end of the year, have at least 1/3 of my time funded by donations and/or earnings of my information products.

    This means doing 3,5 times better than in 2011. It should be doable given that the sales of the Debian Administrator’s Handbook will contribute to this goal (once the translation is over).

That makes up lots of challenges for this year. Feel free subscribe to my newsletter to stay up-to-date with my progress and to get my monthly summary of the Debian/Ubuntu news. It’s also a good way to help me reach those goals since you will be informed of all my new projects.

Review of my Debian related goals for 2011

Last year I shared my “Debian related goals for 2011”. I tend to put more goals than what I can reasonably complete alone and this year was no exception. Let’s have a look.

  1. Translate my Debian book into English: PARTLY DONE
    It took more time than expected to prepare and to run the fundraising campaign but it has been successful and the translation is happening right now.
  2. Finish multiarch support in dpkg: DONE BUT NOT ENTIRELY MERGED YET
    Yes, multiarch support was already in the pipe last year in January. I completed the development between January and April (it was sponsored by Linaro) and since then it has mostly been waiting on Guillem to review it, tweak it, and integrate it.
  3. Make deb files use XZ compression by default: TRIED BUT ABANDONED
    After discussing the issue with Colin Watson and Joey Hess during debconf, I came to the conclusion that it was not really desirable at this point. The objections were that debian-installer was not ready for it and that it adds a new dependency on xz for debootstrap to work on non-Debian systems. I believe that the debian-installer side is no longer a problem since “unxz” is built in busybox-udeb (since version 1:1.19.3-2). For the other side, there’s not much to do except ensuring that xz is portable to all the other OS we care about. DAK has been updated too (see #556407).
  4. Be more reactive to review/merge dpkg patches: PARTLY DONE
    I don’t think we had any patch that received zero input. We still have a backlog of patches, and the situation is far from ideal but the situation improved.
  5. Implement the rolling distribution proposed as part of the CUT project and try to improve the release process: NOT DONE

    We had a BoF during debconf, we discussed it at length on debian-devel, but in the end we did nothing out of it. Except Josselin Mouette who wrote a proof of concept for his idea.

    For me testing is already what people are expecting from a rolling distribution. It’s just a matter of documenting how to effectively use testing, and of some marketing by defining rolling as alias to testing.

  6. Work more regularly on the developers-reference: PARTLY DONE
    I did contribute some new material to the document but not as much as I could have hoped. On the other hand, I have been rather reactive to ensure that sane patches got merged. We need more people writing new parts and updating the existing content.
  7. Write a 10-lesson course called “Smart Package Management”: NOT DONE
  8. Create an information product (most likely an ebook or an online training) and sell it on my blog: NOT DONE
    This was supposed to happen after the translation of the Debian Administrator’s Handbook. Since the translation is not yet over, I did not start to work on this yet.
  9. By the end of the year, have at least 1/3 of my time funded by donations and/or earnings of my information products: NOT REACHED
    My target was rather aggressive with 700 € each month, and given that I did not manage to complete any information product, I’m already very pleased to have achieved a mean amount of 204 € of donations each month (min: 91 €, max: 364 €). It’s more than two times better than in 2010. Thank you! Note that those figures do not take into account the revenues of the fundraising of the Debian Administrator’s Handbook since they will be used for its translation.

That makes quite a lot of red (for things that I did not achieve)… on the other hand I completed projects that I did not foresee and did not plan. For instance improving dpkg-buildflags and then merging Kees Cook work on hardened build flags was an important step for Debian. This was waiting for so long already…

My Debian Activities in December 2011

This is my monthly summary of my Debian related activities. If you’re among the people who made a donation to support my work (364.18 €, thanks everybody!), then you can learn how I spent your money. Otherwise it’s just an interesting status update on my various projects.

Dpkg and Multiarch

I had some hope to have a multiarch-enabled dpkg in sid for Christmas as Guillem told me that it was realistic. Alas Guillem got sick. We’re in January and we’re still not there.

While some of Guillem’s commits in December were related to multi-arch, the size of his pu/multiarch/master branch did not really shrink. We still have 36 commits to merge… most of the work he did was refactoring some parts of the code that were already merged. And he initiated some discussion on interface changes. I participated to those discussions hoping to bring them to a quick resolution.

I’m still maintaining my own pu/multiarch/full branch, it is based on Guillem’s branch but with further fixes that I developed and that he has not yet merged and with a change reverted (Guillem’s branch allows crossgrading packages between different architectures while dpkg does not manage this correctly yet).

I can only hope that January will be the last month of this never-ending saga. It’s been one year since I started working on this project. :-|

Misc dpkg work

I reviewed (and later merged) a patch of Kees Cook to enhance dpkg-buildflags so that it can report which hardening features are enabled. This feature might then be used by tools like lintian to detect missing hardening features.

I mentored Gianluca Ciccarelli who is trying to enhance dpkg-maintscript-helper to take care of replacing a directory by a symlink and vice-versa.

I took care of #651993 so that dpkg-mergechangelogs doesn’t fail when it encounters an invalid version in the changelog, and of #652414 so that dpkg-source --commit accepts a relative filename when a patch file is explicitly given.

Guillem also merged a fix I developed for LP#369898.

Packaging work

WordPress 3.3 came out so I immediately packaged it. Despite my upstream bug report, they did not update their GPL compliance page which offers the corresponding sources for what’s bundled in the tarball. So I hunted for the required sources myself, and bundled them in the debian.tar.xz of the Debian source package. It’s a rather crude solution but this allowed me to close the release critical bug #646729 and to reintroduce the Flash files that were dropped in the past… which is great since the Flash-based file uploader is nicer than the one using the browser’s file field.

Quilt 0.50 came out after 2 years of (slow) development. The Debian package has many patches and several of them had to be updated to cope with the new upstream release. Fortunately some of them were also merged upstream. It still took an entire morning to complete this update. I also converted the packaging from CDBS to dh with a short rules file.

Zim 0.54 came out and I immediately updated the package since it fixed a bug that was annoying me.

Review of the ledgersmb packaging

As the sql-ledger maintainer (and a user of this software for my accounting), I have been hoping to get ledgersmb packaged as a possible replacement for it. I have been following the various efforts initiated over time but none of them resulted in a real package in Debian.

This is a real pity so I tried to fix this by offering to sponsor package uploads. That’s why I did a first review of the packaging. It took several hours because you have to explain everything that’s not good enough.

I also filed a wishlist bug against lintian (#652963) to suggest that lintian should detect improper usage of dpkg-statoverride (this is a mistake that was present in the package that I reviewed).

nautilus-dropbox work

I wanted to polish the package in time for the Ubuntu LTS release and since Debian Import Freeze is in January, I implemented some of the important fixes that I wanted.

The Debian package diverges from upstream in that the non-free binaries are installed in /var/lib/dropbox/ instead of $HOME.
Due to a bug, the files were not properly root-owned so I first fixed this (unpacking the tarball as root lead to reuse of the embedded user & group information, and those information changed recently on the Dropbox side apparently).

Then we recently identified other problems related to proxy handling (see #651065). I fixed this too because it’s relatively frequent that the initial download triggered during the package configuration fails… and in that case it’s the user that will re-trigger a package download after having given the appropriate credentials through PackageKit. Without my fix, usage of pkexec would imply the loss of the http_proxy environment variable and thus it would not be possible for a user to download through a proxy.

Last but not least I reorganized the Debian specific patches to better separate what can and should be merged upstream, from the changes that upstream doesn’t want. Unfortunately Dropbox insists on being able to auto-update their non-free binaries, they are, thus, against the installation under /var/lib/dropbox and the corresponding changes.

Book update

We’re making decent progress in the translation of the Debian Administrator’s Handbook, about 6 chapters are already translated (not yet reviewed though).

The liberation campaign is also (slowly) going forward. We’re at 67% now (thanks to 90 new supporters!) while we were only at 60% at the start of December.

Thanks

See you next month for a new summary of my activities.

My Debian Activities in November 2011

This is my monthly summary of my Debian related activities. If you’re among the people who made a donation to support my work (310.73 €, thanks everybody!), then you can learn how I spent your money. Otherwise it’s just an interesting status update on my various projects.

Dpkg: Multi-Arch Saga

I know lots of people are waiting the landing of multiarch in Debian unstable, and so am I. Things are progressing, though not as quickly as I hoped. Guillem merged about half of the branch between the 24th October and the 6th of November. After that most of the work happened on his personal repository in his pu/multiarch/master branch.

I verify this repository from time to time because Guillem does not inform me when he has made progress. I noticed changes on his repository on the 10th, 19th, 23th, 28th of November and on the 1th of December.

He announced a long time ago that he had some “interface changes” and up to now only wrote about the switch from the command-line option --foreign-architecture (to put in /etc/dpkg/dpkg.cfg) to the explicit command dpkg --add-architecture that only needs to be called once (see mail here). As of today (December 2th), the promised email for the other interface changes is still not here.

On November 23th, I reviewed Guillem’s work and tried to run the code in his branch. I spent the whole day chasing up regressions and submitted lots of fixes to Guillem. Thanks to the extensive test-suite I wrote when I developed my branch, it has been fairly easy to track them all down.

All the issues I reported have been fixed in the latest version of Guillem’s branch although the fixes are often slightly different from those that I submitted.

Dpkg: Squeeze Backport

At the start of the month, I uploaded what I expected to be a fairly uncontroversial backport of dpkg 1.16.1.1. It turns out I was wrong.

After some discussion, I think we came to an agreement that it was acceptable to backport dpkg-dev and libdpkg-perl only. My goal was not to bring the latest dpkg to users but to make it easier for package maintainers to backport packages using new features provided by dpkg-dev >= 1.16 (such as hardening build flags, the makefile snippets provided in /usr/share/dpkg/, or the improved dpkg-buildflags interface).

Thus I modified the source package uploaded to squeeze-backports to build only dpkg-dev and libdpkg-perl. It has been uploaded on November 23th and it’s waiting in the NEW queue for a backports admin to process it.

Misc Dpkg Work

I merged a patch of Colin Watson to be able to verify build-dependencies for a foreign architecture (taking into account the Multi-Arch status of each package listed).

I released dpkg 1.16.1.2 with two minor fixes that were sitting in the sid branch. I wanted to get rid of this so that the path is clear for a 1.16.2 upload with multiarch. The package just migrated to testing so we’re fine.

I spent another day doing dpkg bug triaging on Launchpad, we’re now down to 77 bugs with many of them tagged as incomplete and likely to expire in 2 months.

The Debian Administrator’s Handbook

eBookWe released a sample chapter so that it’s easier to have an idea of the quality of the book. The chapter covers the APT tools quite extensively. I bet that even you could learn something about apt-get/aptitude…

The crowfunding campaign on Ulule ended on November 28th.
With 673 supporters, we raised 24345 EUR. Of those, 14935 EUR have been put in the liberation fund and the rest corresponds to the various pre-orders and rewards offered.

This means that the translation will happen (we just started) but that the book is currently not going to be released under a free license. Don’t despair… As planned, the liberation campaign is carried on until the 25 K€ target is reached!

Instead of being hosted on Ulule, this permanent campaign is on the project website at debian-handbook.info/liberation/. Note that any contribution of 10 EUR or more means that you get a copy of the ebook as soon as it’s available (even if the liberation target is not reached).

Package Tracking System

At the start of the month, I filed two ideas of improvements for the PTS in the bug tracking system: #647258 is about showing outstanding bugs that relate to a release goal and #647901 is about warning maintainers that the package is affected by a current transition. If you’re a coder and want to start contributing to Debian and its QA team, those bugs could be interesting targets for a start. :-) In both cases, I have been in contact with members of the release team because those ideas require some structured data from the release team as input. Thanks to Meddi Dohguy and Niels Thykier for their help.

Later in the month, the topic of relocating the PTS once again came up. For historical reasons, the PTS was hosted on master.debian.org together with the BTS. Nowadays the BTS has its own host and it made no sense anymore to have the PTS separate from the rest of the QA services hosted on qa.debian.org (currently quantz.debian.org). So together with Martin Zobel Helas we took care to plan the migration and on November 19th we executed the plan. It worked like a charm and almost nobody noticed (only one undocumented dependency was missed, which broke the SOAP interface).

Misc Packaging Work

WordPress was broken in Ubuntu and it was also not properly synchronized with Debian due to an almost useless change on their side. Thus I requested a sync so that the working version from Debian gets imported in Ubuntu.

I sponsored the docbook-xsl 1.76.1 upload that I needed for Publican. Then I updated Publican just to discover that the test-suite triggers a new bug in fop (filed as #649476). I disabled the test-suite temporarily and uploaded Publican 2.8 to unstable. BTW, I also filed 2 upstream bugs with patches for issues I discovered while trying to generate the sample chapter of my book (see here and here).

I uploaded a version 0.7.1 of nautilus-dropbox and fixed #648215 at the same time. I made an NMU of bison to fix a long-standing release critical bug that hit me once more during an upgrade (see #645038).

I uploaded to experimental a new version of gnome-shell-timer compatible with GNOME 3.2. I took the opportunity to install from experimental the few GNOME 3.2 packages which are not yet in unstable…

Thanks

See you next month for a new summary of my activities.

My Debian activities in October 2011

This is my monthly summary of my Debian related activities. If you’re among the people who made a donation to support my work (130.30 €, thanks everybody!), then you can learn how I spent your money. Otherwise it’s just an interesting status update on my various projects.

Dpkg work

The month started with fixing newly reported bugs to prepare the 1.16.1.1 release:

  • #644492: there was a flaw in a change I made to the trigger setup code. This resulted in packages being incorrectly marked as configured while they were only unpacked in a new chroot.
  • #642656: dpkg-source’s refusal to build when it detects unrecorded changes broke the (mostly unused, except by the lintian test suite apparently) “2.0” source format.
  • #644412: the Makefile snippet “buildflags.mk” did not respect the new maintainer specific environment variables (like DEB_CFLAGS_MAINT_APPEND) because make does not forward environment variable through $(shell …). Fixed that by manually exporting the required variables.
  • I also disabled dpkg-buildpackage’s output of the build flags since it was confusing several maintainers. dpkg-buildpackage invokes debian/rules and it has no (clean) way to discover the build flags changes that maintainer request by setting the dedicated environment variables in debian/rules. Maintainers expect to see the build flags with the modifications they have requested and not just the default values set by the distribution.

With the help of Guillem, we decided on a proper fix for a race condition sometimes triggered by parallel builds when 2 concurrent dpkg-gencontrol try to update debian/files (see #642608). This ended up requiring a new package (libfile-fcntllock-perl) that the Debian perl team kindly packaged for us. With all this sorted, it was a rather easy fix.

Multiarch progress

I also spent lots of time on multiarch. I fixed an old bug that requested to support the multi-arch paths in case of cross-building (see #595144), the discussion was not really conclusive on which of the two proposed patches was better so I ended up picking my own patch because it was closer to how we currently deal with cross-building. Then I fixed 2 issues that have been reported on Ubuntu’s dpkg. The first one (LP #863675) was rather severe since an installed package ended being “disappeared” in favor of its foreign counterpart that was removed (but that had some config files left). The second one (LP #853679) only affected dselect users (apparently there are still some!) who had a self-conflicting library (Provides: foo, Conflicts: foo) installed for multiple architectures.

But the bulk of the time spent on multiarch has been spent discussing with various parties on how to go forward with multiarch. The release team commented on the schedule of the merge to ensure it makes it into Wheezy, and the Debian project leader also commented on the problems encountered so far.

While not the best course of action I could have hoped for, it certainly helped since Guillem started pushing some reviewed commits. Out of the 66 commits that were in my pu/multiarch/full branch one week ago, 20 have been merged in the master branch already.

Python-django security update and RC bug

Since python-django’s maintainer did not manage to prepare the required security updates, I stepped in and prepared version 1.2.3-3+squeeze2 for Squeeze and 1.0.2-1+lenny3 for Lenny. Unfortunately this security update is an example of how an inactive maintainer is likely to result in a severe delay for the release of security updates.

Furthermore in this specific case, the security team did not want to release the Squeeze security update until the Lenny one had been investigated (which required some time since upstream no longer supports the version in Lenny) but they did not make this very clear.

Later another release critical bug had been filed against the package (#646634) but after investigation, it turned out to be a local configuration problem so I downgraded it. I still forwarded the test suite failure to upstream authors since the test could be enhanced.

In any case, co-maintainers for python-django are welcome. I really preferred the situation where I can quietly sit down as backup maintainer… :-)

WordPress packaging

WordPress sounds similar to python-django. I’m also “only a backup maintainer” but Giuseppe has been inactive for many months and I had to step in August because I wanted the new upstream version. I discovered a bit late that I was not subscribed to wordpress’ bugs and thus the release critical bug #639733 (that I introduced with my new upstream version) went unattended for a rather long time. Once aware, though, I quickly fixed it.

I also took the opportunity to start a discussion on debian-devel about how to deal with embedded javascript libraries and proposed a mechanism of “opportunistic replacement with symlinks”. WordPress is my testbed package for this mechanism, you can check out its debian/dh_linktree that implements the replacement logic.

The discussion has not been very interesting but at least I learned that Debian now requires that each source package shipping minified javascript files includes the original files too. It’s somewhat of a pain since it’s not a license requirement in many cases (many of those libraries are not under the GPL), but just a Debian requirement that many upstreams are not complying with. WordPress is affected and Jakub Wilk thus opened #646729 which is going to be a long-standing RC bug. To give good measures, I spent several hours investigating the case of each javascript file in the WordPress source package and I filed a new ticket on the upstream bugtracker.

Dropbox packaging work

A few months after the introduction of nautilus-dropbox to Debian and Ubuntu, I can say that the decision to only support the download of dropbox in the postinst has been a mistake. Because of this decision I had to make the postinst fail if the download failed. Even if the error message is relatively clear, this lead to many (mostly automated) bug reports on the Ubuntu side. Various other problems cropped up on top of this (trying to start dropbox while the package was not configured would result in an error because the user did not have the required rights to install the software, reinstalling the package while dropbox was running would result in a failure too, etc.).

I have fixed all those issues in the version 0.7.0-2 of the package. Now if the user has to install dropbox, it will use PolicyKit to request the root rights. The postinst will no longer fail if the dropbox download fails since it can be run later by the user. And I fixed the download code to remove the replaced file before unpacking a new file (insead of overwriting the existing file). All this work has been forwarded upstream.

The Debian Administrator’s Handbook Update

I’m glad to tell you that the translation will happen because we reached the minimal funding goal on October 22th with the help of 380 supporters.

Now the fundraising continues, but this time the goal is the liberation of the resulting book. For this to happen, we need to reach 25000 EUR in the liberation fund. So far we’re at 37% of this goal with 9400 EUR in the liberation fund (which means that 59% of the money raised has been put in the liberation fund).

Click here if you want to contribute towards the liberation of this book.

With (less than) 27 days left, it’s going to be a challenge to meet the goal, but we do like challenges, don’t we?

Misc work

  • I filed #644486 against dh-make so that new packages have proper support of dpkg-buildflags from the start.
  • I merged lots of patches from Luca Falavigna in the developers-reference.
  • I discussed debtags integration in the PTS with Enrico Zini and Paul Wise.
  • I updated publican’s packaging for the new upstream version 2.8. I had to write a new patch that I forwarded upstream.
  • I filed an upstream bug on hamster-applet because just running hamster-time-tracker no longer brings its window forward.

Thanks

See you next month for a new summary of my activities.

My Debian activities in September 2011

This is my monthly summary of my Debian related activities. If you’re among the people who made a donation to support my work (144.3 €, thanks everybody!), then you can learn how I spent your money. Otherwise it’s just an interesting status update on my various projects.

Dpkg work

While taking care of the last details for the hardening feature in dpkg 1.16.1, I have mailed debian-devel to find volunteers to handle a hardening release goal. The objective is to ensure a large number of packages have been converted/rebuilt to actually use the new hardening build flags.

Then I prepared the draft of the announce of the dpkg 1.16.1 upload (aka Bits of dpkg maintainers sent to debian-devel-announce) which got expanded by Guillem to also cover new features since dpkg 1.15.7.

update-alternatives got some refactoring by Guillem which resulted in a regression that has been fortunately discovered by Sven Joachim. I fixed that regression and did some further cleanup inspired by the root cause of this regression (see top 4 commits here).

Note that Sven is one of the few persons who are running the git version of dpkg. Hopefully the number of tester will increase since I recently documented the APT repositories with autobuilt versions of dpkg in the wiki.

At the end of the month, I started working on a bugfix release (what’s going to be 1.16.1.1) by fixing some of the unavoidable problems discovered after an upload that accumulated more than 4 months worth of work (see top 4 commits here).

The Debian Administrator’s Handbook

I spent countless hours finalizing the launch of the crowdfunding campaign for the Debian Administrator’s Handbook and it went live on September 27th.

So far it’s on good track with more than 63% of the base funding already secured. But we still have a long way to go to reach the liberation goal (we’re at 21%). It’s still worth nothing that more than 55% of the money raised has been put in the liberation fund so there are many persons who care about getting the book freed.

More than 250 persons are supporting the project currently with an average contribution of 38 EUR. I would have expected much less for the average contribution but many more supporters. I still hope we can get more people on board with the perspective of a good DFSG-free Debian ebook.

Did you order your copy? If not, click here and fix this! ;-) By the way Paypal used to be required but it’s no longer the case, you can support the project just with your usual credit card.

Misc blog updates

Over time, I have written many useful articles for Debian users and Debian contributors. But scattered in the history, they are somewhat difficult to find. To fix this I have created some index pages listing them. Check them out:

Two new articles joined those pages this month: How to triage bugs in the Debian Bug Tracking System and Understand dpkg and don’t get stuck with a maintainer script failure.

While writing the first article, I noticed we lacked a good page showing the most buggy packages so I quickly created it (with the help of UDD): http://qa.debian.org/cgi-bin/bugs-by-source

Misc packaging work

I did a small update to the developer’s reference. Luca Falavigna submitted a patch to clarify how one is supposed to deal with meta-packages (cf #569219), I improved it and integrated the result in the SVN repository.

I upgraded nautilus-dropbox to version 0.6.9 and while doing this I discovered a bug in mergechanges (filed as #640782). I uploaded a new release of quilt mainly to add the Multi-Arch: foreign field so that it can satisfy dependencies of foreign packages (i.e. packages of a different architecture).

Django released some security advisories (tracked in #641405) and since the maintainer did not deal with the issue, I stepped up to the task (I’m a backup maintainer) and released the fixed version 1.3.1 to unstable. I took the opportunity to switch from python-support to dh_python2, and do some misc improvements to the packaging (see changelog).

I wanted to update publican to a newer version but it turned out to be not possible because Debian doesn’t have the latest version of docbook-xsl yet. I also discovered some bugs in the test suite and forwarded upstream the patch I created (see upstream bug). On top of this, fop was failing due to some java problem related to the introduction of multiarch. After having reported the bug, the java maintainers quickly released a fixed version.

So now publican is ready in the git repository but it’s waiting on the docbook-xsl update. I got in touch with the maintainer who said he would have the time to take care of it by mid-october.

Thanks

See you next month for a new summary of my activities.

My Debian activities in August 2011

This is my monthly summary of my Debian related activities. If you’re among the people who made a donation to support my work (91.44 €, thanks everybody!), then you can learn how I spent your money. Otherwise it’s just an interesting status update on my various projects.

Dpkg work

When I came back from Debconf, I merged my implementation of dpkg-source --commit (already presented last month). I continued some work on the hardening build flags but it’s currently stalled waiting on Kees Cook to provide the required documentation to integrate in dpkg-buildflags(1).

Following a discussion held during DebConf, Michael Prokop has been kind enough to setup a git-triggered auto-builder of dpkg (using Jenkins). You can now help us by testing the latest git version. Follow those instructions:

$ wget -O - http://jenkins.grml.org/debian/C525F56752D4A654.asc | sudo apt-key add -
$ sudo sponge /etc/apt/sources.list.d/dpkg-git <<END
deb http://jenkins.grml.org/debian dpkg main
END
$ sudo apt-get update && sudo apt-get upgrade

On the bug fixing side I took care of #640198 (minor man page update), #638291 (a fix to correctly handle hardlinks of conffiles), #637564 (the simplification logic of union dependencies was broken in some cases) and #631494 (interrupting dpkg-source while building a native source package left some temporary files around that should have been cleaned).

WordPress update

I released WordPress 3.2.1 in unstable (after having taken the time to test the updated package on my blog!) and fixed its RC bug (#625773). In the process I discovered a false positive in lintian (I reported it in 637473).

Gnome-shell-timer package

From time to time, I like to use the Pomodoro Technique. That’s why I was an user of timer-applet in GNOME 2. Now with the switch to GNOME 3, I lost this feature. But I recently discovered gnome-shell-timer, a GNOME Shell extension that provides the same features.

I created a Debian package of it and quickly filed some bugs while I was testing it (two usability issues and an encoding problem)

QA Work

During DebConf I met Giovanni Mascellani and he was interested to help the QA team. He started working on the backlog of bugs concerning the Package Tracking System (PTS) and submitted a bunch of patches. I reviewed them and merged them but since they were good, I quickly got lazy and got him added to the QA team so that he can commit his fixes alone. It also helps to build trust when you have had the opportunity to discuss face to face. :-)

Vacation

That’s not so much compared to usual but to my defense I also took 2 weeks of vacation with my family. But somehow even in vacation I can’t really forget Debian. Here’s my son:

Thanks

See you next month for a new summary of my activities.

My Debian activities in July 2011

This is my monthly summary of my Debian related activities. If you’re among the people who made a donation to support my work (170 €, thanks everybody!), then you can learn how I spent your money. Otherwise it’s just an interesting status update on my various projects.

This month passed by very quickly since I attended both the Libre Software Meeting / RMLL and the DebConf.

Libre Software Meeting / RMLL

I attended “only” 3 days out of the 6 but that was a deliberate choice since I was also attending DebConf for a full week later in the month.

During those 3 days I helped with the Debian booth that was already well taken care of by Frédéric Perrenot and Arnaud Gambonnet. Unfortunately we did not have any goodies to sell. We (as in Debian France) should do better in this regard next time.

One of the talks I attended presented EnVenteLibre. This website started as an online shop for two French associations (Ubuntu-fr, Framasoft). They externalize all the logistic to a company and only have to care about ordering goodies and delivering to the warehouse of the logistic company. They can also take some goodies from the warehouse and ship them for a conference, etc. We discussed a bit to see how Debian France could join, they are even ready to study what can be done to operate at the international level (that would be interesting for Debian with all the local associations that we have throughout the world).

Back to the LSM, while I had 3 good days in Strasbourg, it seems to mee that the event is slowly fading out… it’s far from being an international event and the number of talks doesn’t make for a better quality.

BTW, do you remember that Debconf 0 and Debconf 1 were associated to this event while it was in Bordeaux?

dpkg-source improvements

During my time in Strasbourg (and in particular the travel to go there and back!) I implemented some changes to “3.0 (quilt)” source format. It will now fail to build the source package if there are upstream changes that are not properly recorded in a quilt patch:

dpkg-source: info: local changes detected, the modified files are:
 2ping-1.1/README
dpkg-source: info: you can integrate the local changes with dpkg-source --commit
dpkg-source: error: aborting due to unexpected upstream changes, see /tmp/2ping_1.1-1.diff.cki8YB

As the error message hints, there’s a new --commit command supported by dpkg-source that will generate the required quilt patch to fix this. In the process you will have to submit a name and edit the patch header (pre-formatted with DEP3 compatible fields). You can get back the old behavior with the --auto-commit option.

Build flags changes

Ever since we adopted the Ubuntu changes to let dpkg-buildpackage set some build related environment variables (see #465282), many Debian people expressed their concerns with this approach both because it broke some packages and because those variables are not set if you execute debian/rules directly.

In the end, the change was not quickly reverted and we fixed the package that this change broke. Despite this we later decided that the correct approach to inject build flags would be a new interface: dpkg-buildflags.

Before changing dpkg-buildpackage to no longer set the compilation flags, I wanted to ensure dpkg-buildflags had some decent coverage in the archive (to avoid breaking too many packages again). My criteria was that CDBS and dh (of debhelper) should be using it. With the recent debhelper change (see #544844) this has been reached so I changed dpkg-buildpackage accordingly.

Makefile snippets provided by dpkg

At the same time, I also wanted an easy way for maintainers not using dh or CDBS to be able to fix their package easily and go back to injecting the compilation flags in the environment but doing it from the rules files. Starting with the next version of dpkg, this will be possible with something like this:

DPKG_EXPORT_BUILDFLAGS = 1
include /usr/share/dpkg/default.mk

Without DPKG_EXPORT_BUILDFLAGS the variables are not exported in the environment and have no effect unless you use them somewhere.

More than build flags, this will also provide a bunch of other variables that can be useful in a rules files: all the variables provided by dpkg-architecture, vendor related variables/macro and some basic package information (mainly version related).

dpkg-buildflags improvements

Given the renewed importance that dpkg-buildflags will take now that dpkg-buildpackage no longer sets the corresponding environment variables, I thought that I could give it some love by fixing all the open issues and implementing some suggestions I got.

I also had a chat with a few members of the technical committee to discuss how hardening build flags could be enabled in Debian and this also resulted in a few ideas of improvements.

In the end, here are the main changes implemented:

  • new “prepend” directive to inject flags at the start (see commit);
  • new “strip” directive to strip flags from the result returned by dpkg-buildflags (see commit);
  • new environment variables DEB_flag_MAINT_directive that can be set by the maintainer to adjust what dpkg-buildflags will return (see commit);
  • new --export=configure command to inject build flags on the ./configure command line (see commit);
  • new --dump command that is the default (see #603435).

Will all those changes, the complete set of compilation flags can be returned by dpkg-buildflags (before it would only return the default flags and it was expected that the Debian packaging would add whatever else is required afterwards). Now the maintainer just has to use the new environment variables to ensure the returned values correspond to what the package needs.

DebConf: rolling and hardening build flags

I spent a full week in DebConf (from Sunday 24th to Sunday 31th) and as usual, it’s been a pleasure to meet again all my Debian friends. It’s always difficult to find a good balance between attending talks, working in the hacklab and socializing but I’m pretty happy with the result.

I did not have any goal when I arrived, except managing the Rolling Bof (slides and video here) but all the discussions during talks always lead to a growing TODO list. This year was no exception. The technical committee BoF resulted in some discussions of some of the pending issues, in particular one that interests me: how to enable hardening build flags in Debian (see #552688).

We scheduled another discussion on the topic for Tuesday and the outcome is that dpkg-buildflags is the proper interface to inject hardening build flags provided that it offers a mean to drop unwanted flags and a practical way to inject them in the ./configure command line.

Given this I got to work and implemented those new features and worked with Kees Cook to prepare a patch that enables the hardening build flags by default. It’s not ready to be merged but it’s working already (see my last update in the bug log).

A few words about the Rolling BoF too. The room was pretty crowded: as usual the topic generates lots of interest. My goal with the BoF was very limited, I wanted to weigh the importance of the various opinions expressed in the last gigantic discussion on debian-devel.

It turns out a vast majority of attendants believe that testing is already usable. But when you ask them if we must advertise it more, answers are relatively mixed. When asked if we can sustain lots of testing/rolling users, few people feel qualified to reply but those that do tend to say yes.

More dpkg work

Lots of small things done:

  • I did again some bug triaging on Launchpad. But Brian Murray did a lot of it and the result is impressive, we’re down to 154 bugs (from more than 300 a month ago!).
  • I updated my multiarch branch multiple times. I was hoping to meet Guillem during DebConf to make some progress on this front but alas he did not attend. I have been asked a status update multiple times during my time in DebConf.
  • I fixed a regression in update-alternatives (#633627), a test-suite failure when run as root (#634961), a segfault in findbreakcycle. There have been a bunch of minor improvements too (#634510, #633539, #608260, #632937).

Package Tracking System and DEHS

Christoph Berg recently wrote a replacement for DEHS because the latter was not really reliable and not under control of the QA team. This is a centralized system that uses the watch files to detect new upstream versions of the software available in Debian.

I updated the Package Tracking System to use this new tool instead of DEHS. The new thing works well but we’re still lacking the mail notifications that DEHS used to send out. If someone wants to contribute it, that would be great!

Misc packaging work

I did some preliminary work to update the WordPress package to the latest upstream version (3.2). I still have to test the resulting package, replacing upstream shipped copies of javascript/PHP libraries is always a risk and unfortunately all of them had some changes in the integration process.

I also updated nautilus-dropbox to version 0.6.8 released upstream. I also uploaded the previous version (that was in testing at that time) to squeeze-backports. So there’s now an official package in all the Debian distributions (Squeeze, Wheezy, Sid and Experimental)!

Thanks

See you next month for a new summary of my activities.