Freexian’s fourth report about Debian Long Term Support

Like each month, here comes a report about the work of paid contributors to Debian LTS.

Individual reports

In November 42.5 work hours have been equally split among 3 paid contributors. Their reports are available:

  • Thorsten Alteholz did his share as usual.
  • Raphaël Hertzog worked 18 hours (catching up the remaining 4 hours of October).
  • Holger Levsen did his share but did not manage to catch up with the backlog of the previous months. As such, those unused work hours have been redispatched among other contributors for the month of December.

New paid contributors

Last month we mentioned the possibility to recruit more paid contributors to better share the work load and this has already happened: Ben Hutchings and Mike Gabriel join the list of paid contributors.

Ben, as a kernel maintainer, will obviously take care of releasing Linux security updates. We are glad to have him on board because backporting kernel fixes really need some skills that nobody else had within the team of paid contributors.

Evolution of the situation

Compared to last month, the number of paid work hours has almost not increased (we are at 45.7 hours per month) but we are in the process of adding a few more sponsors: Roche Diagnostics International AG, Misal-System, Bitfolk LTD. And we are still in contact with a couple of other companies which have announced their willingness to contribute but which are waiting the new fiscal year.

But even with those new sponsors, we still have some way to go to reach our minimal goal of funding the equivalent of a half-time position. So consider asking your company representative to join this project!

In terms of security updates waiting to be handled, the situation looks better than last month: the dla-needed.txt file lists 27 packages awaiting an update (6 less than last month), the list of open vulnerabilities in Squeeze shows about 58 affected packages in total. Like last month, we’re a bit behind in terms of CVE triaging and there are still many packages using SSLv3 where we have no clear plan (in response to the POODLE issues).

The good side is that even though the kernel update spent a large chunk of time to Holger and Raphaël, we still managed to further reduce the backlog of security issues.

Thanks to our sponsors

My Free Software Activities in November 2014

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it’s one of the best ways to find volunteers to work with me on projects that matter to me.

Debian LTS

This month I have been paid to work 18 hours on Debian LTS (14h allocated by Freexian + 4h I did not spend last month). I did the following tasks:

  • CVE triage: I pushed 19 commits to the security tracker. I also tried to encourage some maintainers to provide security updates for packages that are not in use by the current LTS sponsors and that are thus not in our priority list.
  • DLA 87: dbus update fixing 3 CVE
  • DLA 93: libgcrypt11 update fixing 1 CVE
  • DLA 96: openjdk-6 security update fixing 21 CVE
  • Worked on preparing a security update to linux. It’s not released yet.

Updating the linux source package took a good half of the allocated time. We opted to update the kernel to the upstream version 2.6.32.64. I integrated the upstream patches and identified about 130 patches that we had to disable (because they were already integrated upstream). Then I updated our “openvz flavor” patch to apply on top of the new kernel. This required quite a bit of manual conflict resolution and there are even parts where I was not sure that I took the correct decision. I was not able to find an upstream openvz git tree on this kernel version to to double check.

Instead I asked Ben Hutchings to review my patch. He told me that he did not volunteer to work on LTS, but that he would be open to contribute to it for money. Following this remark, as the coordinator of Freexian’s offer, I offered him to join to the set of paid LTS contributors to take care of the kernel and he accepted.

So hopefully we will be able to wrap this linux upload in the first week of december. We had no uploads of the kernel in Squeeze since July so it’s good to know that we now have someone who will be able to handle it in priority.

Distro Tracker

No new developments this month. Instead I spent some time to import old historic news so that when you lookup removed packages you have some actual content instead of a 404 error. For example you can look at python2.1.

Another thing that I did is to tag some bugs with the newly-announced tag “newcomer”. Those are easy bugs that are ideal targets for new contributors who’d like to get started: here’s the list. It’s up to you now! ;-)

DEP-14: Recommended layout for Git packaging repositories

I have drafted an initial version of a document called Recommended layout for Git packaging repositories and submitted it for discussion on debian-devel.

The discussion has been interesting and constructive (yes this is still possible in Debian!). I have a bunch of improvements in my local copy and needs to process a few more feedback before submitting an updated draft. It’s not a revolution but it’s a good step to try to standardize tags and branches naming conventions.

Systemd, the tech-ctte and our mailing lists

As an old-timer, I care a lot about the governance of Debian and it’s annoying to see how the systemd debate brought back some of our old daemons in terms of hostile atmosphere on our mailing lists.

We can disagree on a lot of things, but we must respect each other and we are here to work together on solutions for everybody. As such I wrote to the persons who cross the line to invite them to behave better. And I’m glad that our listmasters are backing up our calls with bans when appropriate. I believe we must go further in that direction and I shared an idea (on a debian-private thread that should have never existed, much like most of the traffic on that list) that I shall formalize and share on debian-project@l.d.o at some point.

At the same time, we also had another governance-related discussion with the idea to impose some turnover in the technical committee. I’m glad to see that we will soon vote on this topic. This is a good thing in general even though we just had 3 tech-ctte members who retired.

Misc stuff

I sponsored an upload of galette and of 3 of its plugins. I reviewed jitsi-videobridge and jitsi-meet on mentors.debian.net.

I filed a few bugs:

  • #768256 about huge vim icons in the GNOME contextual menus
  • #768540: cdebootstrap: fails to bootstrap old releases with dpkg not supporting data.tar.xz
  • #770011: lynx -dump badly converting …

Thanks

See you next month for a new summary of my activities.

Freexian’s third report about Debian Long Term Support

Like last month, here comes a report about the work of paid contributors to Debian LTS.

Individual reports

In October 2014, we affected 13.75h works hours to 3 contributors:

  • Thorsten Alteholz
  • Raphaël Hertzog worked only 10 hours. The remaining hours will be done over November.
  • Holger Levsen did nothing (for unexpected personal reasons), he will catch up in November.

Obviously, only the hours done have been paid. Should the backlog grow further, we will seek for more paid contributors (to share the workload) and to make it easier to redispatch work hours once a contributor knows that he won’t be able to handle the hours that were affected to him/her.

Evolution of the situation

Compared to last month, we gained two new sponsors (Daevel and FOSSter, thanks to them!) and we have now 45.5 hours of paid LTS work to “spend” each month. That’s great but we are still far from our minimal goal of funding the equivalent of a half-time position.

In terms of security updates waiting to be handled, the situation is a bit worse than last month: while the dla-needed.txt file only lists 33 packages awaiting an update (6 less than last month), the list of open vulnerabilities in Squeeze shows about 60 affected packages in total. This differences has two explanations: CVE triaging for squeeze has not been done in the last days, and the POODLE issue(s) with SSLv3 affects a very large number of packages where it’s not always clear what the proper action is.

In any case, it’s never too late to join the growing list of sponsors and help us do a better job, please check with your company managers. If not possible for this year, consider including it in the budget for next year.

Thanks to our sponsors

Let me thank our main sponsors:

My Debian LTS report for October 2014

During October, I spent 10 hours on paid LTS work. I should have worked 4 hours more, but for various reasons this did not happen. Instead I’ll spend 4 more hours in November.

During this time, I did the following:

  • CVE triage: this month I pushed 23 commits to the security tracker SVN repository, and filed #765352 against wpa.
  • I released a mysql-5.1 update (DLA-75-1) fixing 3 CVE.
  • I sponsored the upload of ppp 2.4.5-4+deb6u1 for DLA-74-1 which had been prepared by Andrew Bartlett.
  • I uploaded a new version of apache2 (DLA-71-1) fixing 2 CVE.
  • I filed a few bugs against debian-security-support to request that some packages be marked as unsupported in squeeze: #765374 for axis2c and
    rampart, #765452 for a new feature allowing binary packages to be marked as unsupported so that glassfish-appserv can be marked as such (this is #765454).

After a few months of work on LTS, I’m starting to have a better grasp on the worflow and on what can be done or not. But I’m still astonished that we have so few squeeze users on the mailing list. If you’re using Squeeze, please subscribe to the list and test the packages that contributors are submitting for tests/validation. It really helps to have some feedback from real users before releasing an update, in particular when the Debian contributor who prepared the update is not a user of said package… not everybody has the skills required to prepare security updates, but everybody can help test packages, you have no excuses. ;-)

And we still need more organizations joining the LTS project, either by providing help (like Catalyst did by letting Andrew Bartlett work on LTS, thanks to them!) or by sponsoring the
project
and letting others do the work.

Freexian’s second report about Debian Long Term Support

Like last month, here comes a report about the work of paid contributors to Debian LTS.

Individual reports

In September 2014, 3 contributors have been paid for 11h each. Here are their individual reports:

Evolution of the situation

Compared to last month, we have gained 5 new sponsors, that’s great. We’re now at almost 25% of a full-time position. But we’re not done yet. We believe that we would need at least twice as many sponsored hours to do a reasonable work with at least the most used packages, and possibly four times as much to be able to cover the full archive.

We’re now at 39 packages that need an update in Squeeze (+9 compared to last month), and the contributors paid by Freexian did handle 11 during last month (this gives an approximate rate of 3 hours per update, CVE triage included).

Open questions

Dear readers, what can we do to convince more companies to join the effort?

The list of sponsors contains almost exclusively companies from Europe. It’s true that Freexian’s offer is in Euro but the economy is world-wide and it’s common to have international invoices. When Ivan Kohler asked if having an offer in dollar would help convince other companies, we got zero feedback.

What are the main obstacles that you face when you try to convince your managers to get the company to contribute?

By the way, we prefer that companies take small sponsorship commitments that they can afford over multiple years over granting lots of money now and then not being able to afford it for another year.

Thanks to our sponsors

Let me thank our main sponsors:

My Debian LTS report for September

Thanks to the sponsorship of multiple companies, I have been paid to work 11 hours on Debian LTS this month.

CVE triagingI started by doing lots of triage in the security tracker (if you want to help, instructions are here) because I noticed that the dla-needed.txt list (which contains the list of packages that must be taken care of via an LTS security update) was missing quite a few packages that had open vulnerabilities in oldstable.

In the end, I pushed 23 commits to the security tracker. I won’t list the details each time but for once, it’s interesting to let you know the kind of things that this work entailed:

  • I reviewed the patches for CVE-2014-0231, CVE-2014-0226, CVE-2014-0118, CVE-2013-5704 and confirmed that they all affected the version of apache2 that we have in Squeeze. I thus added apache2 to dla-needed.txt.
  • I reviewed CVE-2014-6610 concerning asterisk and marked the version in Squeeze as not affected since the file with the vulnerability doesn’t exist in that version (this entails some checking that the specific feature is not implemented in some other file due to file reorganization or similar internal changes).
  • I reviewed CVE-2014-3596 and corrected the entry that said that is was fixed in unstable. I confirmed that the versions in squeeze was affected and added it to dla-needed.txt.
  • Same story for CVE-2012-6153 affecting commons-httpclient.
  • I reviewed CVE-2012-5351 and added a link to the upstream ticket.
  • I reviewed CVE-2014-4946 and CVE-2014-4945 for php-horde-imp/horde3, added links to upstream patches and marked the version in squeeze as unaffected since those concern javascript files that are not in the version in squeeze.
  • I reviewed CVE-2012-3155 affecting glassfish and was really annoyed by the lack of detailed information. I thus started a discussion on debian-lts to see whether this package should not be marked as unsupported security wise. It looks like we’re going to mark a single binary packages as unsupported… the one containing the application server with the vulnerabilities, the rest is still needed to build multiple java packages.
  • I reviewed many CVE on dbus, drupal6, eglibc, kde4libs, libplack-perl, mysql-5.1, ppp, squid and fckeditor and added those packages to dla-needed.txt.
  • I reviewed CVE-2011-5244 and CVE-2011-0433 concerning evince and came to the conclusion that those had already been fixed in the upload 2.30.3-2+squeeze1. I marked them as fixed.
  • I droppped graphicsmagick from dla-needed.txt because the only CVE affecting had been marked as no-dsa (meaning that we don’t estimate that a security updated is needed, usually because the problem is minor and/or that fixing it has more chances to introduce a regression than to help).
  • I filed a few bugs when those were missing: #762789 on ppp, #762444 on axis.
  • I marked a bunch of CVE concerning qemu-kvm and xen as end-of-life in Squeeze since those packages are not currently supported in Debian LTS.
  • I reviewed CVE-2012-3541 and since the whole report is not very clear I mailed the upstream author. This discussion led me to mark the bug as no-dsa as the impact seems to be limited to some information disclosure. I invited the upstream author to continue the discussion on RedHat’s bugzilla entry.

And when I say “I reviewed” it’s a simplification for this kind of process:

  • Look up for a clear explanation of the security issue, for a list of vulnerable versions, and for patches for the versions we have in Debian in the following places:
    • The Debian security tracker CVE page.
    • The associated Debian bug tracker entry (if any).
    • The description of the CVE on cve.mitre.org and the pages linked from there.
    • RedHat’s bugzilla entry for the CVE (which often implies downloading source RPM from CentOS to extract the patch they used).
    • The upstream git repository and sometimes the dedicated security pages on the upstream website.
  • When that was not enough to be conclusive for the version we have in Debian (and unfortunately, it’s often the case), download the Debian source package and look at the source code to verify if the problematic code (assuming that we can identify it based on the patch we have for newer versions) is also present in the old version that we are shipping.

CVE triaging is often almost half the work in the general process: once you know that you are affected and that you have a patch, the process to release an update is relatively straightforward (sometimes there’s still work to do to backport the patch).

Once I was over that first pass of triaging, I had already spent more than the 11 hours paid but I still took care of preparing the security update for python-django. Thorsten Alteholz had started the work but got stuck in the process of backporting the patches. Since I’m co-maintainer of the package, I took over and finished the work to release it as DLA-65-1.

Freexian’s first report about Debian Long Term Support

When we setup Freexian’s offer to bring together funding from multiple companies in order to sponsor the work of multiple developers on Debian LTS, one of the rules that I imposed is that all paid contributors must provide a public monthly report of their paid work.

While the LTS project officially started in June, the first month where contributors were actually paid has been July. Freexian sponsored Thorsten Alteholz and Holger Levsen for 10.5 hours each in July and for 16.5 hours each in August. Here are their reports:

It’s worth noting that Freexian sponsored Holger’s work to fix the security tracker to support squeeze-lts. It’s my belief that using the money of our sponsors to make it easier for everybody to contribute to Debian LTS is money well spent.

As evidenced by the progress bar on Freexian’s offer page, we have not yet reached our minimal goal of funding the equivalent of a half-time position. And it shows in the results, the dla-needed.txt still shows around 30 open issues. This is slightly better than the state two months ago but we can improve a lot on the average time to push out a security update…

To have an idea of the relative importance of the contributions of the paid developers, I counted the number of uploads made by Thorsten and Holger since July: of 40 updates, they took care of 19 of them, so about the half.

I also looked at the other contributors: Raphaël Geissert stands out with 9 updates (I believe that he is contracted by Électricité de France for doing this) and most of the other contributors look like regular Debian maintainers taking care of their own packages (Paul Gevers with cacti, Christoph Berg with postgresql, Peter Palfrader with tor, Didier Raboud with cups, Kurt Roeckx with openssl, Balint Reczey with wireshark) except Matt Palmer and Luciano Bello who (likely) are benevolent members of the LTS team.

There are multiple things to learn here:

  1. Paid contributors already handle almost 70% of the updates. Counting only on volunteers would not have worked.
  2. Quite a few companies that promised help (and got mentioned in the press release) have not delivered the promised help yet (neither through Freexian nor directly).

Last but not least, this project wouldn’t exist without the support of multiple companies and organizations. Many thanks to them:

Hopefully this list will expand over time! Any help to reach out to new companies and organizations is more than welcome.

My Free Software Activities in July 2014

This is my monthly summary of my free software related activities. If you’re among the people who made a donation to support my work (548.59 €, thanks everybody!), then you can learn how I spent your money. Otherwise it’s just an interesting status update on my various projects.

Distro Tracker

Now that tracker.debian.org is live, people reported bugs (on the new tracker.debian.org pseudo-package that I requested) faster than I could fix them. Still I spent many, many hours on this project, reviewing submitted patches (thanks to Christophe Siraut, Joseph Herlant, Dimitri John Ledkov, Vincent Bernat, James McCoy, Andrew Starr-Bochicchio who all submitted some patches!), fixing bugs, making sure the code works with Django 1.7, and started the same with Python 3.

I added a tox.ini so that I can easily run the test suite in all 4 supported environments (created by tox as virtualenv with the combinations of Django 1.6/1.7 and Python 2.7/3.4).

Over the month, the git repository has seen 73 commits, we fixed 16 bugs and other issues that were only reported over IRC in #debian-qa. With the help of Enrico Zini and Martin Zobel, we enabled the possibility to login via sso.debian.org (Debian’s official SSO) so that Debian developers don’t even have to explicitly create their account.

As usual more help is needed and I’ll gladly answer your questions and review your patches.

Misc packaging work

Publican. I pushed a new upstream release of publican and dropped a useless build-dependency that was plagued by a difficult to fix RC bug (#749357 for the curious, I tried to investigate but it needs major work for make 4.x compatibility).

GNOME 3.12. With gnome-shell 3.12 hitting unstable, I had to update gnome-shell-timer (and filed an upstream ticket at the same time), a GNOME Shell extension to start some run-down counters.

Django 1.7. I packaged python-django 1.7 release candidate 1 in experimental (found a small bug, submitted a ticket with a patch that got quickly merged) and filed 85 bugs against all the reverse dependencies to ask their maintainers to test their package with Django 1.7 (that we want to upload before the freeze obviously). We identified a pain point in upgrade for packages using South and tried to discuss it with upstream, but after closer investigation, none of the packages are really affected. But the problem can hit administrators of non-packaged Django applications.

Misc stuff. I filed a few bugs (#754282 against git-import-orig –uscan, #756319 against wnpp to see if someone would be willing to package loomio), reviewed an updated package for django-ratelimit in #755611, made a non-maintainer upload of mairix (without prior notice) to update the package to a new upstream release and bring it to modern packaging norms (Mako failed to make an upload in 4 years so I just went ahead and did what I would have done if it were mine).

Kali work resulting in Debian contributions

Kali wants to switch from being based on stable to being based on testing so I did try to setup britney to manage a new kali-rolling repository and encountered some problems that I reported to debian-release. Niels Thykier has been very helpful and even managed to improve britney thanks to the very specific problem that the kali setup triggered.

Since we use reprepro, I did write some Python wrapper to transform the HeidiResult file in a set of reprepro commands but at the same time I filed #756399 to request proper support of heidi files in reprepro. While analyzing britney’s excuses file, I also noticed that the Kali mirrors contains many source packages that are useless because they only concern architectures that we don’t host (and I filed #756523 against reprepro). While trying to build a live image of kali-rolling, I noticed that libdb5.1 and db5.1-util were still marked as priority standard when in fact Debian already switched to db5.3 and thus should only be optional (I filed #756623 against ftp.debian.org).

When doing some upgrade tests from kali (wheezy based) to kali-rolling (jessie based) I noticed some problems that were also affecting Debian Jessie. I filed #756629 against libfile-fcntllock-perl (with a patch), and also #756618 against texlive-base (missing Replaces header). I also pinged Colin Watson on #734946 because I got a spurious base-passwd prompt during upgrade (that was triggered because schroot copied my unstable’s /etc/passwd file in the kali chroot and the package noticed a difference on the shell of all system users).

Thanks

See you next month for a new summary of my activities.

Spotify migrate 5000 servers from Debian to Ubuntu

Or yet another reason why it’s really important that we succeed with Debian LTS. Last year we heard of Dreamhost switching to Ubuntu because they can maintain a stable Ubuntu release for longer than a Debian stable release (and this despite the fact that Ubuntu only supports software in its main section, which misses a lot of popular software).

Spotify Logo

A few days ago, we just learned that Spotify took a similar decision:

A while back we decided to move onto Ubuntu for our backend server deployment. The main reasons for this was a predictable release cycle and long term support by upstream (this decision was made before the announcement that the Debian project commits to long term support as well.) With the release of the Ubuntu 14.04 LTS we are now in the process of migrating our ~5000 servers to that distribution.

This is just a supplementary proof that we have to provide long term support for Debian releases if we want to stay relevant in big deployments.

But the task is daunting and it’s difficult to find volunteers to do the job. That’s why I believe that our best answer is to get companies to contribute financially to Debian LTS.

We managed to convince a handful of companies already and July is the first month where paid contributors have joined the effort for a modest participation of 21 work hours (watch out for Thorsten Alteholz and Holger Levsen on debian-lts and debian-lts-announce). But we need to multiply this figure by 5 or 6 at least to make a correct work of maintaining Debian 6.

So grab the subscription form and have a chat with your management. It’s time to convince your company to join the initiative. Don’t hesitate to get in touch if you have questions or if you prefer that I contact a representative of your company. Thank you!

My Free Software Activity in June 2014

This is my monthly summary of my free software related activities. If you’re among the people who made a donation to support my work (168.17 €, thanks everybody!), then you can learn how I spent your money. Otherwise it’s just an interesting status update on my various projects.

Debian LTS

After having put in place the infrastructure to allow companies to contribute financially to Debian LTS, I spent quite some time to draft the announce of the launch of Debian LTS (on a suggestion of Moritz Mühlenhoff who pointed out to me that there was no such announce yet).

I’m pretty happy about the result because we managed to mention a commercial offer without generating any pushback from the community. The offer is (in my necessarily biased opinion) clearly in the interest of Debian but still the money doesn’t go to Debian so we took extra precautions. When I got in touch with the press officers, I included the Debian leader in the discussion and his feedback has been very helpful to improve the announce. He also officially “acked” the press release to give some confidence to the press officers that they were doing the right thing.

Lucas also pushed me to seek public review of the draft press release, which I did. The discussion was constructive and the draft got further improved.

The news got widely relayed, but on the flip side, the part with the call for help got almost no attention from the press. Even Linux Weekly News skipped it!

On the Freexian side, we just crossed 10% of a full-time position (funded by 6 companies) and we are in contact with a few other companies in discussion. But we’re far from our goal yet so we will have to actively reach out to more companies. Do you know companies who are still running Debian 6 servers ? If yes, please send me the details (name + url + contact info if possible) to deblts@freexian.com so that I can get in touch and invite them to contribute to the project.

Distro Tracker

In the continuation of the Debian France game, I continued to work together with Joseph Herlant and Christophe Siraut on multiple improvements to distro tracker in order to prepare for its deployment on tracker.debian.org (which I just announced \o/).

Debian France

Since the Debian France game was over, I shipped the rewards. 5 books have been shipped to:

Misc Debian work

I orphaned sql-ledger and made a last upload to change the maintainer to Debian QA (with a new upstream version).

After having been annoyed a few times by dch breaking my name in the changelog, I filed #750855 which got quickly fixed.

I disabled a broken patch in quilt to fix RC bug #751109.

I filed #751771 when I discovered an incorrect dependency on ruby-uglifier (while doing packaging work for Kali Linux).

I tested newer versions of ruby-libv8 on armel/armhf on request of the upstream author. I had reported him those build failures (github ticket here).

Thanks

See you next month for a new summary of my activities.