Freexian’s report about Debian Long Term Support, April 2015

A Debian LTS logoLike each month, here comes a report about the work of paid contributors to Debian LTS.

Individual reports

In April, 81.75 work hours have been dispatched among 5 paid contributors (20.75 hours where unused hours of Ben and Holger that were re-dispatched to other contributors). Their reports are available:

Evolution of the situation

May has seen a small increase in terms of sponsored hours (66.25 hours per month) and June is going to do even better with at least a new gold sponsor. We will have no problems sustaining the increased workload it implies since three Debian developers joined the team of contributors paid by Freexian (Antoine Beaupré, Santiago Ruano Rincón, Scott Kitterman).

The Jessie release probably shed some light on the Debian LTS project since we announced that Jessie will benefit from 5 years of support. Let’s hope that the trend will continue in the following months and that we reach our first milestone of funding the equivalent of a half-time position.

In terms of security updates waiting to be handled, the situation is a bit contrasted: the dla-needed.txt file lists 28 packages awaiting an update (12 less than last month), the list of open vulnerabilities in Squeeze shows about 60 affected packages in total (4 more than last month). The extra hours helped to make a good stride in the packages awaiting an update but there are many new vulnerabilities waiting to be triaged.

Thanks to our sponsors

The new sponsors of the month are in bold.

Freexian’s report about Debian Long Term Support, March 2015

A Debian LTS logoLike each month, here comes a report about the work of paid contributors to Debian LTS.

Individual reports

In March, 61 work hours have been equally split among 4 paid contributors. Their reports are available:

The remaining hours of Ben and Holger have been redispatched to other contributors for April (during which Mike Gabriel joins the set of paid contributors). BTW, if you want to join the team of paid contributors, read this and apply!

Evolution of the situation

April has seen no change in terms of sponsored hours but we have two new sponsors in the pipe and May should hopefully have a few more sponsored hours.

For the need of a LTS presentation I gave during the Mini-DebConf Lyon I prepared a small graph showing the evolution of the hours sponsored through Freexian:
freexian-hours

The growth is rather slow and it will take years to reach our goal of funding the equivalent a full time position (176 hours per month). Even the intermediary goal of funding the equivalent of a half-time position (88h/month) is more than 6 months away given the current growth rate. But the perspective of Wheezy-LTS should help us to convince more organizations and hopefully we will reach that goal sooner. If you want to sponsor the project, check out this page.

In terms of security updates waiting to be handled, the situation looks similar to last month: the dla-needed.txt file lists 40 packages awaiting an update (exactly like last month), the list of open vulnerabilities in Squeeze shows about 56 affected packages in total (2 less than last month).

Thanks to our sponsors

The new sponsors of the month are in bold (none this month).

Freexian’s report about Debian Long Term Support, February 2015

Like each month, here comes a report about the work of paid contributors to Debian LTS.

Individual reports

In February, 58 work hours have been equally split among 4 paid contributors. Their reports are available:

Evolution of the situation

During the last month, we gained 3 paid work hours: we’re now at 61 hours per month sponsored by 28 organizations and we have one supplementary sponsor in the pipe that should bring 4 more hours.

The increase is not very quick but seems to be steady. Hopefully at some point, we will have enough resources to do a more exhaustive job. For now, the paid contributors handle in priority the most popular packages used by the sponsors and there are some packages in the end of the queue which have open security issues for months already (example: CVE-2012-6685 on libnokogiri-ruby).

So, as usual, we are looking for more sponsors.

In terms of security updates waiting to be handled, the situation looks a little bit worse than last month: the dla-needed.txt file lists 40 packages awaiting an update (3 more than last month), the list of open vulnerabilities in Squeeze shows about 58 affected packages in total (5 less than last month). We are getting a bit more effective with CVE triage.

A logo for the LTS project?

Every time that I write an LTS report, I remember that it would be nice if my LTS related articles could feature a nice picture/logo that reminds people of the LTS team/initiative. Is there anyone up for the challenge of creating that logo? :-)

Thanks to our sponsors

The new sponsors of the month are in bold.

Freexian’s report about Debian Long Term Support, January 2015

Like each month, here comes a report about the work of paid contributors to Debian LTS.

Individual reports

In January, 48 work hours have been equally split among 4 paid contributors. Their reports are available:

Evolution of the situation

During the last month, the number of paid work hours has made a noticeable jump: we’re now at 58 hours per month. At this rate, we would need 3 more months to reach our minimal goal of funding the equivalent of a half-time position. Unfortunately, the number of new sponsors actually in the process is not likely to be enough to have a similar raise next month.

So, as usual, we are looking for more sponsors.

In terms of security updates waiting to be handled, the situation looks a bit worse than last month: the dla-needed.txt file lists 37 packages awaiting an update (7 more than last month), the list of open vulnerabilities in Squeeze shows about 63 affected packages in total (7 more than last month).

The increase is not too worrying, but the waiting time before an issue is dealt with is sometimes more problematic. To be able to deal with all incoming issues in a timely manner, the LTS team needs more resources: some months will have more issues than usual, some issues will be longer to handle than others, etc.

Thanks to our sponsors

The new sponsors of the month are in bold.

Freexian’s fifth report about Debian Long Term Support

Like each month, here comes a report about the work of paid contributors to Debian LTS.

Individual reports

In December 46 work hours have been equally split among 4 paid contributors (note that Thorsten and Raphaël have actually spent more hours because they took over some hours that Holger did not do over the former months). Their reports are available:

Evolution of the situation

Compared to last month, the number of paid work hours has almost not increased (we are at 48 hours per month). We still have a couple of new sponsors in the pipe but with the new year they did not complete the process yet. Hopefully next month will see a noticeable increase.

As usual, we are looking for more sponsors to reach our our minimal goal of funding the equivalent of a half-time position. Those of you who are struggling to spend money in the last quarter due to budget overrun, now is a good time to see if you want to include Debian LTS support in your 2015 budget!

In terms of security updates waiting to be handled, the situation looks similar to last month: the dla-needed.txt file lists 30 packages awaiting an update (3 more than last month), the list of open vulnerabilities in Squeeze shows about 56 affected packages in total. We do not manage to clear the backlog but it’s not getting significantly worse either.

Thanks to our sponsors

Freexian’s fourth report about Debian Long Term Support

Like each month, here comes a report about the work of paid contributors to Debian LTS.

Individual reports

In November 42.5 work hours have been equally split among 3 paid contributors. Their reports are available:

  • Thorsten Alteholz did his share as usual.
  • Raphaël Hertzog worked 18 hours (catching up the remaining 4 hours of October).
  • Holger Levsen did his share but did not manage to catch up with the backlog of the previous months. As such, those unused work hours have been redispatched among other contributors for the month of December.

New paid contributors

Last month we mentioned the possibility to recruit more paid contributors to better share the work load and this has already happened: Ben Hutchings and Mike Gabriel join the list of paid contributors.

Ben, as a kernel maintainer, will obviously take care of releasing Linux security updates. We are glad to have him on board because backporting kernel fixes really need some skills that nobody else had within the team of paid contributors.

Evolution of the situation

Compared to last month, the number of paid work hours has almost not increased (we are at 45.7 hours per month) but we are in the process of adding a few more sponsors: Roche Diagnostics International AG, Misal-System, Bitfolk LTD. And we are still in contact with a couple of other companies which have announced their willingness to contribute but which are waiting the new fiscal year.

But even with those new sponsors, we still have some way to go to reach our minimal goal of funding the equivalent of a half-time position. So consider asking your company representative to join this project!

In terms of security updates waiting to be handled, the situation looks better than last month: the dla-needed.txt file lists 27 packages awaiting an update (6 less than last month), the list of open vulnerabilities in Squeeze shows about 58 affected packages in total. Like last month, we’re a bit behind in terms of CVE triaging and there are still many packages using SSLv3 where we have no clear plan (in response to the POODLE issues).

The good side is that even though the kernel update spent a large chunk of time to Holger and Raphaël, we still managed to further reduce the backlog of security issues.

Thanks to our sponsors

Freexian’s third report about Debian Long Term Support

Like last month, here comes a report about the work of paid contributors to Debian LTS.

Individual reports

In October 2014, we affected 13.75h works hours to 3 contributors:

  • Thorsten Alteholz
  • Raphaël Hertzog worked only 10 hours. The remaining hours will be done over November.
  • Holger Levsen did nothing (for unexpected personal reasons), he will catch up in November.

Obviously, only the hours done have been paid. Should the backlog grow further, we will seek for more paid contributors (to share the workload) and to make it easier to redispatch work hours once a contributor knows that he won’t be able to handle the hours that were affected to him/her.

Evolution of the situation

Compared to last month, we gained two new sponsors (Daevel and FOSSter, thanks to them!) and we have now 45.5 hours of paid LTS work to “spend” each month. That’s great but we are still far from our minimal goal of funding the equivalent of a half-time position.

In terms of security updates waiting to be handled, the situation is a bit worse than last month: while the dla-needed.txt file only lists 33 packages awaiting an update (6 less than last month), the list of open vulnerabilities in Squeeze shows about 60 affected packages in total. This differences has two explanations: CVE triaging for squeeze has not been done in the last days, and the POODLE issue(s) with SSLv3 affects a very large number of packages where it’s not always clear what the proper action is.

In any case, it’s never too late to join the growing list of sponsors and help us do a better job, please check with your company managers. If not possible for this year, consider including it in the budget for next year.

Thanks to our sponsors

Let me thank our main sponsors:

Freexian’s second report about Debian Long Term Support

Like last month, here comes a report about the work of paid contributors to Debian LTS.

Individual reports

In September 2014, 3 contributors have been paid for 11h each. Here are their individual reports:

Evolution of the situation

Compared to last month, we have gained 5 new sponsors, that’s great. We’re now at almost 25% of a full-time position. But we’re not done yet. We believe that we would need at least twice as many sponsored hours to do a reasonable work with at least the most used packages, and possibly four times as much to be able to cover the full archive.

We’re now at 39 packages that need an update in Squeeze (+9 compared to last month), and the contributors paid by Freexian did handle 11 during last month (this gives an approximate rate of 3 hours per update, CVE triage included).

Open questions

Dear readers, what can we do to convince more companies to join the effort?

The list of sponsors contains almost exclusively companies from Europe. It’s true that Freexian’s offer is in Euro but the economy is world-wide and it’s common to have international invoices. When Ivan Kohler asked if having an offer in dollar would help convince other companies, we got zero feedback.

What are the main obstacles that you face when you try to convince your managers to get the company to contribute?

By the way, we prefer that companies take small sponsorship commitments that they can afford over multiple years over granting lots of money now and then not being able to afford it for another year.

Thanks to our sponsors

Let me thank our main sponsors:

Freexian’s first report about Debian Long Term Support

When we setup Freexian’s offer to bring together funding from multiple companies in order to sponsor the work of multiple developers on Debian LTS, one of the rules that I imposed is that all paid contributors must provide a public monthly report of their paid work.

While the LTS project officially started in June, the first month where contributors were actually paid has been July. Freexian sponsored Thorsten Alteholz and Holger Levsen for 10.5 hours each in July and for 16.5 hours each in August. Here are their reports:

It’s worth noting that Freexian sponsored Holger’s work to fix the security tracker to support squeeze-lts. It’s my belief that using the money of our sponsors to make it easier for everybody to contribute to Debian LTS is money well spent.

As evidenced by the progress bar on Freexian’s offer page, we have not yet reached our minimal goal of funding the equivalent of a half-time position. And it shows in the results, the dla-needed.txt still shows around 30 open issues. This is slightly better than the state two months ago but we can improve a lot on the average time to push out a security update…

To have an idea of the relative importance of the contributions of the paid developers, I counted the number of uploads made by Thorsten and Holger since July: of 40 updates, they took care of 19 of them, so about the half.

I also looked at the other contributors: Raphaël Geissert stands out with 9 updates (I believe that he is contracted by Électricité de France for doing this) and most of the other contributors look like regular Debian maintainers taking care of their own packages (Paul Gevers with cacti, Christoph Berg with postgresql, Peter Palfrader with tor, Didier Raboud with cups, Kurt Roeckx with openssl, Balint Reczey with wireshark) except Matt Palmer and Luciano Bello who (likely) are benevolent members of the LTS team.

There are multiple things to learn here:

  1. Paid contributors already handle almost 70% of the updates. Counting only on volunteers would not have worked.
  2. Quite a few companies that promised help (and got mentioned in the press release) have not delivered the promised help yet (neither through Freexian nor directly).

Last but not least, this project wouldn’t exist without the support of multiple companies and organizations. Many thanks to them:

Hopefully this list will expand over time! Any help to reach out to new companies and organizations is more than welcome.

Spotify migrate 5000 servers from Debian to Ubuntu

Or yet another reason why it’s really important that we succeed with Debian LTS. Last year we heard of Dreamhost switching to Ubuntu because they can maintain a stable Ubuntu release for longer than a Debian stable release (and this despite the fact that Ubuntu only supports software in its main section, which misses a lot of popular software).

Spotify Logo

A few days ago, we just learned that Spotify took a similar decision:

A while back we decided to move onto Ubuntu for our backend server deployment. The main reasons for this was a predictable release cycle and long term support by upstream (this decision was made before the announcement that the Debian project commits to long term support as well.) With the release of the Ubuntu 14.04 LTS we are now in the process of migrating our ~5000 servers to that distribution.

This is just a supplementary proof that we have to provide long term support for Debian releases if we want to stay relevant in big deployments.

But the task is daunting and it’s difficult to find volunteers to do the job. That’s why I believe that our best answer is to get companies to contribute financially to Debian LTS.

We managed to convince a handful of companies already and July is the first month where paid contributors have joined the effort for a modest participation of 21 work hours (watch out for Thorsten Alteholz and Holger Levsen on debian-lts and debian-lts-announce). But we need to multiply this figure by 5 or 6 at least to make a correct work of maintaining Debian 6.

So grab the subscription form and have a chat with your management. It’s time to convince your company to join the initiative. Don’t hesitate to get in touch if you have questions or if you prefer that I contact a representative of your company. Thank you!