Debian related goals for 2012

Like last year, here’s a list of Debian related goals that I’d like to achieve this year. I might not have the time to implement all the projects, but I like to write them down to keep me motivated. And maybe it can inspire other people to implement some of them (or to help me).

  1. Finish the translation of the Debian Administrator’s Handbook

    The target is to have the book available in April. It would be nice to complete the liberation fund until then so that the book is immediately made available under a DFSG-free license.

  2. Update the Debian Administrator’s Handbook for Wheezy
  3. Translating the book in English is only the start of the journey. The real challenge is to keep the book up-to-date with each subsequent release of Debian. And Wheezy should hopefully be released in 2012 since the freeze is in June.

  4. Design and implement the Debian Package Maintenance Hub

    It’s an ambitious project that aims to merge and replace the PTS, the DDPO and their respective mail variants. It should also standardize the flow of information directed towards package maintainers. I’m going to use the DEP process to drive this project.

    This could easily take most of the year, but hopefully I’ll motivate other people to chime in and help.

  5. Implement dpkg --check-db and dpkg --repair-db

    While dpkg is fairly reliable, it’s not exempt of bugs and more annoyingly, harddrives/filesystems are not 100% reliable either, thus it happens that some internal database files get corrupted. Given that most files are text based, advanced users can manually fix them but many less skilled users are just left with a broken system that they tend to reinstall.

    To avoid this, we could provide a command that would try to automatically bring back the internal database to a sane state by looking for a working backup to restore (while at the same time marking some packages as requiring re-installation since we have some indications that they were present).

  6. Implement storage of dpkg’s internal files in Git

    This would be an extension of the former idea. Installing a package dpkg-db-history (any idea for a better name?) would setup dpkg hooks that would record every database change in a git repository. This repository could then be used to restore the last working version of the database.

Besides those concrete projects, I want to do better than last year on the topic of funding my Debian work. I will thus reiterate some objectives:

  1. Write useful articles for Debian users and Debian contributors.

    They should complete the pages Mastering Debian, Contributing to Debian 101, Debian Packaging Tutorials, and help me increase the audience of this blog.

  2. Write at least one Debian-related ebook (different from the Debian Admin Handbook) and sell it.

    It could be an ebook targetting testing users since I believe that many more users could benefit from it if they had some better knowledge of the limitations and of the way to mitigate the problems that arise from time to time.

    Or maybe it could be an ebook for people who want to start contributing to Debian, it could even be bundled with a few hours of mentoring.

  3. By the end of the year, have at least 1/3 of my time funded by donations and/or earnings of my information products.

    This means doing 3,5 times better than in 2011. It should be doable given that the sales of the Debian Administrator’s Handbook will contribute to this goal (once the translation is over).

That makes up lots of challenges for this year. Feel free subscribe to my newsletter to stay up-to-date with my progress and to get my monthly summary of the Debian/Ubuntu news. It’s also a good way to help me reach those goals since you will be informed of all my new projects.

Do we need project-wide support for Debian rolling?

The discussion about Debian rolling started sooner than expected on debian-devel (see the thread here). I initially wanted to iron out the biggest problems through discussions on my blog and try to submit a somewhat polished proposal… instead we ended up discussing the same things both on -devel and on my blog.

That said it’s not that bad (except for the time I lost to have similar conversations in both places) because the debian-devel discussions included members of the release team and it looks like they are not fundamentally opposed to the idea.

Despite this, the introduction of a “frozen” suite branched off from “testing/rolling” is not really consensual (yet?). But the idea of officially supporting testing on a best-effort basis appears to have almost no opposition.

While some will undoubtedly believe that this is a useless exercise, I believe it would help if the project stated this in a somewhat official manner. My answer to the question in the title is thus:

YES

I am thus considering to submit a general resolution to that effect. My current draft is below.

Title: Debian endorses usage of testing by end-users, and renames it to rolling

The Debian project recognizes that the Debian testing distribution—initially created to make it easier to prepare and test the next stable release—has become a useful product of its own. It satisfies the needs of users who are looking for the latest stable versions of software and who can cope (or even appreciate) a system that’s constantly evolving.

The Debian project decides to endorse this usage and will strive to provide a good experience to users of “testing”. To better communicate this policy change to our users, “testing” will be renamed “rolling”.

While we believe that this is a good move, we would like to remind our users that Debian is a volunteer project and that our resources are not infinite. Package maintainers are contributing to Debian on a best-effort basis. This means that they might not be able to properly support their package(s) in all distributions. In that case, the project recommends that maintainers apply the following priorities:

  1. Support in stable (security updates, release critical bugs)
  2. Preparation of the next stable release
  3. Support in rolling

Note that this general resolution could have amendments with s/rolling/current/ and that would solve the bikeshedding over the name that started several times already.

I deliberately separated “Preparation of the next stable release” and “Support in rolling” so that priorities are clear even if we decide to not freeze “rolling” and to introduce a “frozen” distribution to finalize the next stable release.

I also did not go into too much details on the implications that it might have, it’s best to leave that up to each contributor/team/etc.

I hesitated to add a paragraph stating that we want to try to gradually improve the usability of testing but in the end I think it would be somewhat redundant. We’re always trying to do our best when we decide to take on something.

All comments welcome (even if you just agree and would be willing to second such a GR).

Update: I will tweak the draft included in this article when I get good suggestions. Thanks to Lucas Nussbaum for the first one.

No freeze of Debian’s development, what does it entail?

The main feature of rolling is that it would never freeze. This is not without consequences.

Possible consequences

It can divert developers from working on the release

No freeze means developers are free to continue their work as usual in unstable. Will it be more difficult to release because some people will spend their time working on a new upstream version instead of fixing RC bugs in the version that is frozen? Would we lose the work of the people who do lots of NMU to help with the release?

It makes it more difficult to cherry-pick updates from unstable

No freeze also means that unstable is going to diverge sooner from testing and it will be more difficult to cherry-pick updates from unstable into testing. And the release team likes to cherry-pick updates that have been tested in unstable because updates that comes through testing-proposed-updates have often not been tested and need thus a more careful review.

Frozen earth

My responses to the objections

Those are the two major objections that we’ll have to respond to. Let’s try to analyze them a bit more.

It’s not testing vs rolling

On the first objection I would like to respond that we must not put “testing” and “rolling/unstable” in opposition. The fact that a contributor can’t do its work as usual in unstable does not mean that he will instead choose to work on fixing RC bugs in testing. Probably that some do, but in my experience we simply spend our time differently, either working more on non-Debian stuff or doing mostly hidden work that is then released in big batches at the start of the next cycle (which tends to create problems of its own).

I would also like to argue that by giving more exposure to rolling and encouraging developers to properly support their packages in rolling, it probably means that the overall state of rolling should become gradually better compared to what we’re currently used to with testing.

The objection that rolling would divert resources from getting testing in a releasable shape is difficult to prove and/or disprove. The best way to have some objective data would be to setup a questionnaire and to ask all maintainers. Any volunteer for that?

Unstable as a test-bed for RC bugfixes?

It’s true that unstable will quickly diverge from testing and that it will be more difficult to cherry-pick updates from unstable into testing. This cannot be refuted, it’s a downside given the current workflow of the release team.

But I wonder if the importance of this workflow is not overdone. The reason why they like to cherry-pick from unstable is because it gives them some confidence that the update has not caused other regressions and ensures that testing is improving.

But if they’re considering to cherry-pick an update, it’s because the current package in testing is plagued by an RC bug. Supposing that the updated package has introduced a regression, is it really better to keep the current RC bug compared to trading it for a new regression? It sure depends on the precise bugs involved so that’s why they prefer to know up-front about the regression instead of making a blind bet.

Given this, I think we should use testing-proposed-updates (tpu) as a test-bed for RC bug fixes. We should ask beta-testers to activate this repository and to file RC bugs for any regression. And instead of requiring a full review by a release manager for all uploads to testing-proposed-updates, uploads should be auto-accepted provided that they do not change the upstream version and that they do not add/remove binary packages. Other uploads would still need manual approval by the release managers.

On top of this, we can also add an infrastructure to encourage peer-reviews of t-p-u uploads so that reviews become more opportunistic instead of systematic. Positive reviews would help reduce the aging required in t-p-u before being accepted into testing.

This changes the balance by giving a bit more freedom to maintainers but still keeps the safety net that release managers need to have. It should also reduce the overall amount of work that the release team has to do.

Comments welcome

Do you see other important objections beside the two that I mentioned?

Do you have other ideas to overcome those objections?

What do you think of my responses? Does your experience infirm or confirm my point of view?

Towards Debian rolling: my own Debian CUT manifesto

As you might know, I’m of one the people who promote the idea of having a Constantly Usable Testing (CUT). I will post a set of articles on this topic to help me clarify my ideas and to get some early feedback of other Debian contributors. I plan to use the result of this process to kickstart a broader discussion on debian-devel (where I hope to get the release team involved).

Let’s start by defining my own objectives with Debian CUT. I won’t speak of the part of Debian CUT that plans to make regular snapshots of testing with installable media because that’s not where I will invest my time. I do hope other people will do it though. Instead I will concentrate on changes that would improve Debian testing for end-users. I consider that testing (in its current form) is largely usable already but there are two main obstacles to overcome.

Testing without freezes = rolling

The first one is that testing is no longer testing during freezes. The regular flow of new upstream versions—that makes testing so interesting for many users—is stalled because we’re using testing to finalize the next stable version. That’s why I’d like to introduce a new suite called “rolling” that would work like the current testing except that it never freezes. Testing would no longer be a permanent suite, it would only exist during the freeze and it would be branched off from rolling.

Rolling should be a supported distribution

The second obstacle is not really technical. We must advertise rolling as a distribution that ordinary people can use but we should make it clear that it’s never going to have the same level of polish that a stable release can have. And to back up this assertion, we must empower Debian developers to be able to provide good support of their software in rolling, this probably means using “rolling-proposed-updates” more often to push fixes and security updates when the natural flow of updates is blocked by transitions or other problems. Some maintainers won’t have the time required to provide the same level of support as they currently do for a stable release and that’s okay, it’s not worse than the current testing. The goal is to treat it on a best-effort basis and to try to gradually improve the situation.

My goal for wheezy

This is the the minimal implementation of CUT that I would like to target in the wheezy timeframe. Having testing as a temporary branch of rolling is not a strict requirement, although I’d argue that it’s important to not waste resources towards maintaining two separate yet very similar distributions.

Should Debian embrace those goals?

Ignoring all possible problems that will surface while trying to implement those goals, can we agree that Debian should embrace those goals because it means providing a better service to a class of users that is not satisfied by the current stable release?

I will come back to the expected problems in a subsequent post and we will have the opportunity to discuss them. Here I just want to see whether we can have broad buy-in on the principles behind Debian rolling and CUT.

My Debian related goals for 2011

Like last year, here’s a list of Debian related projects that I’d like to tackle this year. I might not do them all, but I like to write them down, I am more productive when I have concrete objectives.

  1. Translate my Debian book into English.
    I will run a fundraising campaign with Ulule.com and if enough people are interested, I will spend a few months with Roland Mas to translate the book. Hopefully this project can be completed until the summer.
  2. Finish multiarch support in dpkg.
    I’m working on this with Guillem Jover right now, thanks to Linaro who is sponsoring my work.
  3. Make deb files use XZ compression by default.
    It’s a simple change in dpkg-deb and it would literally save gigabytes of space on mirrors. It’s particularly important so that single CD/DVD can have a more complete set of software. #556407 (on DAK) needs to be fixed first though and a project-wide discussion is in order. Some archs might want to have a different default.
  4. Be more reactive to review/merge dpkg patches.
    While we welcome help, we have a backlog of patches sitting in the BTS and it happened several times that we failed to review/merge submitted branches in a decent time. It’s very frustrating for the contributor. I already tried to improve the situation by creating the Review/Merge Queue but nobody stepped up to help review and update the patches.
    As I am getting more familiar with the C part of dpkg, I hope to be able to jump in when Guillem doesn’t have the time or the interest.
  5. Implement the rolling distribution proposed as part of the CUT project and try to improve the release process.
    I really want the new rolling distribution but it will inevitably have an impact on the release process. It can’t be done as a standalone project. I would also like to see progress in the way we deal with transitions (see discussion here).
  6. Work more regularly on the developers-reference.
    Hopefully I will be able to combine this with my blog writing activities, i.e. write blog articles on the topics that the developers-reference shall cover and then adapt my articles with some docbook markup.

To the above list, I shall add a few supplementary goals related to funding my Debian work:

  1. Write a 10-lesson course called “Smart Package Management”.
    It will delivered by email to my newsletter subscribers.
  2. Create an information product (most likely an ebook or an online training) and sell it on my blog.
    The precise topic has not yet been defined although I have a few ideas. Is there something that you would like me to teach you? Leave your suggestions in a comment.
  3. By the end of the year, have at least 1/3 of my time funded by donations and/or earnings of my information products.
    More concretely it means 700 € each month or a 9-fold increase compared to the current income level (around 80 €/month mostly via Flattr).

That makes up lots of challenges for this year. You can help me reach those goals, join my Debian Supporters Guild and you’ll be informed every time that I start something new or when you can help in specific ways.

Librement: a new way to help people who want to contribute to free software

Find your way in the free software worldI have this project in my head, I want to work on it but I always lack the time. In order to go forward, I thought I could write about it, at least it would let me clarify my ideas and the core goals. So here I am, I will present you Librement (I have registered the alioth project but it’s empty).

The core goal is to make it easy for every user to contribute to free software in some way. I will now present the main features that I envision.

Defining skills and interests

In order to propose tasks that the user can do, we must have an idea of his skills. So on the first run (and later through a preferences menu) the user will be invited to define his skills:

  • his native languages (multiple allowed)
  • other languages he can understand
  • programming languages he knows
  • version control systems he can use
  • markup language he knows (HTML, DocBook, Wiki-like formats, etc.)
  • etc.

Maybe we can also ask which skills he would like to learn. Because contributing to free software is a nice opportunity to learn new skills!

We should also find out what the user is interested in. What are his favorite free software projects? What kind of contributions would he like to do (documentation, translation, coding, bug fixing, bug triaging, creating artwork, donations, etc.)?

Choose activities and pick concrete tasks

Based on the user’s skills and his interests, the software shows a list of possible activities. The user can then sort that list, from the most interesting one to those that he doesn’t want to do.

Each activity can generate concrete tasks. For example, the activity “Do translation for Debian” could generate a task “Translate strings in debconf/fr.po” or “Review translations in partman/fr.po”.

Work on tasks

When the user decides to work on a task, a step-by-step assistant helps him/her. It can automate some steps and provide explanations for the remaining ones, for example in the case of a translation for Debian:

  • grab the PO file (from a VCS, from an HTTP URL, from a translation server, etc.);
  • select and install a software to work with PO file (if not already done);
  • edit the PO file with the preferred program;
  • check the PO file (is it complete? is there no mistakes like missing substitutions?);
  • send back the completed PO file in a mail to the Debian bugtracking system.

If the tasks is not completed in one go, the user can resume it the next time.

Each free software project must provide some meta-information describing the various workflows involved for contributing to the different parts of the project. If necessary the project can also provide new plugins to support new operations that are not available in the default library.

Setting goals

In order to keep the user motivated, the software could track how much time he spent contributing to free software and it could verify if the user reached the goals he picked up for himself. Maybe it can also hook into the OMG Trophy Awarding System.

The sky is the limit

I hope that you now have a clearer idea of what this desktop application is supposed to be. There are literally hundreds of ways to contribute to free software and I like the idea that we can streamline the process for most users.

All the plugins implementing activities can use local information (list of packages installed with their versions, configuration settings, etc.) to propose tasks targetted to the user and highly beneficial for the corresponding free software projects. For example, a bug tagged unreproducible might benefit from a few more users trying to reproduce it. The software could direct the user to this bug report if it detects that he/she runs the same version on the same architecture and that this software is regularly run on the system.

Many projects have created “operations” or “events” to encourage people to contribute, they could all be implemented as dedicated activities in Librement. I’m thinking of stuff like Gnome Love, Ubuntu’s 5-a-day, Ubuntu’s 100 papercuts, etc.

Even for people who have no time to contribute, the application can still be useful by referencing the various ways to donate money (or material) to projects that they are using.

Feedback welcome

I’m excited by the potential of such an application, but it’s normal since it’s my idea. Do you believe it can be useful and popular? Do you have ideas of exciting activities that such a framework can offer?

PS: If you wonder how I came up with the name “Librement”, here’s the explanation. It’s a French word which means “freely”. And users who want to give back are trying to live up to the principles of free software, which I sum up by “they are trying to live freely”.

Follow me on Identi.ca, Twitter and Facebook. Or subscribe to this blog by RSS or by email.

Debian related goals for 2010

Here’s stuff that I’d like to do this year, more or less by decreasing order of importance:

  • translate my Debian book into English and get it published;
  • finish the cleanup of the perl API in dpkg-dev in order to create libdpkg-perl;
  • create dpkg-buildflags to export default build flags that packages should use (and get rid of the code setting those environment variables in dpkg-buildpackage), needed to properly fix #489771;
  • ensure the new source formats continue to gain acceptance by improving whatever is needed;
  • design a generic vcs-buildpackage infrastructure to be integrated in dpkg-dev. This design will probably happen through a DEP (Debian Enhancement Proposal) to ensure we have had proper discussion before someone gets to the implementation;
  • continue fixing dpkg bugs faster than they are reported;
  • enhance our infrastructure to ease interaction between contributors and to have a better view of how each package is maintained (see my last blog entry on this topic);
  • update the developers-reference where needed and fix some of the numerous wishlist bugs;
  • rewrite in C the last perl scripts provided by the dpkg binary package (update-alternatives/mksplit mainly, for dpkg-divert there’s a preliminary patch available already) so that it’s easier to build a minimal system without perl-base;
  • integrate the 3-way merge tool for Debian changelogs in dpkg-dev;

All of this probably doesn’t fit in my free time (being a father since last month does not help increasing my free time :-)), so if you’re interested in seeing one or more of those projects completed, and if you know some person/company that could sponsor them, get in touch with me!

Flights for Debconf

I have booked my flights for Debconf9 in July. I will arrive in Madrid on July 23th at 10:30 from Lyon Saint-Exupéry (LYS-MAD, flight AF5891) and I will leave on July 31th at 17:40 (MAD-LYS, flight AF5892). I plan to use the train to go to Cáceres but it’s too early to buy tickets on renfe.es. I also have no idea how far the train station is (from the airport) but it looks like I will have several hours transit time anyway. There aren’t so many trains for Cáceres. The train tickets will likely cost around 30 EUR each.

I’m glad that I can attend again this year. I’m sure it will be very productive. At least concerning dpkg it will be good to meet Guillem Jover IRL.

Resurrecting projects

During this week-end I tried to give initial impulses on several projects of interest for me :

DebianEduFrench: a new mailing list has just been setup for this project. We intend to foster cooperation between several French educational Debian-based projects. We’ll try to integrate their work into Debian directly to avoid needless duplication of effort. This follows my last call for help on debian-devel-announce. If you speak French and have good packaging skills, you’re welcome to join and help us on the mailing list.

Collaborative maintenance infrastructure: this one started as a project to handle orphaned packages but after the talk we gave at the Debian-QA meeting in Darmstadt, we agreed that it is of broader interest: in particular for mentoring future Debian developers. This tool could also interest Ubuntu which has in fact already started something covering a part of this proposal: REVU (development web site). So I crosspossted a call in several Debian & Ubuntu lists in order to create a little team to work on this project and make it happen. But there’s still room for discussions as nothing is in stone yet …

That’s enough for one week-end !