News

Kali Linux 1.0, a new Debian derivative

March 13, 2013 by Raphaël Hertzog

Today, during Blackhat Europe, Offensive Security announced the availability of Kali Linux 1.0, which aims to be the most advanced, robust, and stable penetration testing distribution to date. It is the successor of Backtrack Linux.

Kali’s choice of Debian

Kali’s release is a significant event in the security auditing and penetration testing field, and I’m proud to see that Debian was retained as the best distribution to create this new product. Here’s what Mati Aharoni of Offensive Security told me:

Debian provides a reliable base to build a new distribution and yet can easily be customized to add bleeding edge features, thanks to the unstable and experimental distributions.

Kali’s development policies

Even though Kali was prepared in secret, from now on Kali’s development happens in the open in public git repositories. There are repositories for all the packages that have been created (or forked) as well as for the ISO images creation script.

Debian packages are maintained with git-buildpackage, pristine-tar and the associated helper tools, making it easy to integrate the latest changes of Debian.

Kali packaged several hundreds tools that relate to their field and they intend to contribute those which are DFSG-free back to Debian.

Kali’s technical infrastructure

In the last year, I have been working within the Kali team to setup large parts of their infrastructure as a proper Debian derivative.

Kali’s main ISO images are built with live-build. All the bugfixes that I contributed to Debian Live were the direct result of my work for Kali.

The git repositories are managed with gitolite. The package repositories are built with reprepro. The build daemons use rebuildd and sbuild.

The (push) mirrors are synchronized with the same tools than Debian (based on rsync), but there’s also a central server which redirects to a mirror close to you (and which is used by default everywhere). This one runs mirrorbrain (and not Raphaël Geissert’s redirector).

The ARM build daemons (armel/armhf) run on machines powered by Calxeda’s Highbank (4 cores, 4 GB RAM) that work pretty well. Even better, Offensive Security is willing to dedicate one node of this “cluster” for Debian’s own usage.

The future

This first release is not an end. It’s only the start of a journey. Not all applications have been packaged yet and there’s lot of work left to integrate everything in Debian.

I’m really looking forward to continue my collaboration with the Kali team as this has been one of the most interesting project I ever had as a Debian consultant. And also one of the few where I could really contribute something back to Debian.

My Debian activities in October 2011

November 1, 2011 by Raphaël Hertzog

This is my monthly summary of my Debian related activities. If you’re among the people who made a donation to support my work (130.30 €, thanks everybody!), then you can learn how I spent your money. Otherwise it’s just an interesting status update on my various projects.

Dpkg work

The month started with fixing newly reported bugs to prepare the 1.16.1.1 release:

#644492: there was a flaw in a change I made to the trigger setup code. This resulted in packages being incorrectly marked as configured while they were only unpacked in a new chroot.
#642656: dpkg-source’s refusal to build when it detects unrecorded changes broke the (mostly unused, except by the lintian test suite apparently) “2.0” source format.
#644412: the Makefile snippet “buildflags.mk” did not respect the new maintainer specific environment variables (like DEB_CFLAGS_MAINT_APPEND) because make does not forward environment variable through $(shell …). Fixed that by manually exporting the required variables.
I also disabled dpkg-buildpackage’s output of the build flags since it was confusing several maintainers. dpkg-buildpackage invokes debian/rules and it has no (clean) way to discover the build flags changes that maintainer request by setting the dedicated environment variables in debian/rules. Maintainers expect to see the build flags with the modifications they have requested and not just the default values set by the distribution.

With the help of Guillem, we decided on a proper fix for a race condition sometimes triggered by parallel builds when 2 concurrent dpkg-gencontrol try to update debian/files (see #642608). This ended up requiring a new package (libfile-fcntllock-perl) that the Debian perl team kindly packaged for us. With all this sorted, it was a rather easy fix.

Multiarch progress

I also spent lots of time on multiarch. I fixed an old bug that requested to support the multi-arch paths in case of cross-building (see #595144), the discussion was not really conclusive on which of the two proposed patches was better so I ended up picking my own patch because it was closer to how we currently deal with cross-building. Then I fixed 2 issues that have been reported on Ubuntu’s dpkg. The first one (LP #863675) was rather severe since an installed package ended being “disappeared” in favor of its foreign counterpart that was removed (but that had some config files left). The second one (LP #853679) only affected dselect users (apparently there are still some!) who had a self-conflicting library (Provides: foo, Conflicts: foo) installed for multiple architectures.

But the bulk of the time spent on multiarch has been spent discussing with various parties on how to go forward with multiarch. The release team commented on the schedule of the merge to ensure it makes it into Wheezy, and the Debian project leader also commented on the problems encountered so far.

While not the best course of action I could have hoped for, it certainly helped since Guillem started pushing some reviewed commits. Out of the 66 commits that were in my pu/multiarch/full branch one week ago, 20 have been merged in the master branch already.

Python-django security update and RC bug

Since python-django’s maintainer did not manage to prepare the required security updates, I stepped in and prepared version 1.2.3-3+squeeze2 for Squeeze and 1.0.2-1+lenny3 for Lenny. Unfortunately this security update is an example of how an inactive maintainer is likely to result in a severe delay for the release of security updates.

Furthermore in this specific case, the security team did not want to release the Squeeze security update until the Lenny one had been investigated (which required some time since upstream no longer supports the version in Lenny) but they did not make this very clear.

Later another release critical bug had been filed against the package (#646634) but after investigation, it turned out to be a local configuration problem so I downgraded it. I still forwarded the test suite failure to upstream authors since the test could be enhanced.

In any case, co-maintainers for python-django are welcome. I really preferred the situation where I can quietly sit down as backup maintainer… 🙂

WordPress packaging

WordPress sounds similar to python-django. I’m also “only a backup maintainer” but Giuseppe has been inactive for many months and I had to step in August because I wanted the new upstream version. I discovered a bit late that I was not subscribed to wordpress’ bugs and thus the release critical bug #639733 (that I introduced with my new upstream version) went unattended for a rather long time. Once aware, though, I quickly fixed it.

I also took the opportunity to start a discussion on debian-devel about how to deal with embedded javascript libraries and proposed a mechanism of “opportunistic replacement with symlinks”. WordPress is my testbed package for this mechanism, you can check out its debian/dh_linktree that implements the replacement logic.

The discussion has not been very interesting but at least I learned that Debian now requires that each source package shipping minified javascript files includes the original files too. It’s somewhat of a pain since it’s not a license requirement in many cases (many of those libraries are not under the GPL), but just a Debian requirement that many upstreams are not complying with. WordPress is affected and Jakub Wilk thus opened #646729 which is going to be a long-standing RC bug. To give good measures, I spent several hours investigating the case of each javascript file in the WordPress source package and I filed a new ticket on the upstream bugtracker.

Dropbox packaging work

A few months after the introduction of nautilus-dropbox to Debian and Ubuntu, I can say that the decision to only support the download of dropbox in the postinst has been a mistake. Because of this decision I had to make the postinst fail if the download failed. Even if the error message is relatively clear, this lead to many (mostly automated) bug reports on the Ubuntu side. Various other problems cropped up on top of this (trying to start dropbox while the package was not configured would result in an error because the user did not have the required rights to install the software, reinstalling the package while dropbox was running would result in a failure too, etc.).

I have fixed all those issues in the version 0.7.0-2 of the package. Now if the user has to install dropbox, it will use PolicyKit to request the root rights. The postinst will no longer fail if the dropbox download fails since it can be run later by the user. And I fixed the download code to remove the replaced file before unpacking a new file (insead of overwriting the existing file). All this work has been forwarded upstream.

The Debian Administrator’s Handbook Update

I’m glad to tell you that the translation will happen because we reached the minimal funding goal on October 22th with the help of 380 supporters.

Now the fundraising continues, but this time the goal is the liberation of the resulting book. For this to happen, we need to reach 25000 EUR in the liberation fund. So far we’re at 37% of this goal with 9400 EUR in the liberation fund (which means that 59% of the money raised has been put in the liberation fund).

Click here if you want to contribute towards the liberation of this book.

With (less than) 27 days left, it’s going to be a challenge to meet the goal, but we do like challenges, don’t we?

Misc work

I filed #644486 against dh-make so that new packages have proper support of dpkg-buildflags from the start.
I merged lots of patches from Luca Falavigna in the developers-reference.
I discussed debtags integration in the PTS with Enrico Zini and Paul Wise.
I updated publican’s packaging for the new upstream version 2.8. I had to write a new patch that I forwarded upstream.
I filed an upstream bug on hamster-applet because just running hamster-time-tracker no longer brings its window forward.

Thanks

See you next month for a new summary of my activities.

The promising OpenMediaVault failed its debut as free software project

October 31, 2011 by Raphaël Hertzog

Volker Theile is well known for his work on FreeNAS. But at one point, he decided to give up on it and to restart from scratch but this time based on Debian (instead of FreeBSD). OpenMediaVault is the name of this new Debian based product.

Here’s how he defines the project:

OpenMediaVault is the next generation network attached storage (NAS) solution based on Debian Linux. It contains services like SSH, (S)FTP, SMB/CIFS, DAAP media server, RSync, BitTorrent client and many more. Thanks to the modular design of the framework it can be enhanced via plugins.

OpenMediaVault is primarily designed to be used in home environments or small home offices, but is not limited to those scenarios. It is a simple and easy to use out-of-the-box solution that will allow everyone to install and administrate a Network Attached Storage without deeper knowledge.

Even though all the work was private, he managed to attract an important following, and I must say that I was looking forward to this project. He regularly blogged on his progress, sharing some good-looking video of the resulting product (example here).

The first public release (Version 0.2, codenamed Ix) happened on October 17th. I have yet to try it but I took a look on the website. As a Debian developer, I was keen on seeing the source code and how the project was managed.

The GPLv3 license is presented as an important feature and I was expecting a well-managed open source project. The fact that it was a private one-man project up to now did not bother me, we’re quite used to the “scratch your itch” kind of start for free software projects.

Enough to say is that I have been very disappointed. First you come across a contributor agreement, it’s frowned upon by many free software developers. But why not, maybe he bought the argument of Mark Shuttleworth and wants to give it a try.

But then I looked at the subversion repository, it’s obvious that it’s just a dumping ground of files that are managed somewhere else in another repository. A bit like Android which is not developed in the open but released as a whole from time to time.

But the worst was yet to be found on the licensing page:

Beside being freely available for personal end-users, System Builders and System Integrators, in general Installers, require a commercial license for OpenMediaVault.

Besides the fact that I’m still not sure what this means, I would like to know how he reconciles this requirement with the terms of the GPLv3.

It’s a shame that a so promising project ends up being a disaster from a free software perspective.

Have you tried OpenMediaVault already? If yes, I’d be glad to read your thoughts in the comments.

Trying to make dpkg triggers more useful and less painful

May 30, 2011 by Raphaël Hertzog

Lately I have been working on the triggers feature of dpkg. I would like to share my plan and what I have done so far. I’ll first explain what triggers are, the current problems, and the work I did to try to improve the situation.

Introduction

Dpkg triggers are a neat feature of dpkg that package can use to send/receive notifications to/from other installed packages. Those notifications take the form a simple string.

This feature is heavily used to track changes of packaged files in a list of predefined directories, and to update other files based on this. For instance, man-db is watching the directories containing manual pages so that it can update its cache (in /var/cache/man/). install-info is updating the index of info pages when there have been changes in /usr/share/info. gnome-menus is updating its own copy of the menu hierarchy (with entries from /etc/gnome/menus.blacklist blacklisted) every time that a .desktop file is installed/updated/removed.

From a user’s perspective

You see triggers in action very often during upgrades (in fact too often as we’ll see it later):

Preparing to replace zim 0.52-1 (using .../archives/zim_0.52-1_all.deb) ...
Unpacking replacement zim ...
Processing triggers for shared-mime-info ...
Processing triggers for menu ...
Processing triggers for desktop-file-utils ...
Processing triggers for man-db ...
Processing triggers for hicolor-icon-theme ...
Processing triggers for python-support ...
Processing triggers for gnome-menus ...
Setting up zim (0.52-1) ...
Processing triggers for python-support ...
Processing triggers for menu ...

As you guessed it, those “Processing triggers” lines correspond to the packages which received (one or more) trigger notifications and which are doing the corresponding task.

By default the triggers are processed at the end of the dpkg --unpack invocation which is often too soon because APT will often call dpkg --unpack repeatedly during important upgrades. There are some options to ask APT to use dpkg’s --no-triggers option in order to defer the trigger processing at the end of the APT run. You can put this in /etc/apt/apt.conf.d/triggers:

// Trigger deferred
DPkg::NoTriggers "true";
PackageManager::Configure "smart";
DPkg::ConfigurePending "true";
DPkg::TriggersPending "true";

I have now asked APT maintainers to use those options by default, I filed bug #626599 to track this. At the same time I fixed bug #526774 reported by APT maintainers. This bug forced them to put a work-around in APT which resulted in running triggers sooner than expected.

(And while writing this article I filed bug #628564 and #628574 because it was clearly not normal that the menu triggers was executed twice for the installation of a single package)

From a packager’s perspective

The implementation of triggers has several consequences on the status that packages can have.

Let’s assume that the package A installs a file in a directory that is watched by package B (and that B is currently in the “installed” state). When A is unpacked, dpkg adds B to its “Triggers-Awaited” field and lists the activated trigger in B’s “Triggers-Pending” field. Package A is in “unpacked” state, but B has been changed to “triggers-pending”.

When A is configured, instead of going to the “installed” state, it will go to the “triggers-awaited” state. In that state the package is assumed to NOT fulfill dependencies. However, B—which is still in “triggers-pending” state—does fulfill dependencies.

A and B will switch to “installed” at the same time when the trigger has been processed.

The fact that the triggers-awaited status does not fulfill dependencies means that some common triggers like man-db have to be processed regularly just to be able to ensure dependencies are satisfied before running the postinst of other installed packages.

But a package which ships a manual page can certainly be considered as configured when its postinst has been run even if man-db has not yet updated its cache to know about the new/updated manual page.

When you activate a trigger with the dpkg-trigger command you have an option --no-await to avoid awaiting the trigger processing (and thus to go directly to installed state after the postinst has been run). But with file triggers or activate trigger directives, you do not have this option.

My proposal to improve the situation

This is the problem that I tried to solve during my last vacation. But before changing the inner working of triggers, I wrote a non-regression test suite for that feature (commit here) so I could hack with some confidence that I did not break everything.

The result has been presented on the debian-dpkg mailling list: see the discussion here. I added new directives that can be used in triggers files that work exactly like the current triggers except that they do not put triggering packages in trigger-awaited status.

I believe the code to be mostly ready, but in its current form the patch brings zero benefits until all packages have been converted to use the trigger variants that do not require awaiting trigger processing (and the change requires a pre-dependency on dpkg to ensure we have the required dpkg that understands the new kind of trigger directives).

Remaining question

Thus I wonder if I should not change the default semantic of triggers. The packages which really provide crucial functionality to awaiting packages through triggers would then have to be updated to switch to the new directives.

If you’re a packager using triggers, you can thus help me by answering this question: do you know some triggers where it’s important that the awaiting packages are not considered as configured before the trigger processing? In most of the cases I checked, it’s important for the triggered package rather than for the triggering package.

In truth, a package in triggers-awaited status is usually in a good enough shape to be able to satisfy dependencies (i.e. requirements that other packages can have), but it would still be worth to record the fact that it’s not entirely configured yet because it might be true from the user’s point of view: for example if the menu trigger has not yet been processed, the software might not yet be visible in the application menu.

If you appreciate this kind of groundwork that benefits to the whole Debian ecosystem, please consider supporting my work. Click here and give it a look, there are many ways to contribute and to make a difference for me.