Debian Cleanup Tip #6: Remove automatically installed packages that are no longer needed

Last week we learned how to identify cruft on your Debian system. This week, for the last article in this series, we’ll learn more about automatically installed packages and how to get rid of them when you don’t need them any longer.

APT tracks automatically installed packages

When you install a new package with apt-get/aptitude/synaptic, it’s very common to end up installing many more packages: those are the dependencies of the installed package. Here’s an example:

$ sudo apt-get install pino
[...]
The following extra packages will be installed:
  libdbusmenu-glib1 libgee2 libindicate4 libnotify1 notification-daemon
The following NEW packages will be installed:
  libdbusmenu-glib1 libgee2 libindicate4 libnotify1 notification-daemon pino
0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.
Need to get 478 kB of archives.
After this operation, 2531 kB of additional disk space will be used.
Do you want to continue [Y/n]? 

After this installation, the 5 “extra packages” will be marked as “automatically installed”. What does this mean? It means that you have not explicitly requested their installation and that the system should be free to remove them as soon as they are no longer needed.

You can verify that this is effectively the case with “apt-mark showauto” (it returns a list of the automatically installed packages).

$ apt-mark showauto |grep libdbusmenu
libdbusmenu-glib1
$ apt-mark showauto |grep pino
$

Aptitude shows this information with the “A” letter in its interactive interface and in the “aptitude search” output. “aptitude show” has a dedicated field for this:

$ aptitude show libdbusmenu-glib1
Package: libdbusmenu-glib1               
New: yes
State: installed
Automatically installed: yes
Version: 0.3.7-1
[...]

In Synaptic, it’s not very visible but once you have selected an installed package, you can verify in the “Package” menu whether “Automatically installed” is checked or not.

APT tells you which packages are no longer needed

Over time, some of those automatically installed packages become unnecessary because the packages that depended on them no longer do. It might be that they are using a newer version of the same library, or they switched to use something else, or they are able to do the task themselves.

Whatever the reason, the original dependency has vanished and the automatically installed package is no longer needed on the system.

Aptitude will automatically remove those unneeded packages the next time you run it but apt-get and synaptic do not.

Apt-get will inform you that some packages are no longer needed and will even tell you how you can get rid of them:

$ sudo apt-get remove pino
[...]
The following packages were automatically installed and are no longer required:
  notification-daemon libdbusmenu-glib1 libnotify1 libgee2 libindicate4
Use 'apt-get autoremove' to remove them.
The following packages will be REMOVED:
  pino
0 upgraded, 0 newly installed, 1 to remove and 219 not upgraded.
After this operation, 1225 kB disk space will be freed.
Do you want to continue [Y/n]? 
[...]
$ sudo apt-get autoremove
[...]
The following packages will be REMOVED:
  libdbusmenu-glib1 libgee2 libindicate4 libnotify1 notification-daemon
0 upgraded, 0 newly installed, 5 to remove and 219 not upgraded.
After this operation, 1307 kB disk space will be freed.
Do you want to continue [Y/n]? 
[...]

Synaptic will show you the packages that can be removed in a new section name “Installed (auto removable)” if you select the “Status” button in the bottom-left pane.

It’s thus a good habit to get rid of those unneeded package from time to time.

Use this feature to trim down your system

While APT usually sets the “Automatically installed” flag, you can also set it manually. It’s a very simple way to tell the system “I don’t need this package directly, feel free to remove it if nothing else requires it”.

# With apt-get
$ sudo apt-mark markauto libxml-simple-perl
# Or with aptitude
$ sudo aptitude markauto libxml-simple-perl

You can also do it in the interactive interface of aptitude with the key “M” (and “m” for unmarking). To do it in Synaptic, you have to use the menu entry “Package > Automatically installed”.

Many users like to have a minimal set of packages installed but they don’t really know which packages are really important and trying to remove every package to look what happens is cumbersome.

Thanks to this feature, you don’t try removing packages but you flag them as automatically installed. There is few risks in doing so when it concerns libraries (including python/perl modules). If the package is not indirectly needed by one of your important packages, it will be removed by apt-get autoremove, otherwise it’s kept for as long as it’s needed.

I would suggest to not mark as such packages of priority higher or equal to important to avoid nasty surprises (although I say this to not be blamed in case you remove too much, in theory the system should not remove essential components and all dependencies should be complete).

Also be aware of the consequences when you mark “task” packages like “gnome” as automatically installed… it will suggest you to remove your whole desktop. If you want to trim down the default desktop, you should “unmark” the desktop packages that you want to keep:

$ sudo apt-mark unmarkauto gnome-session gnome-panel

Do you want to read more tutorials like this one? Click here to subscribe to my free newsletter, you can opt to receive future articles by email.

Debian Cleanup Tip #5: identify cruft that can be removed from your Debian system

Last week we learned how to identify and restore packages whose files have been corrupted. This time we’ll concentrate ourselves on the non-packaged files…

Non-packaged files

They are files which are not provided by a Debian package, or in other words, files where dpkg --search finds no associated package:

$ dpkg --search /srv/cvs
dpkg-query: no path found matching pattern /srv/cvs

You always have such files on your system, at least all your own files in /home. But many daemons also create files as part of their work (and they are usually stored in /var): internal files for a database server, mail spool for a mail server, etc. Those are normal and you want to leave them alone.

But you might have non-packaged files in /usr and that should not be the case if you install everything from packages. It would thus be useful to be able to list those files in order to detect a software that has been manually installed.

Manually installed software is not a good idea

Such an installation might cause troubles for example by taking precedence over the same software provided in a Debian package. Over time the local installation will not be upgraded while the packaged one will.

The other packages which depend on this software will believe they have the latest version since their dependency is satisfied but in fact they are using the older version since it takes precedence.

So you want to get rid of those? Let’s see how we can find them.

Use cruft to identify non-packaged files

As I explained above, there are many non-packaged files that are legitimate and that you don’t want to remove. That’s why cruft does something more elaborated than a scan of the filesystem and a check of dpkg’s database.

It provides a way for packages to say which files they might legitimately create during run-time and that cruft should not report. And it knows of many such files. But it’s far from exhaustive and definitely not up-to-date.

So you should always take its output with suspicion and consider twice where the file came from. Do not trust it blindly to remove the files… you have been warned.

How to use cruft

You should give it a list of directories to ignore to reduce the noise in the output, for example like this:

$ sudo cruft -d / -r report --ignore /home --ignore /var --ignore /tmp
$ less report
cruft report: mercredi 23 février 2011, 15:45:34 (UTC+0100)

---- missing: ALTERNATIVES ----
        /etc/alternatives/cli-csc.1.gz
        /usr/share/man/man1/cli-csc.1.gz
---- missing: dpkg ----
        /etc/xdg/autostart/gnome-power-manager.desktop
        /usr/lib/libpython2.6_d.so.1.0-gdb.py
        /usr/share/fonts/X11/100dpi
        /usr/share/fonts/X11/75dpi
---- unexplained: / ----
        /boot
        /dev
        /etc/.java
        /etc/.java/.systemPrefs
[...]
        /usr/lib/pymodules/python2.6
        /usr/lib/pymodules/python2.6/.path
        /usr/lib/pymodules/python2.6/Brlapi-0.5.5.egg-info
[...]

Note that it doesn’t traverse filesystems so if your /usr is on another partition than /, you will need to use the option -d "/ /usr" to have it scan both.

Analyze the report

Now you can quietly go through the report that has been generated and decide which files need to be removed or not. The report also contains missing files (files which should exist according to the dpkg database but which are not there) but the bulk of the listing will be in the “unexplained” section: files which are not part of any package (and whose presence is not explained by any other explain script that packages can ship).

Again take this with great suspicion, and you should rather not delete a file if you don’t know it got there in the first place. For instance, on my system it lists many files below /usr/lib/pymodules/ and those are legitimate: they come from Debian packages but they are copied there dynamically from /usr/{lib,share}/pyshared in order to support multiple python versions. If you remove those files, you effectively break your system.

You will also find many .pyc files created by python packages, they are a byte-compiled version of the corresponding .py file. Removing them breaks nothing but you loose a bit of performance.

On the opposite, most of the files below /usr/local/ are likely the result of some manual software installation and those should be safe to remove (if you know that you are not using the corresponding software).

Conclusion: useful but needs work

In summary, you can use cruft to identify non-packaged files and maybe learn a bit more about what got manually installed on the system, but it requires some patience to go through the report as many of the files reported are false positives.

Yes, cruft badly needs supplementary volunteers to cope with the many ways packages legitimately generate non-packaged files. It’s not even complicated work: the package is mostly in shell and in Perl, and /usr/share/doc/cruft/README.gz explains how it all works.

Do you want to read more tutorials like this one? Click here to subscribe to my free newsletter, you can opt to receive future articles by email.

Debian Cleanup Tip #4: find broken packages and reinstall them

Last week, we learned to get rid of third-party packages, now we’re going one step further: we’ll verify if the files of the installed packages are still exactly like they were when they got installed.

If you’re a tinkerer and hand-edit some files for some quick tests, or if you tend to re-install newer versions of some packages from the sources, you might have overwritten some packaged files and it would be good to be able to detect this (and remedy to the problem). debsums is the tool that makes it possible.

Use debsums to identify modified files

I often use debsums when I take over the maintenance of a Debian server because I want to verify which files have been modified by the former administrator.

Without any argument, debsums is very verbose, it will list every installed file (except configuration files) and tells whether it’s unmodified (“OK”) or not (“FAILED”).

$ sudo debsums
/usr/bin/a2ps                                               OK
[...]

With the --all option, it will verify all files including configuration files. With --config it will verify only the configuration files.

With the --changed option, debsums will only list modified files among those inspected. The following invocation will thus list all files which have been modified on the system and which are not configuration files.

$ sudo debsums --changed
/usr/lib/perl5/AptPkg/Config.pm
/usr/lib/perl5/AptPkg.pm
[...]

Find out the package affected and reinstall it

debsums told me that /usr/lib/perl5/AptPkg.pm was modified. Indeed I remember having manually installed a modified version of that perl module for a quick test.

I find out the affected package with dpkg --search /usr/lib/perl5/AptPkg.pm: it’s libapt-pkg-perl.

Now I just have to reinstall this package to overwrite the modified files with the original ones:

$ sudo aptitude reinstall libapt-pkg-perl
[...]
# Or with apt-get
$ sudo apt-get --reinstall install libapt-pkg-perl
[...]

You might have to repeat the process until debsums no longer reports any modified file.

Do you want to read more tutorials like this one? Click here to subscribe to my free newsletter, you can opt to receive future articles by email.

Debian Cleanup Tip #3: get rid of third-party packages

Last week, we learned how to get rid of obsolete packages. This time, we’re going to learn how to bring back your computer to a state close to a “pure” Ubuntu/Debian installation.

Thanks to the power of APT, it’s easy to add new external repositories and install supplementary software. Unfortunately some of those are not very well maintained. They might contain crappy packages or they might simply not be updated. An external package which was initially working well, can become a burden on system maintenance because it will be interfering with regular updates (for example by requiring a package that should be removed in newer versions of the system).

So my goal for today is to teach you how to identify the packages on your system that are not coming from Debian or Ubuntu. So that you can go through them from time to time and keep only those that you really need. Obsolete packages are a subset of those, but I’ll leave them alone. We took care of them last week.

Each (well-formed) APT repository comes with a “Release” file describing it (example). They provide some values that can be used by APT to identify packages contained in the repository. All official Debian repositories are documented with Origin=Debian (and Origin=Ubuntu for Ubuntu). You can verify the origin value associated to each repository (if any) in the output of apt-cache policy:

[...]
 500 http://ftp.debian.org/debian/ lenny/main i386 Packages
     release v=5.0.8,o=Debian,a=stable,n=lenny,l=Debian,c=main
     origin ftp.debian.org
[...]

From there on, we can simply ask aptitude to compute a list of packages which are both installed and not available in an official Debian repository:

$ aptitude search '?narrow(?installed, !?origin(Debian))!?obsolete'
or
$ aptitude search '~S ~i !~ODebian !~o'

You can replace “search” with “purge” or “remove” if you want to get rid of all the packages listed. But you’re more likely to want to remove only a subset of carefully chosen packages… you’re probably still using some of the software that you installed from external repositories.

With synaptic, you can also browse the content of each repository. Click on the “Origin” button and you have a list of repositories. You can go through the non-Debian repositories and look which packages are installed and up-to-date.

But you can do better, you can create a custom view. Click on the menu entry “Settings > Filter”. Click on “New” to create a new filter and name it “External packages”. Unselect everything in the “Status” tab and keep only “Installed”.

Go in the “Properties” tab and here add a new entry “Origin” “Excludes” “ftp.debian.org”. In fact you must replace “ftp.debian.org” with the hostname of your Debian/Ubuntu mirror. The one that appears on the “origin” line in the output of apt-cache policy (see the excerpt quoted above in this article).

Note that the term “Origin” is used to refer to two different things, a field in the release file but also the name of the host for an APT repository. It’s a bit confusing if you don’t pay attention.

Close the filters window with OK. You now have a new listing of “External packages” under the “Custom Filters” screen. You can see which packages are installed and up-to-date and decide whether you really want to keep it. If the package is also provided by Debian/Ubuntu and you want to go back to the version provided by your distribution, you can use the “Package > Force version…” menu entry.

Click here to subscribe to my free newsletter and get my monthly analysis on what’s going on in Debian and Ubuntu. Or just follow along via the RSS feed, Identi.ca, Twitter or Facebook.

Debian Cleanup Tip #2: Get rid of obsolete packages

Last week, we learned to remove useless configuration files. This week, we’re going to take care of obsolete packages.

An obsolete package is a package who is no longer provided by any of the APT repositories listed in /etc/apt/source.lists (and /etc/apt/sources.list.d/). There can be multiple reasons why a package is no longer available in the repository (or at least not under the same name) :

  • the upstream author stopped maintaining the software a long time ago, nobody else took over and the Debian maintainer preferred to remove the package from Debian. Usually there are alternatives in the Debian archive.
  • the package was orphaned in Debian since a long time, nobody took over and it had very few users. The Debian QA team might have asked its removal.
  • the latest version of the software might have been packaged under a new package name. Either because the amount of changes was so important that it was preferred to not upgrade automatically to the latest version (it has been the case with request-tracker and nagios, they both embed a version number in their package names), or simply because the maintainer wants to let the user install several versions at the same time (that’s the case for example with the Linux kernel, the python interpreter and many libraries).
  • the software has been renamed, the maintainer renamed the packages and kept transitional packages under the old name for one release. Then the transitional packages have been removed.

In any case, it’s never a good idea to keep obsolete packages around: they do not benefit from security updates and they might cause problems during upgrades if they depend on other packages that should be removed to complete the upgrade.

You could blindly remove them with aptitude purge ~o (or aptitude purge ?obsolete) but you might want to first verify what those package are. There might be some packages that you have manually installed, that are not part of any current APT repository, and that you want to keep around nevertheless (I have skype, dropbox and a few personal packages for example). You can get the list with aptitude search ?obsolete

With the graphical package manager (Synaptic), you can find the list of obsolete packages by clicking on the “Status” button and selecting “Installed (local or obsolete)”. You can then go through the list and decide for each package whether you want to keep it or not.

Follow me on Identi.ca, Twitter and Facebook. Or subscribe to this blog by RSS or by email.

Debian Cleanup Tip #1: Get rid of useless configuration files

If you like to keep your place clean, you probably want to do the same with your computer. I’m going to show you a few tips over the next 4 weeks so that you can keep your Debian/Ubuntu system free of dust!

Over time the set of packages that is installed on your system changes, either because you install and remove stuff, or because the distribution evolved (and you upgraded your system to the latest version).

But the Debian packaging system is designed to keep configuration files when a package is removed. That way if you reinstall it, you won’t have to redo the configuration. That’s a nice feature but what if you will never reinstall those packages?

Then those configuration files become clutter that you would rather get rid of. In some cases, those files lying around might have unwanted side-effects (recent example: it can block the switch to a dependency-based boot sequence because obsolete init scripts without the required dependencies are still present).

The solution is to “purge” all packages which are in the “config-files” state. With aptitude you can do aptitude purge ~c (or aptitude purge ?config-files). Replace “purge” by “search” if you only want to see a list of the affected packages.

If you want a machine-friendly list of the packages in that state, you could use one of those commands (and then pass the result to apt-get if you don’t have aptitude available):

$ grep-status -n -sPackage -FStatus config-files
[...]
$ dpkg-query -f '${Package} ${Status}\n' -W | grep config-files$ | cut -d" " -f1
[...]

Note that grep-status is part of the dctrl-tools package.

Of course you can also use graphical package managers, like Synaptic. Click on the “Status” button on the bottom left, then on “Not installed (residual config)” and you have a list of packages that you can purge. You can select them all, right click and pick “Mark for Complete Removal”. See the screenshot below. The last step is to click on “Apply” to get the packages purged.

Synaptic purging residul config files

Do you want to read more tutorials like this one? Click here to subscribe to my free newsletter, you can opt to receive future articles by email.