<html>
<head>
<title>Keeping filesystem images sparse</title>
</head>
<body>
<h2>Keeping filesystem images sparse</h2>
<p>
Filesystem images in local files can be used by many PC emulators and
virtual machines
(<a href="http://user-mode-linux.sourceforge.net/">user-mode Linux</a>,
<a href="http://fabrice.bellard.free.fr/qemu/">QEMU</a> and
<a href="http://wiki.xensource.com/xenwiki/">Xen</a>, to name but three).
Typically these filesystems are created as sparse files using
commands like:
<p>
<pre>
dd if=/dev/zero of=fs.image bs=1024 seek=2000000 count=0
/sbin/mke2fs fs.image
</pre>
where the enormous <code>seek</code> value causes <code>dd</code> to move
forward by 2GB before writing nothing at all. This results in the
creation of a sparse file which takes disk space only for blocks which are
actually used:
<p>
<pre>
$ ls -l fs.image
-rw-rw-r-- 1 rmy rmy 2048001024 Apr 18 19:10 fs.image
$ du -s fs.image
31692 fs.image
</pre>
As the filesystem is used, more and more of the non-existent blocks are
filled with data and the size of the file on disk grows. Sometimes it would
be nice to be able to reclaim unused blocks from a filesystem image. However,
deleting files from the image doesn't return the space to the
underlying filesystem: even free blocks in the image still consume space.
Reclaiming the space can be achieved in two stages:
<ul>
<li>Fill unused blocks with zeroes
<li>Make the file sparse again
</ul>
<p>
One traditional way to zero unused blocks is to create a file that fills
all the free space:
<p>
<pre>
dd if=/dev/zero of=junk
sync
rm junk
</pre>
<p>
The disadvantage of <code>dd</code> in this context is that it destroys
any sparseness that exists: free blocks that were originally represented
as holes in the image file are replaced with actual blocks containing
zeroes.
<p>
As an alternative approach, and as practice in mucking about with ext2
filesystems, I've written a utility which scans the free blocks in an
ext2 filesystem and fills any non-zero blocks with zeroes.
The source, <a href="zerofree-1.0.3.tgz">zerofree-1.0.3.tgz</a>, is
available for download. It's also available in a git repository:
<code>git clone http://intgat.tigress.co.uk/rmy/git/zerofree.git</code>
<p>
<ul>
<li>A cautious user would run fsck on the filesytem both before and after
running zerofree.
<li>The filesystem to be processed should be unmounted or mounted read-only.
<li>The utility also works on ext3 or ext4 filesystems.
<li>Binary packages are available in the standard repositories for Debian and Fedora.
<li>The <a href="http://partedmagic.com">Parted Magic</a> live distribution includes zerofree.
</ul>
<p>
Better than either of these would be to have the guest kernel keep the free
blocks empty. My original inspiration was the
<a href="http://www.uwsg.iu.edu/hypermail/linux/kernel/0401.3/1058.html">
ext2fs privacy (i.e. secure deletion) patch</a> described in a Linux
kernel mailing list thread. I've also made use of a later patch for ext3
entitled
<a href="http://marc.theaimsgroup.com/?l=linux-fsdevel&m=113986429313502&w=2">Secure Deletion Functionality in ext3</a>
from the linux-fsdevel mailing list. (See also the authors' paper on
<a href="http://www.filesystems.org/project-sdfs.html">Secure Deletion File Systems</a>.)
I've modified the patches to make them more suitable for the present purpose.
<ul>
<li><a href="linux-2.6.25-zerofree2.patch">linux-2.6.25-zerofree2.patch</a> (for ext2 filesystems)</li>
<li><a href="linux-2.6.25-zerofree3.patch">linux-2.6.25-zerofree3.patch</a> (for ext3 filesystems)</li>
</ul>
When a filesystem is mounted with the <code>zerofree</code> option (added
by these patches) all the blocks freed when a file is deleted are filled
with zeroes.
Remember, this extra work will hurt disk performance.
Note that the ext3 patch doesn't support data journalling
mode, so deleted metadata isn't zeroed. It also hasn't been tested
as thoroughly as the patch for ext2. And neither has been maintained for a
very long time.
<p>
However, the above techniques are only half the story: the empty free
blocks still consume space in the underlying filesystem, so something
must to be done to reclaim that space. For filesystems and kernels that
support it the fallocate call with <code>FALLOC_FL_PUNCH_HOLE</code> can
be used to deallocate space in a file.
<p>
An existing alternative is to use the sparse file handling capabilities
of the GNU <code>cp</code> command to take a copy of the filesystem image with
<code>cp --sparse=always</code> (though this does require the original
and sparse files to exist at the same time, which may be inconvenient).
<p>
As an alternative alternative I've written a utility which can make
any specified files on an ext2 filesystem
sparse, <a href="sparsify.c">sparsify.c</a>. This doesn't require any
additional disk space to work its magic, but it does require that the
filesystem containing the filesystem image is unmounted, which is just a
different sort of inconvenience.
<p>
<ul>
<li>The usual disclaimers apply: this worked for me when I tested it but it
might destroy your data.</li>
<li>Versions of libext2fs prior to 1.41 do not support ext4;
later versions should work with all filesystem types.</li>
<li>In 1.41 the progress monitor may exceed 100% as the iterator visits some
blocks more than once.</li>
<li>The code doesn't support huge (>2TB) files. It will issue a warning and
refuse to process them.</li>
</ul>
<p>
As an example, suppose we have an unmounted filesystem
image, <code>fs.image</code>, in the directory <code>/data</code>, which is the
root of the <code>/dev/hda2</code> filesystem. We can reclaim deleted
blocks and make it sparse like so:
<p>
<pre>
zerofree /data/fs.image
umount /data
sparsify /dev/hda2 /fs.image
mount /data
</pre>
<p>
<hr>
<address>
<a href="mailto:rmy@tigress.co.uk">Ron Yorston</a><br>
9th August 2012
</address>
</body>
</html>