Planet GLLUG

Rev. Simon Rumble: Sneaky trick to de-obfuscate Omniture JavaScript plugins

Wed, 10 Mar 2010 23:06:06 +0000

My current job involves working extensively with Omniture products. The company has an annoying habit of secrecy, with documentation only available on request for many aspects of their products.

They also attempt to obfuscate their JavaScript, despite the fact that a determined viewer should be able to work it out eventually. I'm told this is so that people aren't tempted to play with the code. The obvious methods of deobfuscation are pretty tedious, and because Omniture don't use standard (minify et al) methods of obfuscation so it seems a little more difficult. Fortunately I lucked onto a better approach.

You'll need Firebug, and if you don't have that already you should anyway. Go to a page that already has the Omniture "plugin" (function) you want. Open the Firebug console and run alert(s.functionNameYouWant) and run it. You'll be shown a nicely-formatted anonymous function, which will be much easier to read than the line noise you'll see in the actual s_code.js file.

In my case I'm after Cross-Visit Participation, and that's used on the Omniture site itself (though an older version than the latest available from Omniture which has a very useful additional feature).

Contact me

Richard WM Jones: rich

Wed, 10 Mar 2010 16:05:03 +0000

You’ve got a partitioned disk image, how do you pull out of that just the filesystem(s)? It’s easy with libguestfs tools:

$ virt-list-filesystems -al disk.img
/dev/sda1 ext4
/dev/vg_f12x32/lv_root ext4
/dev/vg_f12x32/lv_swap swap
$ virt-cat disk.img /dev/sda1 > boot.fs
$ file boot.fs
boot.fs: Linux rev 1.0 ext4 filesystem data (extents) (huge files)
$ virt-cat disk.img /dev/vg_f12x32/lv_root > root.fs

You can also use guestfish to examine the filesystem image:

$ guestfish -a boot.fs -m /dev/sda

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help with commands
      'quit' to quit the shell

><fs> ll /
total 15941
dr-xr-xr-x.  5 root root     1024 Mar  8 19:37 .
dr-xr-xr-x  19 root root        0 Mar  8 13:40 ..
-rw-r--r--.  1 root root  1486532 Nov  7 21:38 System.map-2.6.31.5-127.fc12.i686.PAE
-rw-r--r--.  1 root root   103788 Nov  7 21:38 config-2.6.31.5-127.fc12.i686.PAE
drwxr-xr-x.  3 root root     1024 Mar  8 19:12 efi
drwxr-xr-x.  2 root root     1024 Mar  8 19:49 grub
-rw-r--r--.  1 root root 11253019 Mar  8 19:39 initramfs-2.6.31.5-127.fc12.i686.PAE.img
drwx------.  2 root root    12288 Mar  8 18:45 lost+found
-rwxr-xr-x.  1 root root  3454368 Nov  7 21:38 vmlinuz-2.6.31.5-127.fc12.i686.PAE

><fs> cat /grub/grub.conf
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/mapper/vg_f12x32-lv_root
#          initrd /initrd-[generic-]version.img
#boot=/dev/sda
default=0
timeout=0
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Fedora (2.6.31.5-127.fc12.i686.PAE)
	root (hd0,0)
	kernel /vmlinuz-2.6.31.5-127.fc12.i686.PAE ro root=/dev/mapper/vg_f12x32-lv_root  LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=uk rhgb quiet
	initrd /initramfs-2.6.31.5-127.fc12.i686.PAE.img

><fs>

Richard WM Jones: rich

Tue, 09 Mar 2010 12:41:21 +0000

Something unknown was changing the labels on certain devices behind my back. We couldn’t find out what it was using ordinary diagnostics, so I decided to investigate if we could do this with SystemTap. I quickly found an existing script to monitor changes in ordinary file attributes. This won’t work for SELinux labels though because those are stored in ext2/3/4 extended attributes (xattrs).

Basically I had to modify that script to monitor calls to setxattr instead.

Using the LXR I found that the call is implemented in Linux in fs/xattr.c, function vfs_setxattr. I had to modify the script to probe that kernel function, and the parameters are slightly different too.

I also had to install the correct kernel-{,PAE-}debuginfo package corresponding to my installed kernel. This is how SystemTap is able to resolve symbols in the current kernel.

/* Watch changes to xattrs on an inode.
 * http://rwmj.wordpress.com/2010/03/09/tip-use-systemtap-to-monitor-selinux-changes-to-files/
 */

probe kernel.function("vfs_setxattr")
{
  dev_nr = $dentry->d_inode->i_sb->s_dev
  inode_nr = $dentry->d_inode->i_ino

  if (inode_nr == $1)
    printf ("%s(%d) %s 0x%x/%u %d %s %s\n",
      execname(), pid(), probefunc(), dev_nr, inode_nr, uid(),
      kernel_string ($name), kernel_string_n ($value, $size))
}

Then run it with:

# stap -v /tmp/inodewatchxattr.stp inodenum

The bug turned out to be udevd, which I don’t think anyone was expecting …

libvirtd(4338) vfs_setxattr 0x5/166267 0 security.selinux system_u:object_r:svirt_image_t:s0:c177,c272
udevd(28299) vfs_setxattr 0x5/166267 0 security.selinux system_u:object_r:fixed_disk_device_t:s0
udevd(28299) vfs_setxattr 0x5/166267 0 security.selinux system_u:object_r:fixed_disk_device_t:s0
udevd(28299) vfs_setxattr 0x5/166267 0 security.selinux system_u:object_r:fixed_disk_device_t:s0

All in all, I’m impressed with SystemTap. It’s a simple, strongly-typed, sane programming language with type inference. Thankfully Python was not an influence on it.

Daniel Roseman: Easy create or update

Tue, 09 Mar 2010 10:38:44 +0000

One common database operation that isn't supported out of the box by Django's ORM is create_or_update - in other words, given a set of parameters, either update an existing object or create a new one if there isn't one already.

The naive implementation is to do a get() on the model, catching the DoesNotExist exception if there's no match and instantiating a new object, then updating the attributes and saving. (You wouldn't want to use get_or_create here, as that doesn't allow you to update the instance if it already exists, so you'd have some duplication of code and db queries).

try:
    obj = MyModel.objects.get(field1=value1)
except MyModel.DoesNotExist:
    obj = MyModel()
    obj.field1 = field1
obj.field2 = value2
obj.save()

The only problem with this is that it creates multiple queries: one to get the existing row, and then two to save it - Django checks to see if it should do an insert or an update when you save, which costs another query. Most of the time, this doesn't massively matter: creating and updating is usually done outside of the standard page rendering flow, so it's not a huge problem if it's a tiny bit slower.

But there are times when you do want to optimise this. One, which we recently ran into at work, is when you want to log items to the database in the course of normal page rendering. We do this to let users of our CMS know when they've put items on a page that aren't rendering how they should be, usually because they don't have the right selection of image assets. (There are good operational reasons as to why we can't stop them from entering them in the first place: I won't go into that here.) A further wrinkle for us is that we want to ensure each error only gets one entry in the log table, but should always record the most recent time that particular error scenario was encountered. So, an ideal case for create_or_update, if only it existed.

Of course I can't stand to see unnecessary db queries, so here's an implementation that uses QuerySet.update to do the initial getting and updating if a match exists. The trick is to realise that update returns the number of rows affected by the query - which has been true more or less ever since queryset-refactor landed nearly two years ago, but which was wrongly and explicitly denied in the documentation until recently (and still is denied in the 1.1 docs, even though it's true). We can use this number to tell if a matching row existed - and if it doesn't, we can then simply call create with the same arguments. Simple.

attrs = {'field1': 'value1', 'field2': 'value2'}
filter_attrs = {'filter_field': 'filtervalue'}
rows = MyModel.objects.filter(**filter_attrs).update(**attrs)
if not rows:
    attrs.update(filter_attrs)
    obj = MyModel.objects.create(**attrs)

The attrs dictionary contains the field names/values to use to update the object, and filter_attrs is the filter names/values to find the object to update. If we're creating a new object, it will of course need to set both the attrs values and the filter_attrs, so we update one dictionary from the other.

Now, note that this will always call a db UPDATE, and if no match exists, it will additionally call an INSERT. Compare this with the original version, which always calls a SELECT, plus another SELECT and an UPDATE if the match exists, but just an INSERT if there's no match. So whether this is more efficient will depend on the use case - if you expect more updates than create, this version should be better (a single UPDATE versus SELECT+UPDATE), but if the reverse is true the original implementation will probably be better.

Richard WM Jones: rich

Tue, 09 Mar 2010 10:16:13 +0000

Yesterday we improved virt-inspector so it can now fetch information about Windows guests by reading their Registries. In the XML output, this provides the ProductName and Windows internal version:

$ virt-inspector --xml Win2003x32
[...]
    <name>windows</name>
    <product_name>Microsoft Windows Server 2003</product_name>
    <arch>i386</arch>
    <major_version>5</major_version>
    <minor_version>2</minor_version>
[...]

In the raw output you get even more details from the Registry:

$ virt-inspector --perl Windows7x64
[...]
'arch' => 'x86_64',
'windows_registered_owner' => 'rjones',
'windows_current_type' => 'Multiprocessor Free',
'windows_system_hive' => '/Windows/System32/config/SYSTEM',
'windows_installation_type' => 'Client',
'os_major_version' => '6',
'os_minor_version' => '1',
'systemroot' => '/Windows',
'windows_software_hive' => '/Windows/System32/config/SOFTWARE',
'windows_software_type' => 'System',
'windows_registered_organization' => '',
'windows_current_build' => '7600',
'windows_edition_id' => 'Enterprise',
'product_name' => 'Windows 7 Enterprise',
[...]

Martin A. Brooks: Game review: Assassin's Creed 2

Mon, 08 Mar 2010 20:23:00 +0000

Following my original review of the first Assassin's Creed game, I was dearly looking forward to reviewing the new episode in the series. Alas Ubisoft have taken the skull-smackingly stupid decision of making a single-player game need access to the Internet to work.

Don't buy this game, you will be funding idiocy if you do.

What next, Ubisoft, will you be making me not buy the upcoming Splinter Cell, too?

Richard WM Jones: rich

Mon, 08 Mar 2010 17:02:05 +0000

From the spam folder of this blog today:

The layout for %BLOGURL% is a bit off in iCab. However I like your site. I may have to install a “normal” browser just to enjoy it.

I usually don’t usually post on many Blogs, yet I just has to say thank you for %BLOGTITLE%… keep up the amazing work. Ok regrettably its time to get to school.

Even more incompetently, neither of the spammy URLs they were trying to add actually worked (checked using ‘wget’, not a real browser of course).

Rev. Simon Rumble: UI fail from Exetel

Mon, 08 Mar 2010 01:12:12 +0000

Work is providing me a mobile, so I went to cancel my phone with Exetel. Unfortunately this is the UI you see. So first of all, you can helpfully cancel it in the past. But then the button is labelled "Cancel". So does that means clicking it will cancel my service, or cancel the request to disconnect?

The resulting page is even more confusing. Does that mean my "Cancel" was successful? Or do I now need to "Submit" to make it happen? Terribly confused.

Contact me

www.DavidPashley.com/blog: Mod_fastcgi and external PHP

Sun, 07 Mar 2010 23:02:49 +0000

Has anyone managed to get a standard version of mod_fastcgi work correctly with FastCGIExternalServer? There seems to be a complete lack of documentation on how to get this to work. I have managed to get it working by removing some code which appears to completely break AddHandler. However, people on the FastCGI list told me I was wrong for making it work. So, if anyone has managed to get it to work, please show me some working config.

Read Comments (1)

Martin A. Brooks: Filesystem-based greylisting for Exim

Sun, 07 Mar 2010 18:47:00 +0000

I have written a filesystem-based greylisting engine for Exim. See here for details.

davblog - Dave Cross: Gigs for Old Gits

Sun, 07 Mar 2010 12:21:20 +0000

It's been a busy couple of weeks for gigs. On the assumption that at least some of my readers have similar tastes to me, here are brief reviews of the three gigs I've seen in the last couple of weeks.

Fairport Convention, Union Chapel, 20th Feb
This is the second year running that I've seen Fairport Convention on their "Wintour" at the Union Chapel. Last year was the first time I had seen them (which is bizarre for a band I've been a fan of for over thirty years). I can't quite put my finger on it, but this year's show wasn't as enjoyable as last year's. I suspect it was down to the number of songs taken from later Fairport albums that I'm not at all familiar with. Oh, and the arrangement of Matty Groves was very strange. The long instrumental that ends the song was unrecognisable.

Thomas Dolby and Friends, Union Chapel, 28th Feb
Something a little more up to date. This was Thomas Dolby bringing back together the band who had recorded and toured his second album, The Flat Earth. As an extra twist, the band (who haven't played together for over twenty-five years) didn't rehearse at all. They met on stage and worked the songs out in a two-hour "live rehearsal". They then went of for a brief break before returning to play a half-hour set.

The rehearsal was fun. And the band sounded great for a band eho hadn't played together for so long. There were also a few guest stars - including Trevor Horn who played bass on "Airwaves". The only slight disappointment was that the rehearsal overran so the final set had to be cut short.

John Cale, Royal Festival Hall, 5th March
I'm not a huge John Cale fan. I generally like the stuff of his that I hear, but I haven't really heard much of it. This concert had him playing the whole of hist album "Paris 1919" (from 1973). This isn't an album that I'd heard at all until I started to listen to it in preparation for this show and it's really not that representative of the rest of his music. But it's a great album and it was interesting to hear it all played live. It is, however, a rather short album (many were back in the early 70s) and that part of the show only lasted forty minutes. After a short break (and it was really short) the band returned to play another forty minutes of "the best of John Cale". I was pretty surprised to realise that I recognised most of these songs. All in all, a great night out.

Martin A. Brooks: Automatic email archiving

Sat, 06 Mar 2010 11:22:00 +0000

If you're on as many mailing lists as I am, you might find this to be handy.

Richard WM Jones: rich

Fri, 05 Mar 2010 10:28:52 +0000

You can run arbitrary commands, shells, editors etc during your presentation. Here, I run a gnome-terminal with a prepopulated command history:

Download it from my git repository. The requirements are fairly light: perl, perl-Gtk2 and perl-Gtk2-MozEmbed (all in Fedora).

Previous angry rant about presentation software.

The diagram in the first slide was done using PGF and Tikz 2.00 (examples) (manual).

Richard WM Jones: rich

Wed, 03 Mar 2010 17:41:44 +0000

How many dull presentations have you been to where the presenter simply reads bullet points off slides?

FrobSoft Express 2.0 is:

up to 5% faster
supports Windows
XML enabled!

I’m giving a talk about libguestfs on 18th March and I hate reading out slides to people as much as I hate listening to presenters reading out slides to me. In every talk I’ve given in the last few years I have tried to keep my notes separate (written on paper in front of me, or memorized) from what is on the slides. Presentation software, such as the mighty, all-pervasive OpenOffice, doesn’t make this easy. Nor does it make it easy to demonstrate software in the middle of your talk. You end up having to switch away to another virtual desktop, where (hopefully) you’ve remembered to set up some xterms “su”‘d to root and “cd”‘d into the right directory. I usually need several virtual desktops set up like this so I can demonstrate different parts of the software, so I’m standing in front of an audience using [Alt][←] and [Alt][→] while I hastily try to remember which virtual desktop has the next stage of the talk.

Enough!

Introducing “Tech Talk”. Actually, Tech Talk is too generic in Google, so we brainstormed adding extra words on the end until it became unique: Introducing “Tech Talk Platinum Supreme Edition!” (Tech Talk PSE).

The concept is simple. You create a directory and drop a mixture of HTML files and shell scripts in there:

$ ls
10-hello.html
20-shell.sh
30-goodbye.html
$ techtalk-pse

When Tech Talk PSE runs, it sorts the files numerically, and then displays the HTML ones (using Mozilla embedding) as slides and runs the shell script ones. Next and previous keys move through the slides, ensuring that your demonstrations [the shell scripts] run automatically at the right place in the talk.

Only files matching ^\d+(-.*)\.(html|sh)$ are considered, everything else is ignored. So you can style your HTML using stylesheets, include READMEs and Makefiles, and move common shell functionality into sourced shell files:

#!/bin/bash -
# Source common functions and variables.
source functions
# Pre-populate the shell history.
cat > $HISTFILE <<EOF
guestfish -a vm.img
EOF
# Open gnome-terminal.
exec $TERMINAL --geometry=+100+100

Tech Talk PSE itself doesn’t have to deal with rendering, which is pushed off to a browser, making it far more flexible, powerful and simpler than existing presentation software. This means you can show figures or play video in your presentation, or use Javascript to make your slides resolution-independent or to add animations. Additionally you can use any existing tool you want to write HTML. (If you’re like me, that tool will be emacs.)

You’ll be able to download Tech Talk PSE after my talk in two weeks time, or get early previews from my git repository. Requirements are Perl, Perl Gtk2 and Gtk2::MozEmbed.

Richard WM Jones: rich

Wed, 03 Mar 2010 15:46:51 +0000

KageSenshi’s HOWTO use Linux Containers (LXC) on Fedora 12 with libvirt is interesting. I discovered that they’re using febootstrap (see earlier postings) to build the Fedora root filesystem for the containers.

Richard WM Jones: rich

Wed, 03 Mar 2010 15:31:31 +0000

Watchdog, by Emmanuel Tabard, used with permission from Flickr

In a physical server, a watchdog is a simple piece of hardware which is supposed to restart the server if it hangs without needing any administrator intervention. Watchdogs used to come as separate cards, but nowadays the feature is found in many chipsets, often integrated with other useful bits of server / remote access functionality like remote serial port, wake-on-LAN, hardware event monitoring etc.

So how does the watchdog work? “Is the machine hung?” is a tricky question to answer if you just look at the hardware level. In a “hung” machine, the CPU is most likely still running, and the kernel might still be up and responding to pings.

Hardware watchdogs instead rely on a piece of software running on the machine which must “tickle” a particular port, say every 10 seconds. If the watchdog doesn’t get “tickled” after, say, 60 seconds (so 6 missed events), then it asserts the RESET line which results in a hard reboot. (Of course, how often you must tickle the hardware varies from watchdog device to watchdog device, and is usually configurable. Some hardware watchdogs have more elaborate states — for example, they can deliver a “second chance” interrupt shortly before they deal the final death blow to the machine. In reality no one uses anything but the basic tickle/reset function.)

So the hardware defers to some software which has to keep tickling the hardware, or else face reboot. But how does this software work? In Linux we use the venerable watchdog daemon project. This is an unusual case where you actually want the software to do lots of “useless” work. So the daemon will typically ping a remote network address, do a process listing, maybe write something to disk and access the service, and run some custom scripts, and only if all those succeed will it tickle the watchdog port. Every 10 seconds.

If you think about some ways in which a server can “hang” you can see why this works. Example (1): Hard disk drive stops sending back interrupts. Processes begin to go into the uninterruptible “D” state, and don’t come out. Watchdog daemon itself enters the D state when it writes to the disk, hence the watchdog port is never tickled and the server gets rebooted. Example (2): Web server parent process segfaults. Existing and some new requests are still being serviced, so the server appears to be working from the outside, but less reliably, at least for a little while. The watchdog daemon lists out the processes in the system and notices that the web server parent process is gone (because of a custom test). As a result, it doesn’t tickle the watchdog port, and so the machine is rebooted. Example (3): A large SQL request results in an important database table getting locked. The watchdog daemon periodically fetches a database-backed web page from the web server. The watchdog daemon’s request hangs. The watchdog port isn’t tickled. The web server reboots.

What does this have to do with virtualization? Virtual machines can also hang in various unpredictable ways, and for the same reasons I outlined above, it’s hard to know whether a VM is hanging, overloaded or just slow. And for all the same reasons, you might want to reboot a wedged VM without administrator intervention. For this reason I wrote a virtual watchdog device for qemu and KVM. It’s simple to configure the watchdog using libvirt:

# virsh edit domname

and add <watchdog model='i6300esb'/> into the devices section of the XML.

That will create a virtual Intel 6300 ESB (just the watchdog part of this multi-function Intel chipset). You’ll see this PCI device appear when the VM boots:

$ dmesg | grep 6300
i6300ESB timer: initialized (0xffffc20000016000). heartbeat=30 sec (nowayout=0)
$ /sbin/lspci | grep 6300
00:05.0 System peripheral: Intel Corporation 6300ESB Watchdog Timer

In the guest, install the watchdog daemon. Linux contains a driver already for the i6300ESB, and for Windows you can download drivers from Intel’s website.

Configure /etc/watchdog.conf and perhaps write a few custom tests. Make sure the watchdog service is set to start at boot. Once the watchdog service is started, it will tickle the (virtual) hardware watchdog. If qemu / KVM notices that the software is no longer tickling the virtual port, it will hard reboot the VM.

Richard WM Jones: rich

Tue, 02 Mar 2010 21:21:11 +0000

This question arose at work — is LVM a performance penalty compared to using straight partitions? To save you the trouble, the answer is “not really”. There is a very small penalty, but as with all benchmarks it does depend on what the benchmark measures versus what your real workload does. In any case, here is a small guestfish script you can use to compare the performance of various filesystems with or without LVM, with various operations. Whether you trust the results is up to you, but I would advise caution.

#!/bin/bash -

tmpfile=/tmp/test.img

for fs in ext2 ext3 ext4; do
    for lvm in off on; do
        rm -f $tmpfile
        if [ $lvm = "on" ]; then
            guestfish <<EOF
              sparse $tmpfile 1G
              run
              part-disk /dev/sda efi
              pvcreate /dev/sda1
              vgcreate VG /dev/sda1
              lvcreate LV VG 800
              mkfs $fs /dev/VG/LV
EOF
            dev=/dev/VG/LV
        else # no LVM
            guestfish <<EOF
              sparse $tmpfile 1G
              run
              part-disk /dev/sda efi
              mkfs $fs /dev/sda1
EOF
            dev=/dev/sda1
        fi
        echo "fs=$fs lvm=$lvm"
        sync
        guestfish -a $tmpfile -m $dev <<EOF
          time fallocate /file1 200000000
          time cp /file1 /file2
EOF
    done
done

fs=ext2 lvm=off
elapsed time: 2.74 seconds
elapsed time: 4.52 seconds
fs=ext2 lvm=on
elapsed time: 2.60 seconds
elapsed time: 4.24 seconds
fs=ext3 lvm=off
elapsed time: 2.62 seconds
elapsed time: 4.31 seconds
fs=ext3 lvm=on
elapsed time: 3.07 seconds
elapsed time: 4.79 seconds

# notice how ext4 is much faster at fallocate, because it
# uses extents

fs=ext4 lvm=off
elapsed time: 0.05 seconds
elapsed time: 3.54 seconds
fs=ext4 lvm=on
elapsed time: 0.05 seconds
elapsed time: 4.16 seconds

Richard WM Jones: rich

Tue, 02 Mar 2010 19:08:46 +0000

FAQ entry: The API has hundreds of methods, where do I start? A: Start with the API overview in the manual page.

$ guestfish -h
    Command              Description
help                 display a list of commands or help on a command
quit                 quit guestfish
alloc                allocate an image
echo                 display a line of text
edit                 edit a file in the image
lcd                  local change directory
glob                 expand wildcards in command
more                 view a file in the pager
reopen               close and reopen libguestfs handle
sparse               allocate a sparse image file
time                 measure time taken to run command
add-cdrom            add a CD-ROM disk image to examine
add-drive            add an image to examine or modify
add-drive-ro         add a drive in snapshot mode (read-only)
add-drive-ro-with-if add a drive read-only specifying the QEMU block emulation to use
add-drive-with-if    add a drive specifying the QEMU block emulation to use
aug-close            close the current Augeas handle
aug-defnode          define an Augeas node
aug-defvar           define an Augeas variable
aug-get              look up the value of an Augeas path
aug-init             create a new Augeas handle
aug-insert           insert a sibling Augeas node
aug-load             load files into the tree
aug-ls               list Augeas nodes under augpath
aug-match            return Augeas nodes which match augpath
aug-mv               move Augeas node
aug-rm               remove an Augeas path
aug-save             write all pending Augeas changes to disk
aug-set              set Augeas path to value
available            test availability of some parts of the API
blockdev-flushbufs   flush device buffers
blockdev-getbsz      get blocksize of block device
blockdev-getro       is block device set to read-only
blockdev-getsize64   get total size of device in bytes
blockdev-getss       get sectorsize of block device
blockdev-getsz       get total size of device in 512-byte sectors
blockdev-rereadpt    reread partition table
blockdev-setbsz      set blocksize of block device
blockdev-setro       set block device to read-only
blockdev-setrw       set block device to read-write
case-sensitive-path  return true path on case-insensitive filesystem
cat                  list the contents of a file
checksum             compute MD5, SHAx or CRC checksum of file
chmod                change file mode
chown                change file owner and group
command              run a command from the guest filesystem
command-lines        run a command, returning lines
config               add qemu parameters
cp                   copy a file
cp-a                 copy a file or directory recursively
dd                   copy from source to destination using dd
debug                debugging and internals
df                   report file system disk space usage
df-h                 report file system disk space usage (human readable)
dmesg                return kernel messages
download             download a file to the local machine
drop-caches          drop kernel page cache, dentries and inodes
du                   estimate file space usage
e2fsck-f             check an ext2/ext3 filesystem
echo-daemon          echo arguments back to the client
egrep                return lines matching a pattern
egrepi               return lines matching a pattern
equal                test if two files have equal contents
exists               test if file or directory exists
fallocate            preallocate a file in the guest filesystem
fgrep                return lines matching a pattern
fgrepi               return lines matching a pattern
file                 determine file type
filesize             return the size of the file in bytes
fill                 fill a file with octets
find                 find all files and directories
find0                find all files and directories, returning NUL-separated list
fsck                 run the filesystem checker
get-append           get the additional kernel options
get-autosync         get autosync mode
get-direct           get direct appliance mode flag
get-e2label          get the ext2/3/4 filesystem label
get-e2uuid           get the ext2/3/4 filesystem UUID
get-memsize          get memory allocated to the qemu subprocess
get-path             get the search path
get-pid              get PID of qemu subprocess
get-qemu             get the qemu binary
get-recovery-proc    get recovery process enabled flag
get-selinux          get SELinux enabled flag
get-state            get the current state
get-trace            get command trace enabled flag
get-verbose          get verbose mode
getcon               get SELinux security context
getxattrs            list extended attributes of a file or directory
glob-expand          expand a wildcard path
grep                 return lines matching a pattern
grepi                return lines matching a pattern
grub-install         install GRUB
head                 return first 10 lines of a file
head-n               return first N lines of a file
hexdump              dump a file in hexadecimal
initrd-cat           list the contents of a single file in an initrd
initrd-list          list files in an initrd
inotify-add-watch    add an inotify watch
inotify-close        close the inotify handle
inotify-files        return list of watched files that had events
inotify-init         create an inotify handle
inotify-read         return list of inotify events
inotify-rm-watch     remove an inotify watch
is-busy              is busy processing a command
is-config            is in configuration state
is-dir               test if file exists
is-file              test if file exists
is-launching         is launching subprocess
is-ready             is ready to accept commands
kill-subprocess      kill the qemu subprocess
launch               launch the qemu subprocess
lchown               change file owner and group
lgetxattrs           list extended attributes of a file or directory
list-devices         list the block devices
list-partitions      list the partitions
ll                   list the files in a directory (long format)
ln                   create a hard link
ln-f                 create a hard link
ln-s                 create a symbolic link
ln-sf                create a symbolic link
lremovexattr         remove extended attribute of a file or directory
ls                   list the files in a directory
lsetxattr            set extended attribute of a file or directory
lstat                get file information for a symbolic link
lstatlist            lstat on multiple files
lvcreate             create an LVM volume group
lvm-remove-all       remove all LVM LVs, VGs and PVs
lvremove             remove an LVM logical volume
lvrename             rename an LVM logical volume
lvresize             resize an LVM logical volume
lvs                  list the LVM logical volumes (LVs)
lvs-full             list the LVM logical volumes (LVs)
lxattrlist           lgetxattr on multiple files
mkdir                create a directory
mkdir-mode           create a directory with a particular mode
mkdir-p              create a directory and parents
mkdtemp              create a temporary directory
mke2fs-J             make ext2/3/4 filesystem with external journal
mke2fs-JL            make ext2/3/4 filesystem with external journal
mke2fs-JU            make ext2/3/4 filesystem with external journal
mke2journal          make ext2/3/4 external journal
mke2journal-L        make ext2/3/4 external journal with label
mke2journal-U        make ext2/3/4 external journal with UUID
mkfifo               make FIFO (named pipe)
mkfs                 make a filesystem
mkfs-b               make a filesystem with block size
mkmountpoint         create a mountpoint
mknod                make block, character or FIFO devices
mknod-b              make block device node
mknod-c              make char device node
mkswap               create a swap partition
mkswap-L             create a swap partition with a label
mkswap-U             create a swap partition with an explicit UUID
mkswap-file          create a swap file
modprobe             load a kernel module
mount                mount a guest disk at a position in the filesystem
mount-loop           mount a file using the loop device
mount-options        mount a guest disk with mount options
mount-ro             mount a guest disk, read-only
mount-vfs            mount a guest disk with mount options and vfstype
mountpoints          show mountpoints
mounts               show mounted filesystems
mv                   move a file
ntfs-3g-probe        probe NTFS volume
part-add             add a partition to the device
part-disk            partition whole disk with a single primary partition
part-get-parttype    get the partition table type
part-init            create an empty partition table
part-list            list partitions on a device
part-set-bootable    make a partition bootable
part-set-name        set partition name
ping-daemon          ping the guest daemon
pread                read part of a file
pvcreate             create an LVM physical volume
pvremove             remove an LVM physical volume
pvresize             resize an LVM physical volume
pvs                  list the LVM physical volumes (PVs)
pvs-full             list the LVM physical volumes (PVs)
read-file            read a file
read-lines           read file as lines
readdir              read directories entries
readlink             read the target of a symbolic link
readlinklist         readlink on multiple files
realpath             canonicalized absolute pathname
removexattr          remove extended attribute of a file or directory
resize2fs            resize an ext2/ext3 filesystem
rm                   remove a file
rm-rf                remove a file or directory recursively
rmdir                remove a directory
rmmountpoint         remove a mountpoint
scrub-device         scrub (securely wipe) a device
scrub-file           scrub (securely wipe) a file
scrub-freespace      scrub (securely wipe) free space
set-append           add options to kernel command line
set-autosync         set autosync mode
set-direct           enable or disable direct appliance mode
set-e2label          set the ext2/3/4 filesystem label
set-e2uuid           set the ext2/3/4 filesystem UUID
set-memsize          set memory allocated to the qemu subprocess
set-path             set the search path
set-qemu             set the qemu binary
set-recovery-proc    enable or disable the recovery process
set-selinux          set SELinux enabled or disabled at appliance boot
set-trace            enable or disable command traces
set-verbose          set verbose mode
setcon               set SELinux security context
setxattr             set extended attribute of a file or directory
sfdisk               create partitions on a block device
sfdiskM              create partitions on a block device
sfdisk-N             modify a single partition on a block device
sfdisk-disk-geometry display the disk geometry from the partition table
sfdisk-kernel-geometry display the kernel geometry
sfdisk-l             display the partition table
sh                   run a command via the shell
sh-lines             run a command via the shell returning lines
sleep                sleep for some seconds
stat                 get file information
statvfs              get file system statistics
strings              print the printable strings in a file
strings-e            print the printable strings in a file
swapoff-device       disable swap on device
swapoff-file         disable swap on file
swapoff-label        disable swap on labeled swap partition
swapoff-uuid         disable swap on swap partition by UUID
swapon-device        enable swap on device
swapon-file          enable swap on file
swapon-label         enable swap on labeled swap partition
swapon-uuid          enable swap on swap partition by UUID
sync                 sync disks, writes are flushed through to the disk image
tail                 return last 10 lines of a file
tail-n               return last N lines of a file
tar-in               unpack tarfile to directory
tar-out              pack directory into tarfile
tgz-in               unpack compressed tarball to directory
tgz-out              pack directory into compressed tarball
touch                update file timestamps or create a new file
truncate             truncate a file to zero size
truncate-size        truncate a file to a particular size
tune2fs-l            get ext2/ext3/ext4 superblock details
umask                set file mode creation mask (umask)
umount               unmount a filesystem
umount-all           unmount all filesystems
upload               upload a file from the local machine
utimens              set timestamp of a file with nanosecond precision
version              get the library version number
vfs-type             get the Linux VFS type corresponding to a mounted device
vg-activate          activate or deactivate some volume groups
vg-activate-all      activate or deactivate all volume groups
vgcreate             create an LVM volume group
vgremove             remove an LVM volume group
vgrename             rename an LVM volume group
vgs                  list the LVM volume groups (VGs)
vgs-full             list the LVM volume groups (VGs)
wc-c                 count characters in a file
wc-l                 count lines in a file
wc-w                 count words in a file
write-file           create a file
zegrep               return lines matching a pattern
zegrepi              return lines matching a pattern
zero                 write zeroes to the device
zerofree             zero unused inodes and disk blocks on ext2/3 filesystem
zfgrep               return lines matching a pattern
zfgrepi              return lines matching a pattern
zfile                determine file type inside a compressed file
zgrep                return lines matching a pattern
zgrepi               return lines matching a pattern
    Use -h  / help  to show detailed help for a command.

Richard WM Jones: rich

Tue, 02 Mar 2010 19:03:14 +0000

A shout out to Debian packaging genius Guido Gunther who demonstrates how to use virt-inspector, virt-df, virt-edit and guestfish on Debian virtual machines. You can download the Debian packages via this link on our FAQ page or direct from alioth.

GLLUG: 18th March - Greater London LUG Meeting

Tue, 02 Mar 2010 12:42:11 +0000

Date: Thursday, March 18, 2010
Time: 7:30pm - 9:30pm
Location: Univ of Westminster Campus
Street: New Cavendish Street
City/Town: London

Map:

View Larger Map

There will be a talk for the GLLUG by Richard Jones on libguestfs[1] on Thursday 18th March at 1930. It will be held at the Cavendish campus of the University of Westminster, in the Large Lecture Theatre. I will try and get a more accurate description of the lecture theatre's location and post it to the mailing list and website. If people could arrive at 1900 to make sure we are ready to begin at 1930. After the talk I suggest we go to a decent local pub for a social.

Please email general.mooney-at-googlemail.com if you are attending in order to get an idea of the expected number of people.

Ciaran Mooney

[1] http://libguestfs.org

davblog - Dave Cross: OLB Non Enrolled Non Endorsed 1

Mon, 01 Mar 2010 15:27:02 +0000

When communicating with your customers, it's important to look at the information that you're sending from their point of view. Are they really going to be interested in the information that you send?

Earlier today I finally got round to unsubscribing from the MBNA marketing emails that have been annoying me for months. To confirm my unsubscription they sent me an email which started with this:

We are sorry that you unsubscribed from the newsletter OLB Non Enrolled Non Endorsed 1

Is there really any customer who is going to be even slightly interested in that level of detail? I don't care what your internal name for the newsletter is. I just want to stop seeing it in my inbox.

davblog - Dave Cross: The Learning Guitar

Mon, 01 Mar 2010 12:21:37 +0000

I don't play the guitar very well at all. I'll sometimes say that I play it better than average, but that's a claim that can only be justified by pointing out that the vast majority of people don't play guitar at all so anyone who knows two or three chords is already well above average.

I have, however, been playing guitar (for some loose definition of the word "playing") for a rather long time. Just how long was brought home to me this weekend.

We're having a lot of building work done in our house over the next few months and as a precursor to that we have had to clear pretty much everything out of the first floor. A lot of stuff has gone into storage, but we also took a lot of stuff to our local tip on Saturday. That load included three guitars and one of them was "The Learning Guitar".

The Learning Guitar was (as its name suggests) the guitar that I first learnt to play on. It was a cheap nylon-stringed Spanish guitar that my parents bought me when I started to take lessons. That was very soon after I started at secondary school in September 1974. There was an after school class which I joined. I think I stopped going after only a couple of months as we were learning boring stuff like "When The Saints Go Marching In" when I wanted to be playing stuff by Slade or David Bowie. At the time I assumed that we weren't learning that stuff because it was too difficult for beginners. Later I realised that a lot of the music I enjoyed was actually just as simple as the stuff we were taught - it was just that the teachers were a bit old-fashioned.

I carried on teaching myself though. I bought a Mel Bay book and spent hours practicising in my bedroom. Of course I had no real idea what I was doing and I picked up a number of bad habits that hamper my playing even now. But I was enjoying myself.

Soon after moving to London to go to university I got another guitar. It was a Fender F3. A much nicer-sounding guitar. My original guitar was somewhat ignored. For a year I shared a flat with someone who played guitar really well and by watching him my playing improved a lot.

But the Learning Guitar still had some life in it. Over the next fifteen or twenty years I took to lending it to friends who wanted to learn guitar. The story was always the same. Someone borrowed it for a couple of years and when they thought the time was right, they'd buy a better guitar and give the Learning Guitar back to me. It was during this period that the guitar acquired its nickname. The last person to borrow it like this was my step-daughter who took it with her when she went to university. As always, i came back after a couple of years.

Over the last ten years, I've played guitar a lot less. I couldn't really justify storing the four guitars that I had cluttering up my study. So this weekend they all went except the Fender. We loaded up a van and took them to the Wandsworth Council dump. Of all of the things that I threw away on Saturday, the Learning Guitar was the thing that I felt most guilty about. I threw it high up on a mountain of rubbish at the dump. At one point I considered trying to retrieve it, but it was too far away.

It was never a particularly good guitar. But a lot of people have strummed their first tentative chords on that guitar. It's a shame to see it go.

Later this week, I hope to get rid of my collection of records. That has sat in a cupboard unused for over ten years. There's really no reason to keep it. But if you think I have got needlessly sentimental about an old guitar, you haven't seen anything yet. I'll be getting far more nostalgic about the records.

Richard WM Jones: rich

Sat, 27 Feb 2010 21:59:50 +0000

I enjoyed playing a Japanese import of Densha De Go (an accurate Shinkansen train simulator) on my old Nintendo Wii. I played this legally (albeit expensively) imported game using some other software called Wii Freeloader. Since Nintendo does not like people using software from outside the “right” places, “Freeloader” had to exploit a bug in the firmware to allow Densha de Go to play.

Today I upgraded the firmware on my Wii console.

I am no longer able to play Densha De Go at all. Nintendo have successfully covered all options and there is no way to play my legally purchased and imported software from other “regions” of the world.

So today I learned my lesson. Never absolutely never buy or get involved in proprietary software. Never buy anything ever again from Nintendo. Never buy another phone from Apple, or Microsoft, or any computer with proprietary software no matter how convenient it may seem in the short term.

Enough is enough.

If I bought the hardware, I want to do whatever I want with it.

www.DavidPashley.com/blog: Reducing Coupling between modules

Thu, 25 Feb 2010 09:30:18 +0000

In the past, several of my Puppet modules have been tightly coupled. A perfect example is Apache and Munin. When I install Apache, I want munin graphs set up. As a result my apache class has the following snippet in it:

munin::plugin { "apache_accesses": }
munin::plugin { "apache_processes": }
munin::plugin { "apache_volume": }

This should make sure that these three plugins are installed and that munin-node is restarted to pick them up. The define was implemented like this:

define munin::plugin (
      $enable = true,
      $plugin_name = false,
      ) {

   include munin::node

   file { "/etc/munin/plugins/$name":
      ensure => $enable ? {
         true => $plugin_name ? {
            false => "/usr/share/munin/plugins/$name",
            default => "/usr/share/munin/plugins/$plugin_name"
         },
         default => absent
      },
      links => manage,
      require => Package["munin-node"],
      notify => Service["munin-node"],
   }
}

(Note: this is a slight simplification of the define). As you can see, the define includes munin::node, as it needs the definition of the munin-node service and package. As a result of this, installing Apache drags in munin-node on your server too. It would be much nicer if the apache class only installed the munin plugins if you also install munin on the server.

It turns out that is is possible, using virtual resources. Virtual resources allow you to define resources in one place, but not make them happen unless you realise them. Using this, we can make the file resource in the munin::plugin virtual and realise it in our munin::node class. Our new munin::plugin looks like:

define munin::plugin (
      $enable = true,
      $plugin_name = false,
      ) {

   # removed "include munin::node"

   # Added @ in front of the resource to declare it as virtual
   @file { "/etc/munin/plugins/$name":
      ensure => $enable ? {
         true => $plugin_name ? {
            false => "/usr/share/munin/plugins/$name",
            default => "/usr/share/munin/plugins/$plugin_name"
         },
         default => absent
      },
      links => manage,
      require => Package["munin-node"],
      notify => Service["munin-node"],
      tag => munin-plugin,
   }
}

We add the following line to our munin::node class:

File<| tag == munin-plugin |>

The odd syntax in the munin::node class realises all the virtual resources that match the filter, in this case, any that is tagged munin-plugin. We've had to define this tag ourself, as the auto-generated tags don't seem to work. You'll also notice that we've removed the munin::node include from the munin::plugin define, which means that we no longer install munin-node just by using the plugin define. I've used a similar technique for logcheck, so additional rules are not installed unless I've installed logcheck. I'm sure there are several other places where I can use it to reduce such tight coupling between classes.

Read Comments (2)

davblog - Dave Cross: Homeopathy Petition

Thu, 25 Feb 2010 09:09:48 +0000

We're all, of course, very happy about the results of the House of Commons Science and Technology committee's evidence check on homeopathy. But it's important to realise exactly what has happened. This is a House of Commons committee which has produced a list of recommendations. The government is under no obligation at all to take any notice of those recommendations. Unfortunately, Richard Wiseman's tweet, "yipppeeee it's official, NHS will no longer give people smarties", is likely to be somewhat premature.

So that's why I've set up a petition on the number 10 web site. The petition says:

We the undersigned petition the Prime Minister to Implement the recommendations of the House Commons Science and Technology committee evidence check on Homeopathy.

The House Commons Science and Technology committee has recently undertaken an evidence check on the usefulness of homeopathy and has now published its report.

The conclusions are unequivocal. They say "To maintain patient trust, choice and safety, the Government should not endorse the use of placebo treatments, including homeopathy. Homeopathy should not be funded on the NHS and the MHRA should stop licensing homeopathic products."

The government should implement these recommendations as soon as possible.

If we can get enough people to sign this petition I hope we can send a message to the government letting them know that we support the committee's findings and don't want the NHS's money wasted on nonsense like homeopathy.

Of course, this close to an election, the government is likely to be very wary of making any kind of a statement that might lose them support amongst the woo-mongers. We need to persuade them that skeptical (and rational) voters outnumber the idiots. At the very least, we should be able to get more signatures than this ridiculous petition.

So please sign the petition. And please pass the details on to anyone else who might be interested. The battle has not been won yet.

Karanbir Singh: London Devops mailing list, googlegroups and google accounts

Thu, 25 Feb 2010 01:49:45 +0000

Just signed upto the London Devops mailing list, and thought I'd just point out that one does not need a google account to join a google groups hosted list. Just send an email to the <list-name>+subscribe@googlegroups.com with 'subscribe' in the subject line of the email, and their list managment software will sign you up.

So in this case, sending an email with:


To: london-devops+subscribe@googlegroups.com
Subject: Subscribe

.. would be all thats needed to get you on there.

Google groups has a help page with details on what all services they host which dont need a google account to use.

Original post blogged on b2evolution.

davblog - Dave Cross: NHS Money Wasted on Homeopathy

Wed, 24 Feb 2010 14:59:04 +0000

Don't have time to go into the detail that it deserves, but the House of Commons science and technology committee has published the results of its evidence check on homeopathy. The results won't, of course, come as any surprise to anyone who has been following the debate. But I have to admit to be impressed by the lack of ambiguity in their conclusions. This is paragraph 157:

By providing homeopathy on the NHS and allowing MHRA licensing of products which subsequently appear on pharmacy shelves, the Government runs the risk of endorsing homeopathy as an efficacious system of medicine. To maintain patient trust, choice and safety, the Government should not endorse the use of placebo treatments, including homeopathy. Homeopathy should not be funded on the NHS and the MHRA should stop licensing homeopathic products.

Absolutely no equivocation there.

So what's the next step? When to the homeopathic "hospitals" get closed down? When does the NHS get that money back for real medicine?

Update: The Woo-mongers in the House of Commons don't plan to take this laying down. They've proposed an Early Day Motion criticising the committee's report. Of course, only MPs with no grasp of science will be signing it. If your MP is on this list, then I suggest a strongly worded email might be in order.

Richard WM Jones: rich

Tue, 23 Feb 2010 21:46:48 +0000

I just discovered that virt-top is now in Debian.

virt-top (which I wrote) is the far superior alternative to “xentop”. I wrote it after having to manage some servers with only xentop available. I wrote it because xentop was so crap.

Why is virt-top better? It supports more statistics. It works with libvirt so it works with almost every hypervisor not just Xen. It has a much nicer UI and documentation. And it has advanced logging modes so you can leave it running in the background and capture information for your website/database/stats/monitoring.

Richard WM Jones: rich

Tue, 23 Feb 2010 10:06:40 +0000

Guido Günther has built the very latest libguestfs 1.0.84 official packages for Debian.

Previously Ubuntu packages …

davblog - Dave Cross: Homeopathic Dilutions

Mon, 22 Feb 2010 17:00:42 +0000

Like many press outlets, the Daily Mail pre-empted the publication of the Science and Technology committee report and published a story yesterday summing up the MPs' findings. Of course the Daily Mail is the home of the gullible reader and a good number of the comments on that story are attempting to defend the woo-mongers.

The Mail often stop taking comments on their stories after about a day (giving no indication that they've done so) and the number of comments on this story has stuck at eighteen since I first saw it last night so we must assume that they won't publish any more.

This is a shame as there's quite a lot of unchallenged nonsense there. In particular, the most recent comment published is from Dave in Basingstoke. Someone previously in the discussion had mentioned the ludicrous amount of dilution in homeopathic solutions. Dave replies with this:

"But scientists point to the fact that the 'cures' are so diluted that the cannot possibly contain even a single molecule of the original substance."

Ha! Maybe a climate change pseudo-scientist would say that, but a chemist never would because it isn't true. If you dilute a solution of anything by a million to one, there will still be thousands of molecules of the substance present in the diluted solution. The body can detect that amount, and work on it.

Let's ignore his childish dig at climate change and get to the meat of his argument. Dave thinks that is you dilute something by a factor of a million then there will still be molecules of the original substance in the solution. And he's right there. No one will argue with that fact at all. But homeopathic remedies aren't million to one dilutions.

Homepathic dilutions are given a number on the "C scale". Each time you dilute something by a factor of a hundred, you get another point on the C scale. A dilution of a hundred to one would be called 1C. Dilute that solution be another hundred to one (that's now ten thousand to one from the original solution) and you get to 2C. Another step to 3C gives us a dilution of a million to one from the original solution. That's about the level of dilution that Dave is talking about.

But homeopaths don't stop there. 3C dilutions are nothing. Remember a key tenet of homeopathy is that the weaker the dilution, the stronger the effect. Homeopaths carry on diluting their solutions again and again and again. Next time you're in Boots have a look at the numbers on the tubes of homeopathic remedies that they sell. You'll see that 30C is a really common dilution. That solution has been diluted by a factor of a hundred to one thirty times. The original solution has been diluted by a factor of one to a number which is one followed by sixty zeroes. That's a huge number. With numbers like that involved, it's perfectly reasonable to say that there is none of the original material left.

Here's an example to help you get to grips with those numbers. The number of water molecules in a swimming pool is going to be around a one followed by thirty two zeroes. One molecule of something else in that pool will be equivalent to a 16C homeopathic remedy. See the Wikipedia entry on homeopathic dilutions for more examples like this.

This is why the slogan for the 10:23 campaign is "There's nothing in it". It's literally true. There is no active ingredient left in any homeopathic remedy that you find.

I'd love it it if Dave from Basingstoke found this entry. It would be great if he could see just how misinformed he is.