Planet GLLUG

February 03, 2010 10:52 PM

I used bitstring to reverse engineer the Windows registry “hive” format. I know that bitstring is my own program, but coming back to it two years after I wrote it and using it again for this, I really think this is a brilliant tool. (Bitstring wasn’t my idea — it was inspired by the bitstring manipulation feature in Erlang).

C is supposed to be a good natural programming language for dealing with bits and bytes, right? The ocaml-bitstring program, which analyzes hive files in far more detail than the C program, is half the size and just as fast.

As an example, here’s how we load the hive file and analyze the first part of the header:

let bits = bitstring_of_file filename

(* Split into header + data at the 4KB boundary. *)
let header, data =
  takebits (4096 * 8 ) bits, dropbits (4096 * 8 ) bits

let () =
  bitmatch header with
  { "regf" : 4*8 : string;
    seq1 : 4*8 : littleendian;
    seq2 : 4*8 : littleendian, check (seq1 = seq2);
    last_modified : 64
      : littleendian, bind (nt_to_time_t last_modified);
    1_l (* major *) : 4*8 : littleendian;
    minor : 4*8 : littleendian } ->
      (* ... *)

The bitmatch statement elegantly matches the file. It rejects the file if the first four bytes aren’t “regf” (the file magic number) or if the major version number is not 1. It then unpacks the following fields, converting from the file’s littleendian ordering to host ordering, converting the NT timestamp into a time_t and so on.

Although not shown there, bitstring will also work just fine on arbitrary bit boundaries, albeit more slowly because the generated code is able to make fewer optimizations.

Even though the Windows hive file format is moronic, I successfully used bitstring to reverse engineer it in about 3 days, with some help from the contradictory and often incorrect public documentation out there.

February 02, 2010

Freecycle and Freegle group finder for the iPhone

Recycling Group Finder app for the iPhone

I’ve launched my first iPhone app, the Recycling Group Finder for iPhone. It complements the Recycling Group Finder web app making it even easier to find your closest Freecycle or Freegle group by using the iPhone’s in-built GPS. Check out the information page and give it a go, it’s free.

February 02, 2010 08:40 PM

February 01, 2010

February 01, 2010 10:13 PM

I’ve not used GNU screen too often, just occasional use while running long jobs, uploads, torrents and that sort of thing. To be frank it’s because the default configuration of screen sucks in two important respects: the default escape key is Ctrl-a which conflicts with bash/emacs “go to start of line”, and the very silly “Wuff Wuff” + forced 1 second pause whenever the bell rings.

Thanks go, therefore to Markus Armbruster for giving me some useful screen tips. I dropped the following into my ~/.screenrc and it makes screen very nice and usable:

escape ^^^

vbellwait 0
vbell_msg ^G
bell_msg ^G

hardstatus on
hardstatus alwayslastline
hardstatus string "%H|%w%=|%d/%m %c"

The escape key is Shift-Ctrl-^, not used by anything I know, yet not too inconvenient (and it works over ssh from my laptop and other places). The forced 1 second delay is gone. Nicest of all is a status bar which makes using the multiple window functionality easy.

February 01, 2010 08:33 PM

I wrote this Python libguestfs example as a suggestion for the Fedora AutoQA project. This Python script can be used to preload a kickstart file on to a virt filesystem, the idea being you can script a Fedora DVD install without requiring any network access at all (using ks=hd:vda1:/ks.cfg).

February 01, 2010 09:33 AM

Comments

In the middle of last week I upgraded this site to Movable Type 5. And at (I assume) the same time comments stopped working.

It seems that it was some incompatibility between the MT5 Javascript that drives the comment system and the old MT4 templates that I was using. I've now rebuilt the site using MT5 templates and everything seems to be working.

Sorry for the inconvenience. If you've tried to leave a comment recently and just got a never-ending spinner then I hope you'll try again now.

And a tip to people upgrading to MT5 - rebuild your templates.

January 31, 2010

January 31, 2010 09:54 PM

The libguestfs team have been busy adding features over the past few weeks. Here is a quick round-up.

Writing to Windows Registry files. We haven’t quite got this feature working yet, but we intend that you’ll be able to “shell into” a Windows Registry and change things:

 $ hivexsh SOFTWARE

  Welcome to hivexsh, the hivex interactive shell for examining
  Windows Registry binary hive files.

  Type: 'help' for help summary
        'quit' to quit the shell

  SOFTWARE\> ls
  ATI Technologies
  Classes
  Clients
  Intel
  Microsoft
  ODBC
  Policies
  RegisteredApplications
  Sonic
  Wow6432Node
  SOFTWARE\> cd \Microsoft\Windows\TabletPC\TabSetup
  SOFTWARE\Microsoft\Windows\TabletPC\TabSetup> ls
  SOFTWARE\Microsoft\Windows\TabletPC\TabSetup> lsval
  "TabletSetup"=dword:00000000"
  SOFTWARE\Microsoft\Windows\TabletPC\TabSetup> cd ..
  SOFTWARE\Microsoft\Windows\TabletPC> ls
  Snipping Tool
  TabSetup

Rename LVs and VGs. (Guymon’s incredibly convoluted description of how to do this when you don’t have libguestfs inspired me to add lvrename and vgrename commands).
C# bindings. This doesn’t mean you can use libguestfs on Windows, but it does mean you can use libguestfs from Mono/Linux.
Indian language support. We received Punjabi, Telegu and Oriya translations all at once from the Fedora L10N project so now you can display guestfish in elegant cursive script. “Run a command from the guest filesystem” is “ਗਿਸਟ ਫਾਇਲ-ਸਿਸਟਮ ਤੋਂ ਕਮਾਂਡ ਚਲਾਓ” in Punjabi – how beautiful. The L10N guys work hard doing a pretty thankless job and don’t seem to get much credit for what they do, so thanks guys (and gals).

Ross Lawley

Nginx how to; multiple if statements

I've been doing a lot of Nginx configuration tweaking recently, due to some complex requirements for a virtual host I'm writing for a relaunch due later this month. As I came across a couple of gotchas and problems that required a lot of head scratching, googling and trawling through the forums, I thought I'd write a series of short posts to try and help any other folk that may have similar

January 31, 2010 12:06 PM

Nginx how to; Url cleaning - removing multiple slashes

Carrying on with my short posts on configuring Nginx, heres an easy tip to ensure that all urls are clean, multiple slashes are removed and redirected to the cleaned url eg: http://example.com//search/ and will be redirected to: http://example.com. # Remove any multislashes in the url # $uri is a cleaned version of the url # so we test against the $requested_uri set $clean_uri $

January 31, 2010 12:05 PM

Nginx how to; Server Side Include (SSI) debugging

Server Side Includes (SSI) are a great feature of Nginx allowing you to cache core content of a page but dynamically replace any blocks of the page that are dynamic, for example login links / welcome text. But they can sometimes hold some gotchas, so heres a post on debugging SSI or debugging Nginx configs in general. I spent a morning debugging on Nginx when SSI's weren't acting as expected

January 31, 2010 12:05 PM

January 29, 2010

January 29, 2010 02:23 PM

Andrew Wakefield

After a hearing lasting two and a half years, the General Medical Council has decided that Andrew Wakefield acted unethically in his study which proposed a link between the MMR vaccine and autism.

It's now twelve years since his paper was published. During all of that time there has never been more of a tiny number of scientists who agreed with his findings. Unfortunately the few who do have been widely reported in the press and this has lead to a climate where a large proportion of the general public still believe that his findings are valid. Before Wakefield's study was published, 91% of children in the UK were receiving the vaccine. By 2003 that had fallen to 80%. in 2006 the UK had its first death from measles for fourteen years.

Wakefield should, of course, be ashamed of the effect he has had on the immunisation figures, but a lot of the blame must also be shouldered by the press who reported his findings as fact and who still refuse to admit that no link exists despite the number of studies which comprehensively disprove Wakefield's theories.

Reporting on yesterday's events, the Daily Mail still talk about "the controversial study" and devote a lot of column inches to interviewing people who still (with, it has to be emphasised, no scientific basis whatsoever) believe that Wakefield was right.

Yes, children contract autism. That's a sad fact. And there is a tendency for it to first be noticed at about the same time as the MMR vaccine is administered. But in study after study it has been shown that there is no causal link between the two events. Children who don't have the MMR vaccine are just as likely to become autistic as those who do.

But this is a classic example of a mob reaction that can't be turned off. The idea of the link is now out there and no amount of proof seems to counter that. You only have to read the comments on the Mail story to see that. The verdict from the GMC investigation is seen as Big Pharma trying to shut up the one man who is telling the truth. Or it's our Marxist Government trying to stifle dissent. Here are some examples:

Parents aren't daft and know full well the MMR damaged their children. He is a very brave man and a hero to many.
- Pippa, Notts, UK

A smear campaign was launched to discredit his findings.
In years to come I am sure that Dr Wakefield will be proved to have been correct in his beliefs about the MMR vaccine.
- Retired paediatric nurse, Surrey

Oh dear. Someone questioned the efficacy of expensive treatment courtesy of big pharma? Heaven forbid that someone would question the status quo of treating disease with expensive drugs for life, these alternatives might be more effective and actually cure, cheaply.
- tom bowden, perth australia

This is what happens under communist marxist Labour,,anyone who dares to disagree,rightly or wrongly is punished, now this doctor is struck off, this is to remind others not to dare confront these parasites, 3 months to go, then back to Democracy.
- jack, ashford.england

I particularly want to draw attention to this one:

If there is every any shred of doubt about the safety of a medicine, no responsible parent should even consider giving it to their children.
- Susie Squeegee, Leicester, England

I agree with this. If there is a shred of doubt then, of course, no-one should be expected to give it to their child. But there isn't any doubt about MMR. Any doubt was manufactured by a hysterical press misreporting a flawed study. I really wish there was some way that the journalists and editors responsible for publishing these stories could be held accountable for their actions. They have probably caused more (and more lasting) damage than the original study.

As you'd expect, Ben Goldacre has more detail on the affair. His book also covers this area in some depth and is highly recommended.

PCC: An Important Update

Last week I pointed you at a petition that you could sign to support our suggestions for reforming the PCC. Things went fine for a day or so and the number of signatures easily sailed past 500. Then, for reasons we still haven't understood, the petition vanished from the site where it was hosted. Despite valiant efforts by Tim Ireland to resolve the problem, the petition remains missing in action.

This leaves us with a huge problem. The deadline for submitting these proposals is approaching fast. In fact one of the deadlines is today. We no longer have access to the petition and we can't even get the list of names of people who had already signed.

There is one solution. We can all submit the suggestions independently by email.

If you support our suggestions (and even if you have already signed the petition), can you please email a copy of the suggestions to the two email addresses below. The suggestions are listed on my previous blog entry. The two email addresses are Vivien Hepworth, Chairman, (PCC) Independent Governance Review - governancereview@pcc.org.uk and Ian Beales, Code Committee Secretary, Editor's Code of Practive Committee ianbeales@mac.com. The deadline for the first address is today (hence the urgency) and the deadline for the second address is 31st January. I suggest you do what I'm going to do and send the same mail to both addresses this afternoon.

Apologies for the short notice and for asking you to support the same campaign twice.

More details over on Tim's blog.

January 29, 2010 02:18 PM

January 25, 2010

Karanbir Singh

Going to Fosdem 2010

Although, only for the 5th and 6th, will be back in London on the 6th evening.

- KB

Original post.

January 25, 2010 11:28 PM

January 25, 2010 01:49 PM

I think I’ve got the treadmill desk sorted out finally (see all posts in this series).

Here are the parts you will need for a cheap treadmill desk:

Treadmill (duh!). They all say you should buy an expensive treadmill, and they are right. The one I got was very cheap new, and has lots of problems, like … It forgets how far you’ve gone after 5 minutes (so if you go to grab a cup of tea, then it resets itself). It has massive RF interference problems (now solved — see below). And it picks up mobile phone signals and emits a loud disturbing noise.
How to get an expensive treadmill at a reasonable price? I suggest ebay. Now is about the time when those unwanted Christmas presents should start coming online …
RF filter. There’s only one I would recommend, which is the Tacima Mains Conditioner and Radio Frequency Interference Filter. Note that a standard “surge-protected socket” is not the same and won’t filter RFI. This looks like one of those bollox “audio buff” annealed cables that are aligned to the earth’s magnetic field and cost thousands of pounds. It’s not — this one really works, and it’s a reasonable price. Make sure you plug only the treadmill into this otherwise the leads going to other equipment will also act as antennae for the unwanted RF radiation.
Desk (aka “a piece of wood”). I taped a piece of wood to the arms of the treadmill with gaffer tape, and then I piled some boxes up to lift the laptop up to a comfortable height: a lo-tech, no cost solution which has proven sturdy and effective. Even food and drinks haven’t been a problem.
Padded trainers. Thick padded trainers have really made the difference for me. I happened to have some old trainers, but you don’t need to spend much money on this.
Anti-static wrist strap. Yes, with some combinations of clothes, trainers and humidity, you will start generating static. This can damage computer equipment easily, not forgetting that it bloody hurts! A cheap anti-static wrist strap cures the problem.

I’ve only been doing this for two weeks, but I feel noticeably better already. The other good thing is it doesn’t feel like exercise. I’ve done about 4 hours today, and most of the time I forgot I was on a treadmill.

Update

Some mainstream press articles about treadmill desks: 1, 2, 3, 4, 5.

Generating a plist file in rails

I recently wrote an iPhone app (Waiting for approval in the app store at the time of writing) that needed data exported from a website (recyclinggroupfinder.com). The simplest way of handling external data in an app it seems is using a plist file, so I wrote this to generate one for me.

First of all I made my action respond to the plist format:

Next I created a builder file to format the data:

Then register the MIME type at the bottom of environment.rb:

Mime::Type.register "text/plist", :plist

And that’s it! Well mostly. The XML file generated can be made significantly smaller by converting it into the binary plist format, run this on the command line in terminal after downloading the generated XML plist.

cat things_xml.plist | plutil -convert binary1 - -o things.plist

The resultant binary plist is almost half the size of the XML one, much better for inclusion in an iPhone app:

pleb:~ will$ ls -l things* -rw-r--r-- 1 will will 1247300 20 Jan 18:50 things.plist -rw-r--r--@ 1 will will 2110437 20 Jan 18:50 things_xml.plist

Of course it would be much better to generate the binary format directly, and the plist-official gem looks like it can handle that and I mean to investigate, but I wrote the XML version before finding the gem, and it works for me!

January 25, 2010 12:01 PM

January 22, 2010

January 22, 2010 07:25 PM

If you use Obj.magic, or if you have buggy libraries which use Obj.magic (cough Extlib cough), then sometimes you’ll end up corrupting the OCaml heap. An easy way to track these problems down is to add this checkpoint function near the top of your code:

let checkpoint p =
  Gc.compact ();
  prerr_endline ("checkpoint at position " ^ p)

then place calls to checkpoint "A"; (checkpoint "B" etc) around suspect code.

The checkpoint function does two things: Gc.compact () does a full major round of garbage collection and compacts the heap. This is the most aggressive form of GC available, and I’ve found that it’s highly likely to segfault if the heap is corrupted. The second statement, prerr_endline, prints a message to stderr and crucially also flushes stderr, so you’ll see the message printed immediately.

So the effect is that if the checkpoint function prints something, you can be very sure that your heap is not corrupt at that point in the program.

By placing these in and around suspect code, you can quickly narrow down the place where corruption happens.

My previous postings about the treadmill desk …

Quick stats. At 1.3 km/h (0.8 mph), each hour:

I burn 100 calories (according to the probably-inaccurate treadmill meter)
I walk 3000 steps (according to pedometer)
I walk 0.8 miles (obviously …)

I’m now doing around 6 hours per day.

I’ve learned that good footwear is essential for this. Flat-soled gym shoes are uncomfortable after about 2-3 hours. Bare feet were slightly better, but well-padded trainers are best.

BTW to spammers who try to add “treadmill” linkspam in comments, go get a life and a proper job that contributes to society.

January 22, 2010 12:04 PM

January 21, 2010

Huw Lynes

Flexlm License Servers and Firewalls

If you are lucky enough to run your flexlm servers on a tightly controlled corporate network then you probably just turn the firewall off on those servers and get on with your life. Everyone else goes through a certain amount of hair-pulling before they work out how to make flexlm play nicely with firewalls. So I’m writing this post to document the process as much for me as anyone else.

So let us say that you have bought five copies of Bob’s Magical Pony Viewer an awesome graphical client that you can run to show you ponies. In order for Bob to be sure that you only run five copies he has used flexlm to secure his software. You have received a network license for BobSoft that looks like:
SERVER license1 0000eeeeeeee 2020 VENDOR bobs_lm FEATURE PonyL bobs_lm 1.0 06-jan-2011 5 \ SIGN="EEEE EEEE EEEE EEEE EEEE EEEE EEEE EEEE \ EEEE EEEE EEEE EEEE EEEE EEEE EEEE EEEE"
So you think, great we can set that up with only port 2020 open on the license server and everything will excellent. Ponies for five concurrent users, hurray!

Except of course when you try that Pony Viewer adamantly claims that it can’t contact the server. Even though you can netcat/telnet to port 2020 on that server and the flexlm logs tell you that the server is running just fine.

It’s helpful at this point to have a copy of lmutil around to debug the problem. I don’t know where to get lmutil from as it came bundled with the license server software from one of our vendors. But it’s very useful when trying to work out what is going on.

So lets try some things.
#>lmutil lmstat -c 2020@license1 lmutil - Copyright (c) 1989-2004 by Macrovision Corporation. All rights reserved. Flexible License Manager status on Thu 1/21/2010 19:56

Error getting status: Server node is down or not responding (-96,7)

This is the point at which one normally starts with the hair-tearing. The thing to realise about a flexlm server is that it’s actually two daemons working together. The lmgrd which is running on port 2020 and the vendor daemon (in this case bob_lm) which will start up on a RANDOM port. What is even better is that the vendor daemon will choose a different random port every time you restart the license server.

While discussing this with some fellow sysadmins it turns out that there is another option you can add to flexlm license files which ends this misery. You can tell the vendor daemon to start on a specific port. Like so:
SERVER license1 0000eeeeeeee 2020 VENDOR bobs_lm port=2021 FEATURE PonyL bobs_lm 1.0 06-jan-2011 5 \ SIGN="EEEE EEEE EEEE EEEE EEEE EEEE EEEE EEEE \ EEEE EEEE EEEE EEEE EEEE EEEE EEEE EEEE"

And now when we try lmutil
#>lmutil lmstat -c 2020@license1 Flexible License Manager status on Thu 1/21/2010 20:06

License server status: 2020@license1
License file(s) on license1: /opt/BobSoft/license.dat:

license1: license server UP (MASTER) v11.6

Vendor daemon status (on license1):

bob_lm: UP v11.6

Hurray Ponies!

One last thing to note make sure the hostname you specify in the license file matches the hostname of the license server and also the hostname you use when connecting to the server. This is because flexlm sends the hostname asked for as part of the license request and if the two don’t match you won’t get any ponies.

In short flexlm is a dreadful license server it’s just that all the others are even worse.

January 21, 2010 08:12 PM

January 18, 2010

Press Complaints Commission

One of the major topics on this blog is the nonsense that is printed by a significant proportion of the British press. The press in this country is "controlled" by the Press Complaints Commission, but in the majority of cases the PCC either can't or won't adjudicate effectively.

Every year the PCC invites interested parties to submit suggestions of ways that its Editors' Code of Practice could be improved. Along with a group of other bloggers with an interest in this area, I've been thrashing out a list of suggestions to submit to the committee. We've come up with a list of five suggestions which we intend to send to the committee before the 31st January deadline. Our suggestions are as follows:

SUGGESTION ONE: Like-for-like placement of retractions, corrections and apologies in print and online (as standard).

Retractions, corrections, and apologies should normally be at least equally prominent to the original article, in both print and online editions. Any departure from this rule should only be in exceptional circumstances, and the onus on showing such circumstances should be on the publication.

SUGGESTION TWO: Original or redirected URLs for retractions, corrections & apologies online (as standard).

Retractions, corrections, and apologies in respect of online articles should always be displayed either at the original URL or at a URL to which the reader is redirected.

SUGGESTION THREE: The current Code contains no reference to headlines, and this loophole should be closed immediately.

Headlines should be covered by the same rules as the rest of a story. Further, headlines and titles for links should never be misleading in what they imply or offer and should always be substantiated by the article/contents.

SUGGESTION FOUR: Sources to be credited unless they do not wish to be credited or require anonymity/protection.

Sources should normally be credited. Any departure from this rule should only be when the source does not wish to be credited or if the source requires anonymity/protection.

SUGGESTION FIVE: A longer and more interactive consultation period for open discussion of more fundamental issues.

We submit all of the above without implying support for the PCC, the remainder of Code as it stands, or even the concept of self-regulation, and request that the 20th year of the PCC be marked with an open debate about its progress to date, and its future direction.

We think that this is a sensible set of suggestions and that no reasonable newspaper editor could disagree with them. Time will, of course, tell.

Now, we could put these suggestions forward as coming from a small group of bloggers but we'd like it to be wider than that. We've put up a petition where you can register your support for our ideas. If we can get a large number of people signing the petition then hopefully that will make it less likely that the PCC can dismiss our submission out of hand. If we can get some press interest around our campaign, then that would be even better.

So please sign the petition and please pass on the information to anyone else who might be interested. With your help we might just be able to change something.

For more details, see Tim's blog post on the campaign.

These suggestions were decided upon by Tim Ireland, Kevin Arscott, Adam Bienkov, Dave Cross, Sunny Hundal, Jack of Kent, Justin McKeating, MacGuffin, Mark Pack, septicisle, Jamie Sport, Clive Summerfield, Unity, Anton Vowl.

January 18, 2010 04:53 PM

January 16, 2010

Visualizing libvirt development history using gource and code swarm

Michael DeHaan yesterday posted an example using gource to visualize Cobbler development history. Development on Cobbler started in April 2006, making it a similar vintage to libvirt whose development started in November 2005. So I thought it would be interesting to produce a visualization of libvirt development as a comparison.

Head over to the YouTube page for this video if it doesn't show the option watch in highdef in this embedded viewer. HD makes it much easier to make out the names

Until July last year, libvirt was using CVS for source control. Among a great many disadvantages of CVS is that it does not track author attribution at all, so the first 3 & 1/2 years show an inaccurately small contributor base. Watching the video it is clear when the switch to GIT happened as the number of authors explodes. Even with the inaccuracies from the CVS history, it is clear from the video just how much development of libvirt has been expanding over the past 4 years, particularly with the expansion to cover VirtualBox and VMWare ESX server as hypervisor targets. This video was generated on Fedora 12 using

 # gource -s 0.07 --auto-skip-seconds 0.1 \
          --file-idle-time 500 --disable-progress \
          --output-framerate 25 --highlight-all-users \
          -1280x720 --stop-at-end --output-ppm-stream - \
 | ffmpeg -y -b 15000K -r 17 -f image2pipe -vcodec ppm \
          -i - -vcodec mpeg4 libvirt-2010-01-15-alt.mp4

gource isn't the only source code visualization application around. Last year a project called code swarm came along too. It has a rather different & simpler physics model to gource, not showing the directory structure explicitly. As a comparison I produced a visualization of libvirt using code_swarm too:

Head over to the YouTube page for this video if it doesn't show the option to watch in highdef in this embedded viewer. HD makes it much easier to make out the names

In this video the libvirt files are coloured into four groups, source code, test cases, documentation and i18n data (ie translated .po files). Each coloured dot represents a file, and each developer exerts a gravitional pull on files they have modified. For the years in which libvirt used CVS there were just a handful of developers who committed changes.This results in a visualization where developers have largely overlapping spheres of influence on files. In the last 6 months with GIT, changes have correct author attribution, so the visualization spreads out more accurately reflecting who is changing what. In the end, I think I rather prefer gource's results because it has a less abstract view of the source tree and better illustrates the rate of change over time.

Finally, can anyone recommend a reliable online video hosting service that's using HTML5 + Ogg Theora yet ? I can easily encode these videos in Ogg Theora, but don't want to host the 200 MB files on my own webserver since it doesn't have the bandwidth to cope.

January 16, 2010 10:01 PM

January 15, 2010

Customer Relationship Failure

Jessops Logo

I generally don't like getting marketing email. Whenever I buy something online or sign up to a new web site, I always make sure that the "please send me email" is not checked. This doesn't, of course, stop me getting marketing email but it does give me the moral high ground if I choose to complain about it.

There are, however, a small number of companies and organisations who I'm happy to receive mail from. Generally that's campaigns that I support and things like that. There are only a couple of retailers in that list. One of them is Jessops, the photographic people. Photography is a hobby that I don't have time to follow that closely, so I'm happy for Jessops to send me information about new products every few weeks. Sometimes it might even get me into a shop to buy something.

But over the last couple of months my patience with Jessops has been sorely tried. I think they have started to use a new CRM or mass-mail tool. Whatever the issue is, the result has been that the email I get from them has become really impersonal and, actually borders on being rude.

The problem is that somehow they have got the parts of my name confused. Perhaps they got the forename and surname fields the wrong way in some data import exercise. Or perhaps they are using the wrong data field in their mail merge process. But where they think they are writing email with friendly subjects like "Dave, 3 for 2 offers on photo products", I'm actually seeing "Cross, 3 for 2 offers on photo products".

To be honest, I prefer it if retailers address me as "Mr Cross" (I realise I may be a bit out of date there) but I'm reasonably happy for them to call me "Dave". Calling me "Cross" just isn't acceptable. The first time it happened I assumed it was a glitch that would be fixed before the next run. But I've since received three or four other messages with the same error in the subject line.

Each time one of those messages arrives it lowers my respect for Jessops. Each time I get closer to just removing myself from their mailing list. But I do still find the contents of the mail interesting. Today I got another message and I replied to it asking them to fix the problem. But I strongly suspect that the reply address won't go to a real person - that seems to be standard (if broken) practice these days.

It'll only take a couple more of these messages to push into unsubscribing. And Jessops will probably lose the small amount of custom that they currently get from me. I don't suppose they really care.

By the way. Whilst we're talking about Jessops, does anyone else think that their new slogan is a little excessive? Does anyone really go to a camera shop for "advice for life"?

January 15, 2010 12:40 PM

January 14, 2010

January 14, 2010 07:30 PM

The schedule of talks has been announced for the OCaml Users Meeting in Paris, France, in April. Here are my photos from the last meeting.

OCamlCore.org news and projects
Enforcing Type-Safe Linking using Inter-Package Relationships for OCaml Debian packages (Stefano Zacchiroli)
Ocamlviz (Julien Robert, Guillaume Von Tokarski, Sylvain Conchon, Jean-Christophe Filliâtre, Fabrice Le Fessant)
Cluster computing in OCaml (at mylife.com) (Gerd Stolpmann)
OCaml in a web startup (about ocsigen) (Dario Teixeira)
React, functional reactive programming for OCaml (Daniel Bünzli)
OASIS, a Cabal like system for OCaml (Sylvain Le Gall)

The second day will be the first “OCaml hacking day”. Bring a laptop and your OCaml questions.

Subscription is mandatory (and free) for both events. See the web pages for details: 1 2

January 14, 2010 11:20 AM

I’ve learned a lot about RFI this past couple of days … The treadmill radiates RFI through the power cable, back into the power socket, and from there through the entire house’s ring main. Ring mains, a UK peculiarity invented after the war to save copper, are particularly susceptible to this.

You can easily hear the interference using an analogue radio and placing the radio around the house with the treadmill on and off. This page (currently not available) has some nice graphs from a radio frequency analyzer.

The solution was to purchase a power strip with a big choke and plug the treadmill (and nothing else) into it:

Source: Amazon.co.uk

Although the lead from the treadmill to the power strip (and the treadmill triac itself) can still be heard on the radio emitting a lot of noise, it’s much better than having the entire ring main act as a huge antenna, and most importantly it no longer interrupts my ADSL line. Success!

Rev. Simon Rumble

Who should get my Haiti donation?

I've got a bit of a dilemma. I want to give some money for disaster relief in Haiti following the terrible earthquake. Problem is, none of the relief organisations active in Haiti meet my criteria for donations.

My criteria are:

Not a religious charity or affiliated with a religion.
Respects my request not to be spammed (email, phone or mail).
Does not use chuggers.
Does not spend inordinate amounts on administration and fundraising.

Oxfam uses chuggers and failed on the second item after my donation from the Indian Ocean Tsunami appeal. They also ignored my complaint about same.

Medecins Sans Frontieres, Red Cross also use chuggers.

Water Aid, much to my dismay as I respect the charity enormously, use chuggers.

Some will defend charities using these techniques, because the cause is too important. That doesn't wash with me because I expect a code of ethics to be applied across the organisation. I once had an argument with a telemarketer who called to solicit donations for Police Citizens Youth Clubs, along the lines of the government Do Not Call list excludes charities, as if that somehow excuses ignoring my expressly-stated preference.

Chuggers really got me annoyed when I worked on New Oxford Street in London and every time I set foot on Tottenham Court Road I'd be accosted by some slimebag raising money for him or herself in the guise of charity.

So can anyone suggest a charity worthy of my cash?

Contact me

January 14, 2010 01:06 AM

January 13, 2010

Dean Wilson

Grocery Arrival Excitement?

Many years ago, in the first dotcom boom, I worked for a website performance monitoring company. I was one of the early employees (developer number 3 and sysadmin number 2) and I remember being in a meeting with the company CEO who was telling us about a new pitch we were doing for $SUPERMARKET, they were going to try this new idea of shopping online and then delivering it to your door.

The worst part of it was that they didn't just want monitoring, they wanted a full transaction engine (with some basic OCR), a product I can probably get away with confessing that we didn't have at the time of the sales pitch. We all knew the deal, if we didn't get it life was going to be very hard there for the next six months, so we all knuckled under. The road was long, difficult and uphill in the snow in both directions but eventually we got to the day of the pitch. Which we aced in an astounding display of luck - the new app sometimes got itself in to a little bit of a state if their website had a failure - which it did about 20% of the time. They loved the demo and wanted us to give them full coverage while they did maintenance work. If we pulled it off then we'd pretty much get the deal, none of our competition at the time could match the features, it was just the uptime that was a little worrying.

So we went out and bought a dozen small desktops, monitors and networking kit, installed them all in our spare store room, put some tables and chairs in and had a company meeting. The management were completely open about what was happening, they took questions and then asked how far we'd go to help. We covered the whole weekend from Friday night to Monday morning. Nearly the entire company chipped in, from three letter titles to sales to dev to systems to HR. We had eyes on the machines over the whole period, including when the Solaris admin, the only person to let us down, didn't make his time slot. Out of all the transactions the worst was beans, they had a new version of the code on some of the servers and it'd return very odd results for beans and break the transaction runner in horrible ways. I'll never forget the 4am calls asking what we do when they offer you a lawn-mower instead.

I placed my first ever order online with the $SUPERMARKET yesterday and hopefully it should arrive in the next couple of hours. The interface may have changed and so many of its users take the service for granted that it's a little humbling to realise how much the Internet's changed so very many things. I guess this post's about a combination of things, the best job I ever had (the company was sold in the end to one of it's competitors. I left happy in the knowledge that we ate their lunch until they gave up trying to compete and bought us), how dedicated staff can be in the right environment, why you should push the boundaries of your industry and how sometimes even cans of beans can be exciting.

I had to put a single can in the order to complete the circle. Here's to hoping they don't charge me for a lawn-mower.

Update: They didn't deliver on the night, there was a "problem with the payment" so they took the money out, using the same details and delivered it two nights later. I'll class this one as a draw.

January 13, 2010 08:05 PM

January 12, 2010

Dean Wilson

Spreadsheets Vs Post-It Notes

I'm a fan of documentation, over the years I've ended up supporting more than one business critical system that has less documentation than you get from a cat /dev/null.

The only downside, and I've been bitten by a couple of things like this over the last week is the case of the spreadsheet vs the post-it note - if you have a lovely, well formatted and information dense spreadsheet that says "A is 1" and when you get to the server room the switch has a post-it, in bad scrawl, that says "B is 2" which do you believe?

January 12, 2010 11:32 PM

January 12, 2010 11:26 PM

Here’s a fun thing (not). I noticed today that the DSL connection was pretty rough, constantly reconnecting. It turns out after a lot of: run downstairs, check wires, DSL is OK, run back upstairs, still OK, start the treadmill and work, hmm it’s disconnected again, that this cheap treadmill is a big source of RFI that interferes with DSL.

Apparently this is a known problem: 1, 2, 3, 4.

(By the way, anyone considering getting the Roger Black GM-41001 Silver Medal Treadmill … DON’T! It may be very cheap, but it really is a piece of crap).

Why didn’t I get interference yesterday? Good question, but the router has been moved slightly, and perhaps the wire doesn’t pick it up so much.

Sadly, wrapping tinfoil around the phone line and earthing it did not work.

I ordered one of these mains conditioners with a big choke for £25 (about 10% of the cost of the treadmill). We shall see tomorrow if this works …

January 12, 2010 04:16 PM

Binary Dates

There's a strange meme going round that today is the last "binary date" for a hundred years. I'm not sure where it came from but a couple of people have repeated it on Twitter over the last couple of days.

A binary date is one which is made up completely of ones and zeroes. So in order to find any we'll need to ignore the "2" in "2010" and assume that we're writing dates in the shorthand dd/mm/yy format - or your local variation thereof.

In that form we can easily agree that, yes, 11/01/10 is a binary date. And so was yesterday (10/01/10) and the 1st of January (01/01/10). But to suggest that there are no more for a hundred years is clearly nonsense. Anyone suggesting that is guilty of not giving the matter the smallest amount of thought.

There are still six more binary dates to come this year (01/10/10, 10/10/10, 11/10/10, 01/11/10, 10/11/10 and 11/11/10) and another nine to come next year (01/01/11, 10/01/11, 11/01/11, 01/10/11, 10/10/11, 11/10/11, 01/11/11, 10/11/11 and 11/11/11).

After 11th November next year, there will be no more binary dates until the year 2100.

If you're going to pass on memes, at least check them for accuracy first.

Next NWRUG meeting 21st January – UNIX: Rediscovering the wheel

The original announcement is available on the NWRUG site.

This month John Leach of Brightbox will be talking on UNIX: Rediscovering the wheel.

“Those who don’t understand UNIX are condemned to reinvent it, poorly.”

We in the Ruby Community seem to have a habit of re-inventing things. Sometimes this is for good reason, but in some cases we don’t know we’re even doing it! We’re wasting valuable time that could be spent learning Erlang!

UNIX-like operating systems have been around for decades and lots of problems have come and gone in that time. I’m going to talk about some of the tools available that can be used to solve common Ruby and Rails deployment and development problems.

Brightbox will also be sponsoring the event so there will be pizzas available after the meeting in Odder across the road from the meeting.

Schedule

6:30pm :: Welcome & Pre-session bar visit.
7:00pm :: UNIX: Rediscovering the wheel by John Leach of Brightbox
7:45pm :: Pizzas at the Odder bar across the road sponsored by Brightbox

If you want more information email nwrug@willj.net, call Will on 07939 547 962 or tweet @will_j.

Sign Up

If you would like to attend this event please sign up here as the BBC need a list of attendees before the event, and I need to arrange the correct amount of food.

Location

This meeting is being held at one of our regular venues, the BBC Manchester main building on Oxford Road in central Manchester (Directions). If you get lost call Will on 07939 547 962.

January 12, 2010 02:49 PM

Whenever a link on your website opens in a new window a panda cries

You’ve got a great website. It’s amazing. It’s so good no-one will want to leave it. Ever. Here’s what you’re thinking:

OMG wow. Our website is amazing. It’s so good no-one will want to leave it. Ever. Let’s help users enjoy our website forever by making all external links open in new windows so then they close the other websites our site will still be open. Our users will thank us until the end of time for making it easier to stay on our site, and anyway Marketing said we had to do it and they know the internet better than anyone!

Sad Panda

Oh dear. Most people don’t know this, but making external website links open in a new window makes pandas sad. Look, here’s a sad panda made sad because it used a website that opened external links in new windows.

Doesn't this panda look sad?

You did that, with your new window link opening. (Panda by sholt).

Why Pandas cry

You need to consider that your website is going to be just one part of a users browsing session. The user will probably already have open tabs in their current browser window and the tab they have your website in will probably have history before your site. When you force a new window to open for a user you are interrupting their browsing flow. When this happens the user has a jarring user experience because of your website. Well done your website.

There are already controls in browsers to let users open links in new windows or tabs, in Safari they are the the first two options in the right-click context menu, or a Cmd+click:

Browser controls already exist giving the user control over where links open

When you force the user to open links from your website in a new window you are taking away control the user already has.

It’s a PITA and I have to work around it

Here’s what I personally do when your website opens a link in a new window:

Your website forces new window to open when I click on a link.
New window opens, I close it immediately.
On your website again I Cmd+click the link or right click and select ‘open in new tab’.
I close the tab your website was in and re-position the new tab with the new website in where the tab for your website used to be.
I mentally remove one karma point from your website in my internal website excellence tracker.

Look at the amount of messing around your website made me do. And now, because of this messing around, your website is no-longer accessible via my browser back button. You’ve succeeded in making your website even less accessible, the exact opposite of that you were trying to achieve.

Luckily I’m mentally tough much like Chuck Norris and so can take this two, maybe three times before cracking, but Pandas aren’t as tough as me. If this happened to a panda the panda would just cry. Sad.

January 12, 2010 01:57 PM

January 12, 2010 10:42 AM

How hard is it to write a bug tracking system that doesn’t suck? Quite hard it seems …

I spent ten minutes trying to work out how to enter a bug in a Flyspray-based BTS. And I gave up. I gave up entering a bug! Isn’t that, like, the most important single task that BTSes should enable me to do?

Trac is a pile of crap too. The enter a bug button is hidden away somewhere (it’s called “New ticket”), and the main interface seems to be arranged around producing arbitrarily chosen “reports”, none of which I have ever found to be in the slightest bit useful.

Mantis … urrrgghhh. OCaml upstream use this to track bugs and it’s horrible.

And .. Bugzilla. Oh Bugzilla. When I get my own starship, I hope it’ll be simpler to use (and faster) than Bugzilla.

I would encourage everyone to try out (proprietary) FogBUGZ for an example of a usable BTS. It’s proprietary software, but you can learn from it. You can learn that (a) most of your users won’t be BTS experts, and (b) you should encourage as much feedback from all your users as possible. That means doing usability tests and concentrating mainly on making it as easy to use as possible. It’s the most important thing for BTS authors to do. Clue: if your users cannot work out how to enter a bug, then you should go back to the drawing board.

January 11, 2010

Following gphoto SVN development with GIT

Since I started developing the Capa photo capture application, I've been following development of gphoto much more closely. Unfortunately gphoto is using subversion for source control. There are many things wrong with subversion in comparison to modern SCM systems like Mercurial or GIT. In this particular case though, the main problem is speed, or lack thereof. gphoto uses sourceforge as its hosting service and sf.net subversion servers are slower than you can possibly imagine. As an example, run 'svn log' to browse changes and you'll be waiting 30 seconds for it to even start to give you an answer. Then run 'svn diff' to look at the contents of a change and you'll be waiting another 30 seconds or more. Totally unacceptable. Once you've used a distributed SCM system like Mercurial or GIT, you cease to have tolerance for any operations which take longer than a 2-3 seconds.

Fortunately, GIT has the ability to checkout directly from SVN repository. The gphoto SVN repository actually contains many separate sub-projects in it and I didn't want to import them all to my local GIT repository. This meant I couldn't make use of the branch / tag tracking support directly and had todo things the long way. The good news is that the long way has already been blogged about and it isn't hard.

There were two projects I was interested in getting, libgphoto (the main library) & gphoto (the command line frontend) and I wanted each to end up in their own GIT repository. For both, I wanted the trunk and 2.4.x branch. Starting with gphoto, since it has much less history, the first step was to clone the trunk

# git svn clone https://gphoto.svn.sourceforge.net/svnroot/gphoto/trunk/gphoto2 gphoto2

This takes a fairly long time because it pulls down every single SVN changeset in the repository. Once that's complete though, the .git/config contains

[svn-remote "svn"]
        url = https://gphoto.svn.sourceforge.net/svnroot/gphoto/trunk/gphoto2
        fetch = :refs/remotes/git-svn

And the local 'master' branch is connected to the 'git-svn' remote.

$ git branch -a
* master
  remotes/git-svn

Anytime further changes are made in the SVN repository, those can be pulled down to the local GIT repository using git svn fetch git-svn. At this point it is possible to add in the branches. Simply edit the .git/config file and add another 'svn-remote' entry, this time pointing at the branch path.

[svn-remote "svn24"]
        url = https://gphoto.svn.sourceforge.net/svnroot/gphoto/branches/libgphoto2-2_4/gphoto2
        fetch = :refs/remotes/git-svn-2.4

And then pull down all the changes for that branch, and create a local branch for this

# git svn fetch svn24
# git checkout -b v2.4 git-svn-2.4

This leaves a local branch 'v2.4' and a remote branch 'git-svn-2.4'

$ git branch -a
  master
* v2.4
  remotes/git-svn
  remotes/git-svn-2.4

That takes care of the gphoto2 frontend command line app codebase. It is then a simply matter to repeat the same thing substituting libgphoto2 into the SVN paths to checkout the library codebase. Though this takes a little longer because it has much much more history. This little upfront pain to clone the SVN repo to GIT will be paid back many hundreds of times over thanks to the speed that GIT brings to SCM operation.

The moral of the story is simple: Don't ever choose subversion. If you have the choice, use GIT. If you don't have the choice, then mirror SVN to GIT anyway.

Edit: One thing I forgot to mention is that after setting up all branches, run a git gc on the repo. This will dramatically reduce the disk usage & speed up GIT operations further

$ du -h -c -s .
45M .
45M total
$ git gc
Counting objects: 3695, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (3663/3663), done.
Writing objects: 100% (3695/3695), done.
Total 3695 (delta 3081), reused 0 (delta 0)
$ du -h -c -s .
5.0M .
5.0M total

Going from 45 MB to 5 MB is quite impressive !

January 11, 2010 12:53 PM

January 11, 2010 11:56 AM

Finally a crappy photo of my treadmill desk:

Because of network problems, I didn’t manage to start using it until today. The layout is fine (laptop is actually on a pile of books not shown in the photo, in order to raise it to the correct height). The treadmill itself is cheap and nasty, and occasionally emits a disturbing electrical brrrr noise. Not all the time thankfully. Typing and using the trackpad and phone is no different from at a real desk — no problem at all. (I’m typing this at the desk). According to the electronic display I’m burning ~~more than 1 calorie a minute~~ about 95 calories per hour at 0.8 mph (1.3 km/h).

January 10, 2010

Using GObject Introspection + Gjs to provide a JavaScript plugin engine

In writing the Capa photo capture application, one of the things I wanted to support was some form of plugin engine to allow 3rd parties to easily extend its functionality. The core application code itself is designed to have a formal separation of backend and frontend logic. The backend is focused on providing the core object model & operation, typically wrapping external libraries like HAL, libgphoto, lcms in GObject classes, with no use of GTK allowed here. The primary frontend builds on this backend, to produce a GTK based user interface. It is also intended to build another frontend that provides a GIMP plugin.

Back to the question of plugins for the main frontend. If the goal is to allow people to easily write extensions, a plugin engine based on writing C code is not really very desirable. Firefox uses JavaScript for its plugin engine and this has been hugely successful in lowering the bar for contributors. Wouldn't it be nice if any GTK application could provide a JavaScript plugin engine ? Yes, indeed and thanks to the recent development of GObject introspection this is incredibly easy.

GObject introspection provides a means to query the GObject type system and discover all classes, interfaces, methods, properties, signals, all data types associated with their parameters and any calling conventions. This is an incredibly powerful capability with far reaching implications, the most important being that you will never again have to write a language binding for any GObject based library. There is enough metadata available in the GObject introspection system to provide language bindings in a 100% automated fashion. Notice I said "provide", rather than "generate" because if targetting a dynamic language (Perl, Python JavaScript) it won't even be necessary to auto-generate code ahead of time - everything can and will happen at runtime based on the introspection data. Say goodbye to hand written language bindings. Say goodbye to Swig. Say goodbye to any other home grown code generators.

Adding support for introspection

That's the sales pitch, how about the reality ? The Capa code is based on GObject and was thus ready & willing to be introspected. The first step in adding introspection support is to add some m4 magic to the configure.ac to look for the introspection tools & library. This is simple boilerplate code that will be identical for every application using GObject + autoconf

GOBJECT_INTROSPECTION_REQUIRED=0.6.2
AC_SUBST(GOBJECT_INTROSPECTION_REQUIRED)

AC_ARG_ENABLE([introspection],
        AS_HELP_STRING([--enable-introspection], [enable GObject introspection]),
        [], [enable_introspection=check])

if test "x$enable_introspection" != "xno" ; then
        PKG_CHECK_MODULES([GOBJECT_INTROSPECTION],
                          [gobject-introspection-1.0 >= $GOBJECT_INTROSPECTION_REQUIRED],
                          [enable_introspection=yes],
                          [
                             if test "x$enable_introspection" = "xcheck"; then
                               enable_introspection=no
                             else
                               AC_MSG_ERROR([gobject-introspection is not available])
                             fi
                          ])
        if test "x$enable_introspection" = "xyes" ; then
          AC_DEFINE([WITH_GOBJECT_INTROSPECTION], [1], [enable GObject introspection support])
          AC_SUBST(GOBJECT_INTROSPECTION_CFLAGS)
          AC_SUBST(GOBJECT_INTROSPECTION_LIBS)
          AC_SUBST([G_IR_SCANNER], [$($PKG_CONFIG --variable=g_ir_scanner gobject-introspection-1.0)])
          AC_SUBST([G_IR_COMPILER], [$($PKG_CONFIG --variable=g_ir_compiler gobject-introspection-1.0)])
        fi
fi
AM_CONDITIONAL([WITH_GOBJECT_INTROSPECTION], [test "x$enable_introspection" = "xyes"])

The next step is to add Makefile.am rules to extract the introspection data. This is a two step process, the first step runs g-ir-scanner across all the source code and the actual compiled binary / library to generate a .gir file. This is an XML representation of the introspection data. The second step runs g-ir-compiler to turn the XML data into a machine usable binary format so it can be efficiently accessed. When running g-ir-scanner on a binary, as opposed to a library, it is necessary for that binary to support an extra command line flag called --introspect-dump. I add this code the main.c source file to support that

#if WITH_GOBJECT_INTROSPECTION
    static gchar *introspect = NULL;
#endif

    static const GOptionEntry entries[] = {
        ...snip other options...
#if WITH_GOBJECT_INTROSPECTION
        { "introspect-dump", 'i', 0, G_OPTION_ARG_STRING, &introspect, "Dump introspection data", NULL },
#endif
        { NULL, 0, 0, 0, NULL, NULL, NULL },
    };

    ...parse command line args...

#if WITH_GOBJECT_INTROSPECTION
    if (introspect) {
        g_irepository_dump(introspect, NULL);
        return 0;
    }
#endif

Back to the Makefile.am rules. g-ir-scanner has quite a few arguments you need to set. The --include args provide the names of introspection metadata files for any libraries depended on. The -I args provide the CPP include paths to the application's header files. The --pkg args provide the names of any pkg-config files that code builds against. There are a few others too which I won't cover - they're all in the man page. The upshot is that the Makefile.am gained rules

if WITH_GOBJECT_INTROSPECTION
Capa-0.1.gir: capa $(G_IR_SCANNER) Makefile.am
        $(G_IR_SCANNER) -v \
                --namespace Capa \
                --nsversion 0.1 \
                --include GObject-2.0 \
                --include Gtk-2.0 \
                --include GPhoto-2.0 \
                --program=$(builddir)/capa \
                --add-include-path=$(srcdir) \
                --add-include-path=$(builddir) \
                --output $@ \
                -I$(srcdir)/backend \
                -I$(srcdir)/frontend \
                --verbose \
                --pkg=glib-2.0 \
                --pkg=gthread-2.0 \
                --pkg=gdk-pixbuf-2.0 \
                --pkg=gobject-2.0 \
                --pkg=gtk+-2.0 \
                --pkg=libgphoto2 \
                --pkg=libglade-2.0 \
                --pkg=hal \
                --pkg=dbus-glib-1 \
                $(libcapa_backend_la_SOURCES:%=$(srcdir)/%) \
                $(libcapa_frontend_la_SOURCES:%=$(srcdir)/%) \
                $(capa_SOURCES:%=$(srcdir)/%)

girdir = $(datadir)/gir-1.0
gir_DATA = Capa-0.1.gir

typelibsdir = $(libdir)/girepository-1.0
typelibs_DATA = Capa-0.1.typelib

%.typelib: %.gir
        g-ir-compiler \
                --includedir=$(srcdir) \
                --includedir=$(builddir) \
                -o $@ $

CLEANFILES += Capa-0.1.gir $(typelibs_DATA)

endif # WITH_GOBJECT_INTROSPECTION

After making those changes & rebuilding, it is wise to check the .gir file, since the g-ir-scanner doesn't always get everything correct. It may be necessary to provide annotations in the source files to help it out. For example, it got object ownership wrong on some getters, requiring annotations n the return values such as

/**
 * capa_app_get_plugin_manager: Retrieve the plugin manager
 *
 * Returns: (transfer none): the plugin manager
 */

The final step was add rules to the RPM specfile, which are fairly self-explanatory

%define with_introspection 0

%if 0%{?fedora} >= 12
%define with_introspection 1
%endif
%if 0%{?rhel} >= 6
%define with_introspection 1
%endif

%if %{with_introspection}
BuildRequires: gobject-introspection-devel
BuildRequires: gir-repository-devel
%endif


%prep
....
%if %{with_introspection}
%define introspection_arg --enable-introspection
%else
%define introspection_arg --disable-introspection
%endif

%configure %{introspection_arg}

%files
....
%if %{with_introspection}
%{_datadir}/gir-1.0/Capa-0.1.gir
%{_libdir}/girepository-1.0/Capa-0.1.typelib
%endif

That is all. The entire API is now accessible from Perl, JavaScript, Python without ever having written a line of code for those languages. It is also possible to generate a .jar file to make it accessible from Java.

Adding support for a JavaScript plugin engine

Since the API is now accessible from JavaScript, adding a JavaScript plugin engine ought to be easy at this point. There are in fact 2 competing JavaScript engines supporting GObject introspection, Gjs and Seed. Seed looks more advanced, documented & polished, but Gjs was what's in Fedora currently, so I used that. Again the first step was checking for it in configure.ac

AC_ARG_WITH([javascript],
      AS_HELP_STRING([--with-javascript],[enable JavaScript plugins]),
      [], [with_javascript=check])

if test "x$with_javascript" != "xno" ; then
  if test "x$enable_introspection" = "xno" ; then
    if test "x$with_javascript" = "xyes"; then
      AC_MSG_ERROR([gobject-introspection is requird for javascript plugins])
    fi
  fi

  PKG_CHECK_MODULES(GJS, gjs-1.0 >= $GJS_REQUIRED)
  AC_SUBST(GJS_CFLAGS)
  AC_SUBST(GJS_LIBS)

  PKG_CHECK_MODULES(GJS_GI, gjs-gi-1.0 >= $GJS_REQUIRED)
  AC_SUBST(GJS_GI_CFLAGS)
  AC_SUBST(GJS_GI_LIBS)

  with_javascript=yes
  AC_DEFINE([WITH_JAVASCRIPT], [1], [enable JavaScript plugins])
fi
AM_CONDITIONAL([WITH_JAVASCRIPT], [test "x$with_javascript" = "xyes"])

I won't go into any details on the way Capa scans for plugins (it uses $HOME/.local/share/capa/plugins//main.js), merely illustrate how to execute a plugin once it has been located. The important object in the Gjs API is GjsContext, providing the execution context for the javascript code. It is possible to have multiple contexts, so each plugin is independent and potentially able to be sandboxed. The JavaScript file to be invoked is main.js in the plugin's base directory. The first step is to setup the context's search path to point to the plugin base directory:

void runplugin(const gchar *plugindir) {
    const gchar *searchpath[2];
    GjsContext *context;

    searchpath[0] = plugindir;
    searchpath[1] = NULL;

    context = gjs_context_new_with_search_path((gchar **)searchpath);

The context is now ready to execute some javascript code. The Capa plugin system expects the main.js file to contain a method called activate. To start the plugin, we can thus simply evaluate const Main = imports.main; Main.activate();

   const gchar *script = "const Main = imports.main; Main.activate();";

   gjs_context_eval(context,
                     script,
                     -1,
                     "main.js",
                     &status,
                     NULL);

   if (status !=0) {
     fprintf(stderr, "Loading plugin failed\n");
   }

Presto, you now have a javascript plugin running, having written no JavaScript at any point in the process. There is one slight issue in this though - how does the plugin get access to the application instance ? One way would be to provide a static method in your API to get hold of the application's main object, but I really wanted to pass the object into the plugin's activate method. This is where I hit Gjs's limitations - there appears to be no official API to set any global variable except for ARGV. After much poking around in the Gjs code though I discovered an exported method, which wasn't in the header files

JSContext* gjs_context_get_context(GjsContext *js_context);

And decided to (temporarily) abuse that until a better way could be found. I have an object instance of the CapaApp class which I wanted to pass into the activate method. The first step was to set this in the global namespace of the script being evaluated. Gjs comes with an API for converting a GObject instance into a JSObject instance which the runtime needs. Thus I wrote a simple helper

static void set_global(GjsContext *context,
                       const char *name,
                       GObject *value)
{
    JSContext *jscontext;
    JSObject *jsglobal;
    JSObject *jsvalue;

    jscontext = gjs_context_get_context(context);
    jsglobal = JS_GetGlobalObject(jscontext);
    JS_EnterLocalRootScope(jscontext);
    jsvalue = gjs_object_from_g_object(jscontext, value);
    JS_DefineProperty(jscontext, jsglobal,
                      name, OBJECT_TO_JSVAL(jsvalue),
                      NULL, NULL,
                      JSPROP_READONLY | JSPROP_PERMANENT);
    JS_LeaveLocalRootScope(jscontext);
}

There was one little surprise in this though. The gjs_object_from_g_object method will only succeed if the current Gjs context has the introspection data for that object loaded. So it was necessary to import my application's introspection data by eval'ing const Capa = imports.gi.Capa. That done, it was now possible to pass variables into the plugin. The complete revised plugin loading code looks like

void runplugin(CapaApp *application, const gchar *plugindir) {
    const gchar *script = "const Main = imports.main; Main.activate(app);";
    const gchar *searchpath[2];
    GjsContext *context;

    searchpath[0] = plugindir;
    searchpath[1] = NULL;

    context = gjs_context_new_with_search_path((gchar **)searchpath);
   
    gjs_context_eval(context,
                     "const Capa = imports.gi.Capa",
                     -1,
                     "dummy.js",
                     &status,
                     NULL);

    set_global(context, plugin, "app", application);

    gjs_context_eval(context,
                     script,
                     -1,
                     "main.js",
                     &status,
                     NULL);

    if (status !=0) {
      fprintf(stderr, "Loading plugin failed\n");
    }

This code is slightly simplified, omitting error handling, for purposes of this blog post, but the real thing is not much harder. Looking at the code again, there is really very little (if anything) about the code which is specific to my application. It would be quite easy to pull out the code which finds & loads plugins into a library (eg "libgplugin"). This would make it possible for any existing GTK applications to be retrofitted with support plugins simply by generating introspection data for their internal APIs, and then instantiating a "PluginManager" object instance.

In summary, GObject Introspection is an incredibly compelling addition to GLib. With a mere handful of additions to configure.ac and Makefile.am, it completely solves "language bindings" problem for you. I'd go as far as to say that this is a single most compelling reason to write any new C libraries using GLib/GObject. Furthermore if there are existing C libraries not using GObject, then provide a GObject wrapper for them as a top priority. Don't ever write or auto-generate a language binding again. Writing GTK applications either entirely in JavaScript, or in a mix of C + JavaScript plugins is also a really nice development, avoiding the issue of "clashing runtime environments" seen when using Python + GTK. The Gjs/Seed/GObject developers deserve warm praise for these great enhancements.

January 10, 2010 09:14 PM

January 08, 2010

Flushing memcached servers from Ruby

In Flushing memcached servers the easy way I highlighted a way to flush a memcached server without restarting it:

$ echo ”flush_all” | nc localhost 11211

However I almost never use the actual shell version of this, mostly I do the equivalent in Ruby by opening up a socket and communicating through that. Here’s a simple example:

socket = TCPSocket.new( '127.0.0.1', 11211 ) socket.write( "flush_all\r\n" ) result = socket.recv(2) puts "Success!" if result == 'OK' socket.close

January 08, 2010 12:25 PM

January 07, 2010

Using QCow2 disk encryption with libvirt in Fedora 12

Without ~~much~~any fan-fare we slipped a significant new feature into libvirt in Fedora 12, namely the ability to encrypt a virtual machine's disks. Ordinarily this would have been widely publicised as a Fedora 12 release feature, but the code arrived into libvirt long after the Fedora feature writeup deadline. Before continuing, special thanks are due to Miloslav Trma? who wrote nearly all of this encryption/secrets management code for libvirt!

Why might you want to encrypt a guest's disk from the host, rather than using the guest OS's own block encryption capabilities (eg the block encryption support in anaconda) ? There's a couple of reasons actually...

The host is using a network filesystem (like NFS/GFS) for storing guest disks, a guarantee is required that no one can snoop on any guest data, regardless of guest OS configuration.
Guest OS can boot without needing any password prompts, since libvirt can supply the decryption key directly to QEMU on the host side when launching the guest.
Allows integration with a key management server. Libvirt provides APIs for setting the keys associated with a guest's disks. An key management service can use these APIs to set/clear the keys for each host to match the list of guests it is intended to run.

There are probably more advantages to managing encryption on the virtualization host but I'm not going to try to think about them now. Instead the rest of this posting will give a short overview of how to use the new encryption capabilities

Secret management

There are many objects managed by libvirt which can conceivably use/require encryption secrets. It is also desirable that libvirt be able to integrate with external key management services, rather than always having to store secrets itself. For these two reasons, rather than directly set encryption secrets against virtual machines, or virtual disks, libvirt introduces a simple set of "secrets" management APIs. The first step in using disk encryption is thus to define a new secret in libvirt. In keeping with all other libvirt objects, a secret is defined by a short XML document

# cat demo-secret.xml
<secret ephemeral='no' private='no'>
  <uuid>0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f</uuid>
  <usage type='volume'>
    <volume>/home/berrange/VirtualMachines/demo.qcow2</volume>
  </usage>
</secret>

The "ephemeral" attribute controls whether libvirt will store a persistent copy of the secret on disk. If you had an external key management server talking to libvirt you would typically set this to 'yes', so that keys were never written to disk on individual virtualization hosts. Most people though will want to set this to 'no', so that libvirt stores the secret, otherwise you'll loose all your keys when you reboot which probably isn't what you want ! When running against a privileged libvirtd instance (eg with the qemu:///system URI), secrets are stored in /etc/libvirt/secrets, while when running unprivileged (qemu:///session), secrets are stored in $HOME/.libvirt/secrets.

The "private" attribute controls whether you can ask libvirt to give you the value associated with a secret. When it is 'yes', secrets are "write only", once you've set the value, libvirt will refuse to tell you what it is. Again this is useful if you are using a key management server, because it allows it to load a secret into libvirt in order to start a guest, without allowing anyone else who is connected to libvirt to actually see what its value is.

The "uuid" is simply a unique identifier for the secret, when defining a secret this can be left out and it will be auto-generated.

Finally the "usage" element indicates what object the secret will be used this. This is not technically required, but when you have many hundreds of secrets defined, it is useful to know what objects they're associated with, so you can easily purge secrets which are no longer used/required.

Having created the XML snippet for a secret as above, the first step is thus to load the secret definition into libvirt. If you are familiar with libvirt API/command naming conventions, you won't be surprised to find out that this is done using the 'virsh secret-define' command

  # virsh secret-define demo-secret.xml
  Secret 1a81f5b2-8403-7b23-c8d6-21ccc2f80d6f created

Notice how we have not actually set the value of the secret anywhere, merely defined metadata. We explicitly choose not to include the secret's value in the XML, since that would increase the risk of it being accidentally exposed in log files / bug reports / etc, etc. Thus once the secret has been defined, it is neccessary to set its value. There is a special virsh command for doing this called 'virsh secret-set-value', which takes two parameters, the UUID of the secret and then the value in base64. If you're one of these people who can't compute base64 in your head, then there's of course the useful 'base64' command line tool

  # MYSECRET=`echo "open seseme" | base64`
  # virsh secret-set-value 0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f $MYSECRET
  Secret value set

There are a few other virsh commands available, for managing secrets, but those two are the key ones you need to know in order to provision a new guest using encrypted disks. See the 'virsh help' output for the other commands

Virtual disks

Being able to define secrets isn't much fun if you can't then put those secrets to use. The first interesting task is probably to create an encrypted disk. At this point in time, libvirt's storage APIs only support encryption of the qcow1, or qcow2 formats. It would be very desirable to support dm-crypt too, but that's an outstanding feature request for someone else to implement. I've got a directory based storage pool configured in my libvirt host which points to $HOME/VirtualMachines, defined with the following XML

# virsh pool-dumpxml VirtualMachines
<pool type='dir'>
  <name>VirtualMachines</name>
  <source>
  </source>
  <target>
    <path>/home/berrange/VirtualMachines</path>
  </target>
</pool>

To create a encrypted volume within this pool it is neccessary to provide a short XML document describing the volume, what format it shoould be, how large it should be, and what secret it should use for encryption

# cat demo-disk.xml
<volume>
  <name>demo.qcow2</name>
  <capacity>5368709120</capacity>
  <target>
    <format type='qcow2'/>
    <encryption format='qcow'>
      <secret type='passphrase' uuid='0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f'/>
    </encryption>
  </target>
</volume>

Notice that we set the volume format to 'qcow2' since that is the type of disk we want to create. The XML then has the newly introduced "encryption" element which says that the volume should be encrypted using the 'qcow' encryption method (this is the same method for both qcow1, and qcow2 format disks). Finally it indicates that the 'qcow' encryption "passphrase" is provided by the secret with UUID 0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f. The disk can now be created using the "virsh vol-create" command, for example,

# virsh vol-create VirtualMachines demo-disk.xml
Vol demo.qcow2 created from demo-disk.xml

An oddity of the qcow2 disk format is that it doesn't actually need to have the encryption passphrase at the time it creates the volume, since it only encrypts its data, not metadata. libvirt still requires you set a secret in the XML at time of creation though, because you never know when qcow may change its requirements, or when we might use a format that does require the passphrase at time of creation. Oh and if you are cloning an existing volume, you would actually need the passphrase straight away to copy the data.

Virtual Machines

Now that we have created an encrypted QCow2 disk, it would be nice to use that disk with a virtual machine. In this example I've downloaded the PXE boot initrd.img and vmlinuz files for Fedora 12 and intend to use them to create a brand new virtual machine. So my guest XML configuration will be setup to use a kernel+initrd to boot, and have a single disk, the encrypted qcow file we just created. The key new feature in the XML format introduced here is again the "encryption" element within the "disk" element description. This is used to indicate what encryption method is used for this disk (again it is the 'qcow' method), and then associates the qcow decryption 'passphrase' with the secret we defined earlier

# cat demo-guest.xml
<domain type='qemu'>
  <name>demo</name>
  <memory>500200</memory>
  <vcpu>4</vcpu>
  <os>
    <type arch='i686' machine='pc'>hvm</type>
    <kernel>/home/berrange/vmlinuz-PAE</kernel>
    <initrd>/home/berrange/initrd-PAE.img</initrd>
    <boot dev='hd'/>
  </os>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/berrange/VirtualMachines/demo.qcow2'/>
      <target dev='hda' bus='ide'/>
      <encryption format='qcow'>
        <secret type='passphrase' uuid='0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f'/>
      </encryption>
    </disk>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes'/>
  </devices>
</domain>

With that XML config written, it is a simple matter to define a new guest, and then start it

# virsh define demo-guest.xml
Domain demo defined from demo-guest.xml

# virsh start demo
Domain demo started

# virt-viewer demo

If everything has gone to plan upto this point, the guest will boot off the kernel/initrd, hopefully taking you into anaconda. Everything written to the guest disk will now be encrypted using the secrets defined.

Future work

You'll have noticed that all these examples are using the low level virsh command. Great if you are the king of shell scripting, not so great if you want something friendly to use. So of course this new encryption functionality needs to be integrated into virt-install, and virt-manager. They should both allow you say that you want an encrypted guest, prompt you for a passphrase and then setup all the secrets automatically from there.

The libvirtd daemon has the ability to store secrets and their values persistently on disk, but this is not really secure, since the secrets are stored in unencrypted base64 format ! Clearly the next step here is for libvirtd to at the very least have the option of using gpg to encrypt the base64 files. The problem is that this then introduces a boot-strapping problem - what key does libvirt use for gpg ! This is a familiar problem to anyone who's ever had to setup apache with SSL and wondered how to give apache the key to decrypt its SSL server key upon host startup.

As mentioned earlier on, the libvirt secrets management public API was designed to be easy to integrate with external key management services. For a desktop virtualization application like virt-manager there is an opportunity to integrate with gnome-keyring (or equivalent). When defining secrets in libvirt, virt-manager would mark them all as ephemeral so that libvirt never stored them in itself. At the time of starting a guest, virt-manager would query gnome-keyring for the disk decryption keys, and pass them onto libvirt. This would ensure no one could ever run your guests unless they are able to login & authenticate to your gnome-keyring service. A server virtualization application like oVirt could do much the same, perhaps storing keys in FreeIPA (if it had such a capability).

Being restricted to qcow2 disk formats isn't all that nice because qcow2 isn't the fastest virtual disk format to start off with, and adding encryption doesn't improve matters. Many people, particularly in server virtualization environments, prefer to use LVM or raw block devices (SCSI/iSCSI). There are hacks which let you tell QEMU to write to the block device in qcow2 format, but they make me feel rather dirty. The kernel already comes with a generic block device encryption capability in the form of 'dm-crypt'. libvirt really ought to support creation of encrypted block devices using dm-crypt.

January 07, 2010 12:54 PM

January 07, 2010 10:48 AM

Libvirt is a great way to manage all sorts of virtualization systems (Xen, KVM, VMWare, OpenVZ, etc). And there are some nice command line and graphical tools built on top of it like virsh and virt-manager. It’s very mature, having been in development for something like 4 years.

But for some reason, not everyone in the world is using libvirt! Why don’t / can’t / won’t you use libvirt? Share your pain …

Rev. Simon Rumble

Getting rid of my landline

I'm getting rid of my landline. And no, not just to avoid the Telstra tax, but because the people who call land lines have no social graces. The only people who call landlines are old people and telemarketers.

Telemarketers are scum and I don't want to talk with them, especially since our number is on the Do Not Call list and they're willing to ignore that (yes I know you're a charity, but that doesn't give you an excuse for rudeness).

Old people have no understanding of the social graces modern technology allows. They think the primary advantage of a mobile phone is that the caller can contact the callee anywhere, anytime. The real advantage of mobiles is the ability to switch it off and divert to voicemail, to deal with later. My Mum answers the phone during dinner, during her favourite (untimeshifted) TV shows, any time it rings. Hell, she's polite to telemarketing scum.

Old people don't get this. Yesterday I had one trying to call me while I'm at work. I rejected the call and diverted to voicemail. He hangs up without calling leaving a message. He calls the home phone, where Holly and Louis get woken up, massively improving everyone's mood. It goes to the answering machine. The caller doesn't leave a message. And redials the home phone. Then calls my mobile again.

Now granted, in a genuine emergency this might be reasonable, but if I pick up the phone in these circumstances and there's no emergency, don't get all offended when I tell you to stop being so rude and hang up.

But the oldies don't get it. They're PAYING to talk to you, so you should answer. They come from a time before texts, voicemail, answering machines. Hell my Mum's first home phone was a "party line" shared with half the street so they didn't even have privacy.

So I'm gonna reduce the options. I've had no luck getting the message through that serial dialling isn't on, so I'm dropping the home phone. I'll use voip for cheaper outbound calls. And no, if you're over 50 you don't get to have my work landline number either.

Contact me

January 07, 2010 12:43 AM

January 06, 2010

January 06, 2010 06:45 PM

Somehow I managed to mess up a RHEL 6 Alpha machine’s SELinux configuration so seriously that I could no longer log in:

$ ssh root@192.168.122.242
root@192.168.122.242's password:
Unable to get valid context for root
Last login: Wed Jan  6 18:34:46 2010
Connection to 192.168.122.242 closed.

However this is easy to fix with guestfish. Important: before you do this, make sure the guest is shut off.

# guestfish -i RHEL6Alpha

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help with commands
      'quit' to quit the shell

><fs> ls /
.autofsck
bin
boot
dev
etc
home
lib
lib64
lost+found
media
mnt
opt
proc
root
sbin
selinux
srv
sys
tmp
usr
var
><fs> touch /.autorelabel
><fs> sync
><fs> exit

It’ll fix the SELinux labels for you when it boots. Lo and behold I can log in again.

Karanbir Singh

This is my screenrc - whats yours

Everyone uses screen right ? If not, what's wrong with you crazy person ?

Also everyone at some point or the other realises that its worth getting a decent screenrc in place. So did I, many years back. Essentially the screenrc lets you setup a few things about how you want screen to look and work. A config file, yes. And here is what I currently use:

caption string "%?%F%{= Bk}%? %C%A %D %d-%m-%Y %{= kB} %t%= %?%F%{= Bk}%:%{= wk}%? %n " hardstatus alwayslastline hardstatus string '%{= kG}[ %{G}%H %{g}][%= %{= kw}%?%-Lw%?%{r}(%{W}%n*%f%t%?(%u)%?%{r})%{w}%?%+Lw%?%?%= %{g}][%{B} %d/%m %{W}%c %{g}]'

You can download it here : http://www.karan.org/stuff/MyScreenRc : remember to move it to ~/.screenrc .

What you see here is the result of what came from many iterations of changes. I remember in 2006 my screenrc put up a 3 line display with everything that anyone would ever need, including load on a machine, number of unread emails, days-for-present-sprint to end etc. And to be honest, while most of that is good to know I think the only things one needs on the console are : hostname, screen windows and labels for those and finally the time. On a nice dull background so its not in your face too much. Which is exactly what my existing screenrc is setup to be.

One interesting thing is that often when pairing with me or when people see my console shell they would ask why bother with the hostname, specially since you should know from the shell prompt what machine you are on. Well, yes - but then that does not work out too well when you start cascading shells between machines. Eg: machine 1 -> machine 2 -> machine 3. It can get tricky, but because I have my status bar setup on each shell - here is what my terminal looks like in that case:

stacked screens

So you can easily see how the screens are stacked up and on what machine with screen windows on each machine. Quite like that.

So do you have a screenrc setup ? tell me about it. If not - then well, this one here should be a good place to start from - and tell me if you like it, hate it or think it can be improved in some way.

- KB

Original post.

January 06, 2010 04:38 PM

January 06, 2010 01:32 PM

Snow and Global Warming

I'm reading reports that Tory MP Ann Winterton has made a complete fool of herself in today's Prime Minister's Questions. She has suggested that the government should cancel "wasteful expenditure" on wind farms as the current weather clearly demonstrates that global warming is nonsense.

Despite being complete nonsense, this is an argument that I hear on a regular basis. It's trotted out by someone every time the weather takes a turn for the worse. Here's another Tory MP, Douglas Carswell[1], making the same claim during December's snows. [Update: And here's the the front page of today's Express: Snow Chaos - And They Still Claim It's Global Warming]

I was about to write a post pointing out the obvious flaws in this argument, but there was a feeling of deja vu niggling away at the back of my head. Then I remembered what I wrote during the snow we had last February.

[1] You might want to take a look around Carswell's site. There's a lot of nonsense there. I'm planning to cover him in more detail in the coming weeks.

January 06, 2010 09:00 AM

We worked out yesterday (thanks Matthias Bolte) how to use the libguestfs tools like virt-inspector and guestfish to examine VMWare VMs.

VMWare’s native disk format is VMDK, which is only partially understood by free tools like qemu-img. qemu-img breaks quite badly on the newer variant that ESX 4 uses. Then there’s the issue of how you get the VMDK file from VMWare’s storage. Use their proprietary storage APIs?

Well it turned out both problems could be solved easily. VMWare ESX servers make the storage available over https connections, so you can use a URL like https://root:password@esxserver/folder/ to browse available storage on the server. And VMWare also makes the raw (“flat”) disk images available in the same way.

libvirt has supported ESX management for a while (thanks again to Matthias Bolte), so you can do:

$ sudo virsh -c esx://192.168.2.121/?no_verify=1 list --all
Enter username for 192.168.2.121 [root]:
Enter root password for 192.168.2.121:
 Id Name                 State
----------------------------------
  - TestLinux            shut off
  - TestWin              shut off

(Note that the domains must be shut off before VMWare will allow you to access the flat disk images).

Then we can get the storage URL:

$ sudo virsh -c esx://192.168.2.121/?no_verify=1 dumpxml TestLinux > /tmp/xml
$ grep '<source file' /tmp/xml
      <source file='[Storage1] TestLinux/TestLinux.vmdk'/>

And from that storage URL you can grab the disk image directly:

$ wget --no-check-certificate 'https://root:password@192.168.2.121/folder/TestLinux/TestLinux-flat.vmdk?dcPath=ha-datacenter&dsName=Storage1'

The large flat file downloaded is a straight raw disk image that can be examined directly in programs like guestfish and virt-inspector:

$ virt-list-filesystems -al TestLinux-flat.vmdk
/dev/sda1 ext4
/dev/vg_testlinux/lv_root ext4
/dev/vg_testlinux/lv_swap swap

$ guestfish --ro -a TestLinux-flat.vmdk -m /dev/vg_testlinux/lv_root

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help with commands
      'quit' to quit the shell

><fs> ll /
total 116
dr-xr-xr-x.  23 root root  4096 Dec 30 06:27 .
dr-xr-xr-x   29 root root     0 Dec 21 07:59 ..
-rw-r--r--.   1 root root     0 Dec 30 06:27 .autofsck
drwx------.   3 root root  4096 Dec 17 11:58 .dbus
-rw-r--r--.   1 root root     0 Dec 17 12:50 .readahead_collect
dr-xr-xr-x.   2 root root  4096 Dec 17 12:11 bin
[etc]

It’s probably also possible to avoid the download step, since libguestfs is built on qemu which should support http(s) connections directly, but I didn’t try this yet.

January 05, 2010

January 05, 2010 10:20 PM

In other news security theatre, meet paedo-terror!

I’m feeling distinctly unfearful about this. Terrorism is incredibly rare, I’m far more likely to die of circulatory disease or cancer. I’m more likely to die of a lightning strike. What is wrong with me?!

Ross Lawley

2010 resolutions and predictions

Resolutions Blog and blog often. I work on some cool and varied technologies and software at Global Radio. As I've learnt loads from leading a team of python developers, writing and supporting a CMS and its ecosystem of software for our various radio websites, its high time I shared and reflected on those learnings. So to kickstart my return to blogging, I've been telling various members of

January 05, 2010 09:41 PM