public service announcement

Sometimes, somebody does a web site listing protests against the cuts so you don't have to.

fail

Odd thought, working on NetworkX visualisations of lobbying activity. So much of whether the finished image is any use or not is an aesthetic decision, nothing to do with the actual graph metrics in any formal or programmatic way.

Randomly fiddle with the scaling factor, or increase the multiplier you apply to the nodes' link degree, and things look different - and possibly better. Also, more information may not be better - I can't help but think it was a lot easier to derive conclusions before I gave the various ministries node weights to reflect their relative power and divided the ministry weighting on each link by the number of organisations taking part in the meeting, which should be an improvement.

Also, after sweating quite a bit over this, does anyone know if NetworkX even lets you position nodes using polar coordinates? This looks useful.



This is far from ideal, but it should give some idea. This one is using the standard force-directed spring layout and just the data I have from June, with the nodes coloured by their degree in the chart and sized by their meeting count. Downing Street (the large purple plook) is fixed in place and the others adjusted. I rather like the fact that the lobbyists look like bacteria swarming into a cell, or possibly sperm trying to fertilise David Cameron.

An interview with Radio Free Europe

So Viktor Bout got extradited at last. As a result, I was interviewed by Radio Free Europe's Irina Lagunina. They wanted to hear about how the whole Viktor-blogging project got started - I told them about the Defence Energy Support Center files and the T-DODAACs and the like, and the plane spotter websites, and the fact I basically borrowed the idea from the people who were trying to monitor CIA rendition flights.

They wanted to hear about bloggers, and I thanked everyone who took part in Operation Firedump and made the point that all sorts of people across the ideological spectrum had taken part. They asked about my feelings about the extradition, and I said that I'd done the rejoicing when he was arrested and had since been mostly interested in seeing if there was any noticeable change in air movements through the UAE. What really cheered me was when the UAE government kicked out the An-12 operators.

They also wanted to know about the Russian government using VB's companies to move humanitarian aid. Specifically, they referred to something in Doug Farah's book that apparently quotes me on this. Unfortunately, my review copy never turned up so I'm completely in the dark as to what this might be. I said that given the numbers of heavy, ex-Soviet tactical airlifters that he controlled at the peak of his career, it was unlikely that many humanitarian agencies had been able to avoid occasionally dealing with him, and that they were probably right on balance to do the job rather than worry too much. If you need to move 40 tons of drinking water or flour labelled "Gift of the European Union" or whatever before people starve or get cholera, I'm not going to whine about it.

Recombination

In the last month or so I've begun adding blogs to my RSS queue again. I'm surprised by this; I didn't expect to find that the form still had so much energy. Perhaps this isn't so optimistic after all. Meanwhile, watch this space.s

le bateau ivre, port of call: Mogadishu

I wrote about this piece about brewing beer in southern Sudan, and incidentally creating a power station and a water works and a tiny industrial working class.

Well, SAB-Miller did a scenario-planning exercise about this sort of thing and came up with some truly odd answers. In the worst-case scenario:

a market with limited access to water and high energy costs, where people would migrate from areas of water shortage or turbulent weather

their proposed solution involved installing a brewery aboard a ship, so it could sail to wherever the thirsty masses had pitched up. Interestingly, as the story was released through the Grauniad's "Guardian Professional" advertorial division, this is also something SABMiller management was willing to pay good money to tell us about. Time was when big business wanted to keep its apocalyptic fantasies secret.

(Tagged "uncategorized", but should probably be under "uncategorizable".)

local news: power cuts

We keep having power cuts. In eight days, we had three, all of them between 5.30am and 6am, which all lasted most of the morning. I know exactly when they happen, because the smoke alarms start beeping and wake me up. My partner claims she was warmer on a demo than she was in the flat during no.2. As they seem to follow a pattern of happening at the same time every other day - just about when power demand starts to turn back up in earnest - I was wide awake at 0545 today waiting for the plaintive beeps. But no - looking at the chart, the ramp-up is later on a Sunday. Mind you, later in the day the lights flickered repeatedly for half a minute.

So I rang up UK Power Networks' (what used to be EDF Energy Networks, what used to be the London Electricity Board) press office and announced myself as a blogger. And the lights immediately went out...no, actually, they issued the following statement.

UK Power Networks would like to apologise to some customers in the
Holloway Road area of London who have experienced a series of power interruptions over recent weeks.

In the latest incident, power was interrupted to 327 customers at 5.54am today and restored to all customers affected by 11.20am.

The cause of the problem is believed to be an intermittent fault on an underground cable which our engineers are currently trying to trace. This can happen when the heat generated within the cable seals the damaged section, making it difficult to trace

Can it indeed. Let's hope they don't end up needing to do one of these.

hoisted from comments

From comments on this post, Against Viktorfeed:

*sigh* When is it going to get recognised that you can't spot an arms flight just by who used to own the aircraft. Jubba Airways isn't some unknown cargo entity - it's one of the main commercial passenger carriers into Mogadishu and other parts of Somalia. Are they a bit dodgy? Probably. Do they have dodgy planes? Yes. Does the fact that their dodgy planes may have been formerly owned by arms traffickers mean they're arms traffickers themselves? No.

To illustrate this whole *you can't spot an arms flight through the ownership of the plane* thing: Amnesty recently reported that arms were being flown from Bulgaria into Kigali in late 2008, just as Kigali was ramping up support for the CNDP in eastern DRC, on a *standard Air France passenger flight*. Which I'm pretty sure wouldn't have shown up on the 'Viktorfeed'.

www.amnesty.org/en/library/info/ACT30/015/2010/en

The key question is always who owns the cargo, and (for charter flight), who is *chartering* the plane. Not who owns or even operates the plane.

I've never been entirely confident on this point. Thoughts?

killing data.gov.uk, and thinking aloud about mapping the lobbysphere

So the government thinks this is clever. They also think it constitutes a "searchable online database". It is not searchable, nor is it a database. It is a collection of links to department web sites, some of which actually lead to useful documents, some of which lead to utterly pointless intermediary pages, some of which lead to documents in a sensible format, some of which lead to documents in pointlessly wrong formats, and some of which lead to PDF files. It provides no clue how often this data will be released or when or where. The URIs sometimes suggest that they might be predictable, sometimes they are just random alphanumeric sequences. Basically, what he said.

Meanwhile, very few of these documents have made it onto data.gov.uk, the government's data web site (pro-tip: the hint is in the name) which provides all that stuff out of the box. This is not just disappointing - this is actively regressive. Is it official policy to break data.gov.uk?

Anyway, I've been fiddling with NetworkX, the network-graph library for Python from Los Alamos National Laboratory. Sadly it doesn't have a method networkx.earth_shattering_kaboom(). I've eventually decided that the visualisation paradigm I wanted was looking me in the eye all along - kc claffy's Skitter graph, used by CAIDA to map the Internet's peering architecture.

The algorithm is fairly simple - nodes are located in terms of polar coordinates, on a circular chart. In the original, the concept is that you are observing from directly above the north or south pole. This gives you two dimensions - angle, or in other words, how far around the circle you are, and radius, your location on the line from the centre to the edge. claffy et al used the longitude of each Autonomous System's WHOIS technical contact address for their angles, and the inverse of each node's linkdegree for the radius. Linkdegree is a metric of how deeply connected any given object in the network is; taking the inverse (i.e 1/linkdegree) meant that the more of it you have, the more central you are.

My plan is to define the centre as the prime minister, and to plot the ministries at the distance from him given by the weighting I'd already given them - basically, the prime minister is 1 and the rest are progressively less starting with Treasury and working down - and an arbitrary angle. I'm going to sort them by weight, so that importance falls in a clockwise direction, for purely aesthetic reasons. Then, I'll plot the lobbies. As they are the unknown factors, they all start with the same, small node weighting. Then add the edges - the links - which will have weights given by the weight of the ministry involved divided by the number of outside participants at that meeting, so a one-on-one is the ideal case.

When we come to draw the graph, the lobbies will be plotted with the mean angle of the ministries they have meetings with, and the inverse of their linkdegree, with the node size scaled by its traffic. Traffic in this case basically means how many meetings it had. Therefore, it should be possible to see both how effective the lobbying was, from the node's position, and how much effort was expended, from its size. The edges will be coloured by date, so as to make change over time visible. If it works, I'll also provide some time series things - unfortunately, if the release frequency is quarterly, as it may be, this won't be very useful.

Anyway, as always, to-do no.1 is to finish the web scraping - the Internet's dishes. And think of a snappy name.

CSM: a useful tip-off

Following up on this post, I get e-mail from Matthew Turner, who points to an explanation. Communications & Strategy Management Ltd. does indeed have a web site of sorts, at csm-limited.com. It appears to be a political consulting/lobbying firm that does the Tories' demographics/voter-database work, which basically consists of the Richard Murphy mentioned in the past post. The Web site was registered in April, around the same time as the company, through an American proxy registrar. Although the company is registered in somebody's house in Stratford-upon-Avon, its operational address places it at:

The Manor
Coleshill Manor Campus
Birmingham Road
Coleshill
West Midlands
B46 1DL

Coleshill Manor is the headquarters of a variety of Tory organisations, notably the West Midlands Tory party itself and the commercial entity known variously as Constituency Campaigning Services Ltd. or Coleshill Campaigning Services Ltd, which essentially provided the services of the party's central institutions to individual Conservative Associations against a contribution, organising things like junk mail, phone calls, and other jewels of our society.

The purpose of this structure was to render CCS a commercial company in the meaning of the Political Parties, Elections, and Referendums Act - if you gave money to an individual MP's campaign, who then used it to pay for the services of a political consultancy, you would fall under the provisions of the Act and your donation would be both recorded and limited. If, however, you gave it to a commercial company which then sold its services to the campaign, not so much. Here's the Other Taxpayers' Alliance.

Various investigations have occurred into Coleshill - notably that the Tories use large amounts of office space there paid for by various millionaires via the companies based in the building. CCS alone received £1m in donations before the 2005 election. There's also a major call centre there.

Rounding it up, we have here part of a political campaign structure designed to accept external donations outside the legal framework, that is plugged into the Whips' Office at the far end. Neat!

I didn't think I'd still be blogging this six years later

Well, this looks pretty ugly. I have a question. We know that unofficial, non-doctrinal training material was being circulated around the joint services intelligence centre in Chicksands in 2003-2005 - there's an interesting quote about it in the Guardian piece here:

Any public inquiry into the activities of the JFIT would be expected to examine the extent to which it was supervised by military lawyers. It is now known that at least some of the training material used by F Branch at Chicksands between 2003 and 2005 escaped the scrutiny of the training centre's in-house lawyer, Brigadier David Yates, who told the Mousa inquiry that he did not "have the capacity" to check it.

This is important because, as the piece also points out, most of the interrogators were reservists. They would have gone first of all to the Chilwell mobilisation centre to do fitness tests, draw additional kit, get their vaccinations, complete their admin, and to do refresher courses on things like first aid, marksmanship, and anti-terrorist precautions. Then, later, they would have gone for a period of pre-deployment training, which would concentrate on preparing for their specific role in Iraq, before finally shipping out via South Cerney and RAF Brize Norton. It would make sense if the reservist intelligence people were sent to their trade's headquarters, which for most of them would also be their unit's peacetime depot, for their specialised pre-deployment course. (I think I have the process right, but several readers can correct me.)

Now, we also know that the Americans began with the torture in 2002, and that Major-General Geoffrey Miller was transferred from Guantanamo to Iraq with his infamous directive to "Gitmo-ize" the detention camps in the summer of 2003. So, where did this documentation come from?

Better procurement

Via this Wired piece, I see that the US Special Forces are giving up on the various "Land Warrior" projects to load down soldiers with specialised electronic gear - some readers may remember the BBC documentary on one of MOD's efforts in this line and the image of a file of soldiers attempting to move stealthily across Salisbury Plain with enormous objects like big plastic mushrooms lashed on top of their helmets. Instead, they've issued an RFP for software applications running on Android-based devices to achieve the same aims.

The RFP is here, and what an RFP it is too. It is, among other things, clear, and clearly drafted by someone with substantial technical competence. You try finding many contract managers who know what RFC 5740 specifies. From a technical point of view it's pretty demanding: multiple video streams, reliable delivery, in an environment of restricted connectivity - rather you than me. (This 2008 RFP may be part of the explanation.)

However, it's also true that a valid strategy for delivering high bandwidth traffic like video is to shift it from the classic unicast (i.e. one stream per user, from the same source) to a broadcast or multicast route. I wonder if whoever answers it will be an early user of the Stream Control Transmission Protocol, which among other things allows multi-homing at the connection level, so that the same connection between two logical addresses can involve more than one physical source?

This also reminds me of the mid-2000s IP Multimedia Subsystem hype - collaborative whiteboarding was an example use case that came up in literally every vendor presentation, and they would occasionally do demonstrations at conferences, which always turned out to be really awful. Sometimes this was because the server was back in Finland and the endpoints were roaming on a Singaporean operator's 3G network, with hilarious latency consequences. Sometimes it was because IMS just wasn't a very good idea.

not the Thursday music link

It's not Thursday, so it must be time for non-Thursday music link. You know the rules - only covers or remixes are acceptable. Here's DatA's mix of Danger's 19H11:

Sometimes, FaxYourMP Just Isn't Enough

I don't quite know what to make of this:

Q What other sites, remember any particular internet sites you looked at?

A When I was doing research about MPs, I looked at one called theyworkforyou.com and I think another one was called publicwhips [publicwhip.org.uk].

Q So, have you carried out any research to ... about Stephen Timms.

A Yeah, on ... I looked up, I found, I Googled him, I found out he had a website, I found a page about him on theyworkforyou.com ... if you follow that link it shows information about how he voted on different things related to the Iraq war and the build up towards it. I found out that ... he very strongly agreed with the invasion of Iraq and they said very strongly because they worked out all his votes for everything related to that and it came up to something like 99.9% support or something like that.

Q How does that make you feel?

A That made me feel angry because the whole Iraq war is just based on lies and he just voted strongly for everything as though he had no mercy. As though he felt no doubts that what he was doing was right, even though it was such an arrogant thing to do and I just felt like if he could treat the Iraqi people so mercilessly, then why should I show him any mercy?

Q What, what makes you think that it's your place to go and stab him?..

Exactly what is Communication Strategy & Management Ltd?

So I scraped the government meetings data and rescraped it as one-edge-per-row. And then, obviously enough, I tidied it up in a spreadsheet and threw it at ManyEyes as a proof-of-concept. Unfortunately, IBM's otherwise great web site is broken, so although it will preview the network diagram, it fails to actually publish it to the web. Oh well, ticket opened, etc.

Anyway, I was able to demonstrate the thing to Daniel Davies on my laptop, on the bar of the Nelson's Retreat pub in Old Street. This impressed him excessively. Specifically, we were interested by an odd outlier on the chart. Before I get into that, though, here are some preliminary findings.

1 - Clegg's Diary

At first sight, Nick Clegg appears to be unexpectedly influential. His calender included meetings with NATO, the World Bank, the Metropolitan Police, the Gates Foundation, and oddly enough, Lord Robertson of Port Ellen. Not only that, he had one-to-one meetings with all of them. However, he also got The Elders (i.e. retired politicos playing at shop) and the leader of the Canadian opposition, one Michael Ignatieff, Esq. God help us, is Clegg turning out to be a Decent?

2 - Dave from PR's surprisingly dull world

The Prime Minister, no less, meets with some remarkably dull people. In fact, he met quite a lot of people who you'd expect to be left to flunkies while leaving quite a lot of important people to Nick Clegg. He did get BP, Shell, Pfizer, Rupert Murdoch, the TUC general secretary, and Ratan Tata (twice!) as one-on-ones, but he also met a surprising number of minor worthies from Cornwall and vacuous photocalls with people from Facebook.

3 - Francis Maude, evil genius of the coalition

Secretary of State for the Cabinet Office and Paymaster-General, Francis Maude MP, is the surprise hit, as far as I can make out. He seems to have a special responsibility for anything that smacks of privatisation - therefore, the monetary value of meeting him is probably high. Of course, if your evil genius is Francis Mediocritus, you've got problems. No wonder we're in such a mess. All these points are also true of Oliver Letwin.

4 - Communication and Strategy Management Ltd

This is our far outlier. Some of the least significant people on the chart appear to be government whips, which is obviously an artefact of the data set. The data release does not cover intra-governmental or parliamentary meetings, nor does it cover diplomatic activity. Whips, of course, are a key institution in the political system. Given their special role with regard to both the government and parliament, it's not surprising that they appear to be sheltered from external lobbying - access to the Whips' Office would be such a powerful and occult influence that it must be held closely.

So what on earth is Communication and Strategy Management Ltd., a company which had one-on-one access to the Government Chief Whip, the Rt. Hon. Patrick McLoughlin MP, and which according to Companies House was founded on the 11th of April? It has no web site or perceptible public presence. It is located in what looks like a private house, here, not far from Stratford upon Avon:


View Larger Map

Evidently the hub of political influence, but those are the facts. The directors are Elizabeth Ann Murphy and Richard Anthony Cubitt Murphy*, ignoring a company-formation agent who was a director for one day when setting up the company. It's not as if C&SM Ltd is a constituent of McLoughlin's - he's MP for the Derbyshire Dales. Actually, either the directors are related or else there was a cockup, as Murphy's name on the books was amended from Bromley the day after the company was formed and both were born in 1963. The Companies House filing* doesn't give any other information - accounts aren't due for a while - except that the one share issued is held by Norman Younger, who is a partner in the company formation service that was used.

Anyway, the next stop is to learn how this works and put up a nice little dashboard page to help watch the lobbysphere. I'd be happier doing something with python - such as nodebox - but the diagram is already too big to be useful without interactivity, and you can't stick a NodeBox window in a web page.

*Not the Richard Murphy, who is too young.
*WebCheck - it's not an ugly website, it's a way of life...

the House of Lords is not just stranger than you think..

This has me thinking one thing - TheyWorkForYou needs to integrate the text-mining tool researchers used to estimate the point at which Agatha Christie's Alzheimer's disease set in by analysing her books. We could call it WhatHaveTheyForgotten? Or perhaps HowDrunkIsYourMP? Jakob Whitfield pointed me to the original paper, here. It doesn't seem that complicated, although I have a couple of methodological questions - for a start, are there enough politicians with a track record in Hansard long enough to provide a good baseline for time-series analysis?

Instead, we could do a synchronic comparison and look at which politicians seem to be diverging from the average. Of course, some might object that this would be a comparison against a highly unusual and self-selected sample. Another objection might be that the whole idea is simply too cruel. Yet a further objection might be the classic one that there are some things man should not know.

Update: Implemented!

so you want to know who's lobbying?

So I was moaning about the Government and the release of lists of meetings with external organisations. Well, what about some action? I've written a scraper that aggregates all the existing data and sticks it in a sinister database. At the moment, the Cabinet Office, DEFRA, and the Scottish Office have coughed up the files and are all included. I'm going to add more departments as they become available. Scraperwiki seems to be a bit sporky this evening; the whole thing has run to completion, although for some reason you can't see all the data, and I've added the link to the UK Open Government Licence twice without it being saved.

A couple of technical points: to start with, I'd like to thank this guy who wrote an alternative to Python's csv module's wonderful DictReader class. DictReader is lovely because it lets you open a CSV (or indeed anything-separated value) file and keep the rows of data linked to their column headers as python dictionaries. Unfortunately, it won't handle Unicode or anything except UTF-8. Which is a problem if you're Chinese, or as it happens, if you want to read documents produced by Windows users, as they tend to use Really Strange characters for trivial things like apostrophes (\x92, can you believe it?). This, however, will process whatever encoding you give it and will still give you dictionaries. Thanks!

I also discovered something fun about ScraperWiki itself. It's surprisingly clever under the bonnet - I was aware of various smart things with User Mode Linux and heavy parallelisation going on, and I recall Julian Todd talking about his plans to design a new scaling architecture based on lots of SQLite databases in RAM as read-slaves. Anyway, I had kept some URIs in a list, which I was then planning to loop through, retrieving the data and processing it. One of the URIs, DEFRA's, ended like so: oct2010.csv.

Obviously, I liked the idea of generating the filename programmatically, in the expectation of future releases of data. For some reason, though, the parsing kept failing as soon as it got to the DEFRA page. Weirdly, what was happening was that the parser would run into a chunk of HTML and, obviously enough, choke. But there was no HTML. Bizarre. Eventually I thought to look in the Scraperwiki debugger's Sources tab. To my considerable surprise, all the URIs were being loaded at once, in parallel, before the processing of the first file began. This was entirely different from the flow of control in my program, and as a result, the filename was not generated before the HTTP request was issued. DEFRA was 404ing, and because the csv module takes a file object rather than a string, I was using urllib.urlretrieve() rather than urlopen() or scraperwiki.scrape(). Hence the HTML.

So, Scraperwiki does a silent optimisation and loads all your data sources in parallel on startup. Quite cool, but I have to say that some documentation of this feature might be nice, as multithreading is usually meant to be voluntary:-)

TODO, meanwhile: at the moment, all the organisations that take part in a given meeting are lumped together. I want to break them out, to facilitate counting the heaviest lobbyists and feeding visualisation tools. Also, I'd like to clean up the "Purpose of meeting" field so as to be able to do the same for subject matter.

Update: Slight return. Fixed the unique keying requirement by creating a unique meeting id.

Update Update: Would anyone prefer if the data output schema was link-oriented rather than event-oriented? At the moment it preserves the underlying structure of the data releases, which have one row for each meeting. It might be better, when I come to expand the Name of External Org field, to have a row per relationship, i.e. edge in the network. This would help a lot with visualisation. In that case, I'd create a non-unique meeting identifier to make it possible to recreate the meetings by grouping on that key, and instead have a unique constraint on an identifier for each link.

Update Update Update: So I made one.

loose ends tied up

Remember this post from back in September, 2005? (And what a fine month that was.) A Lebanese court has sentenced several people involved to terms of imprisonment, including highly dubious aviation identity Imad Saba and two of his managers. The aircraft captain, who survived, and Saba's representative, who also survived after haranguing the other pilot (who didn't) into taking off with the aircraft several tonnes overloaded and out of balance, were also convicted in absentia and remain on the run.