review of a movie that doesn't exist yet

I think most of my readers also read Patrick Lang's blog, but I think this guest post is the best thing yet written on the Taliban/SIS/McChrystal/Petraeus fake sheikh affair. Really, there's a great movie to be made here - the multiplicity of motives, the ironic contrast between the absurd story and the deadly serious interests and emotions that drive it forward, the eternal ambiguity of the relationship between the manipulator and the manipulated.

The ISI comes out of it as being dastardly clever, but in a deeply futile way. They succeed in preventing a dangerous outbreak of peace and sanity, but what have they gained? The wars grind on, the butcher's bill ticks up, the fantasy of a Pakistani empire of trucks and pipes across the Hindu Kush is as far away as ever, the Indians continue with their industrialisation across the other border.

The Americans come out of it as being well-meaning but naive. After all, they only get into this story because they want peace. So does the real Taliban leader. They both share a sort of big, stupid nobility.

The British do almost as badly as the ISI; not only do they end up being the dupes of the piece, they do so without the saving grace of having good intentions. They're as naive as the Americans but more underhanded. SIS gets involved purely as a way of sucking up to the Americans and putting one over its real enemies, GCHQ, Her Majesty's Forces, MI5, and the main-line Foreign Office diplomats. The Government is desperately keen on the project for similarly base reasons - to suck up to the Americans, to grab at an opportunity to solve its problem in Afghanistan, and of course to embarrass the Labour Party. Of course, it would have been a brilliant political fix had it come off - but the master manipulator is not Bismarck but William Hague.

The fake sheikh, meanwhile, is a classic example of the Pinocchio/Hauptmann von Kopenick theme - the puppet of bigger forces who becomes a power in his own right. Without his successful performance, of course, none of the many expectations curling around the tale have a hope of happening. His agency is real, and his character expands to fill the role. The fact that the whole project is an exercise in theatre is interesting in itself - a film within the film. The actors in the film are, of course, puppets of the script and the direction, and it is a work of fiction. The enduring purpose of the theatre and the cinema, however, is that works of fiction have real influence on their audiences. Like the fake sheikh.

After all, the grocer of Quetta (not a bad title) is the only character in the drama who successfully pursues his interests. He gets some interesting time off away from his bazaar stall, and even gets rich. You could play this as the ordinary man who succeeds in making fools of the powerful who insist on involving him in their schemes, or perhaps as a microcosm of all the people who are getting rich off the continued war, Mother Courage rather than Kopenick. Alternatively he could be killed off, casting the whole thing as an utterly bleak tragedy. However, arguably the classic in this vein is The Third Man and that sticks with the tragicomic.

the intersection of Wikileaks and Viktorfeed

If Bout agreed to cooperate with the Justice Department in a plea agreement sparing him life in prison, “we’ll certainly want to know more from him about the circumstances of those Iraq contracts,” said a law-enforcement official with detailed knowledge of the criminal case against Bout. The official was referring to large U.S. military contracts in 2003 and 2004 in which Bout’s cargo companies were used as subcontractors to deliver military supplies to U.S. forces in Iraq. “We’d want to understand if our officials knew exactly who they were dealing with. If U.S. officials knew they were dealing with Bout, that’s uncomfortable news.” Could U.S. officials face prosecution? “The conspiracy laws are broad,” the official said without elaboration.

Well, that would be fun. I'm also interested by the suggestion of a plea bargain. From here. This is something I don't think Irina from RFE/RL quite grasped.

There are some interesting Viktor Bout-related cables in the Wikileaks dump - this one suggests that part of his defence against extradition was to suggest he was in Thailand as part of a "government to government submarine deal". This one details a visit by US diplomats to Ras al-Khaimah airport, where they viewed some rotting Il-76s and made enquiries about what may have been Viktor's maintenance base. This was in any case around about the same time that the UAE gave the Antonov 12 operators the boot.

I'm surprised there aren't more, although there may be more releases yet. The relevant tags would be AE for the UAE and perhaps some others - NEA for the Near East desk is one.

Update: Moar!

False dichotomy watch

We've seen plenty of this sort of stuff before:

These networks such as the UnCut movement or the student movement with outstanding micro-organisations such as the UCL Occupation (which has received over 60,000 hits on its blog in a little under a week) who have so dynamically organised yesterday, today and going forward will inevitably be more flexible and effective than organisations with generic ‘leaderships’ such as major businesses, the police or even the National Union of Students.

Well, if your benchmark of effectiveness is the NUS... Mere snark, though. This particular Internet prairie fire does seem to be spreading nicely and doing more than tweeting. However, whenever someone starts going on like this, I do tend to suspect what they mean is "...more flexible and effective than organisations with people with funny accents who are train drivers an stuff".

I've said before, though, that I suspect that a lot of this network organising is structurally suited to negativity. Look at the 'baggers, for example. The classic examples political science types use, like open-source software projects, tend to be very different to the implementations in politics - rather than trying to recruit masses of people, they're usually driven by a hard core of the obsessed, or of people whose job it is. Order is difficult, mayhem is easy. Specifically, you can contribute significantly to mayhem by putting in an hour here or there.

On the other hand, though, it's not as if we're likely to run out of rage. This is 'bagger lesson one. Negative tactics and the expression of inchoate rage are not without value. Nigel Stanley gets it right - it's a false dichotomy. Getrennt marschieren, vereint schlagen, and we'll all get there in the end.

the lack of liberals

So I went to protest the London Lib Dems' conference, held in the late Blairite magnificence of Haverstock School NW3. Arriving punctually, what did I find?



No Liberals. In fact, not only had they vanished from the Haverstock, it turned out they'd given up on the whole idea of having a conference and punted it to February. Now you know why the party's colour is yellow. Not only were they afraid of the general public, but their reputation is now so toxic that nobody wants to give them house room.

Meanwhile, here's a prominent London Lib Dem in action.

"Right at this moment of financial peril to the nation is perhaps not the moment to introduce mandatory pay audits."

Just two years ago, the Liberal Democrat MP backed mandatory measures, saying: "A voluntary audit system for private industry is hardly worth the paper it's printed on. We need to know when the government actually plans to step in if progress isn't made."

The Liberal Democrat manifesto pledged to introduce fair-pay audits for all but the smallest companies.

Today Featherstone said: "It was a different world two years ago – financially and in terms of pressures on business. We are in a completely new landscape now … Much more of partnership working, no longer government dictates, this is absolutely the time to make voluntary pay-reporting work."

Two years ago? Two years ago was December, 2008 - hardly a moment of expansive prosperity. Banks were falling like seagull shit. People were trying to estimate what the absolute minimum level of cash balances was that could prevent the bankruptcy of the entire GM and Ford supply chains. Flocks of great empty ships were gathering in Falmouth harbour and off Singapore, forever delayed by the drying up of trade credit.

Also, check out that last sentence, a real classic of pseudo-Blair verbiage without enough verbs.

public service announcement

Sometimes, somebody does a web site listing protests against the cuts so you don't have to.

fail

Odd thought, working on NetworkX visualisations of lobbying activity. So much of whether the finished image is any use or not is an aesthetic decision, nothing to do with the actual graph metrics in any formal or programmatic way.

Randomly fiddle with the scaling factor, or increase the multiplier you apply to the nodes' link degree, and things look different - and possibly better. Also, more information may not be better - I can't help but think it was a lot easier to derive conclusions before I gave the various ministries node weights to reflect their relative power and divided the ministry weighting on each link by the number of organisations taking part in the meeting, which should be an improvement.

Also, after sweating quite a bit over this, does anyone know if NetworkX even lets you position nodes using polar coordinates? This looks useful.



This is far from ideal, but it should give some idea. This one is using the standard force-directed spring layout and just the data I have from June, with the nodes coloured by their degree in the chart and sized by their meeting count. Downing Street (the large purple plook) is fixed in place and the others adjusted. I rather like the fact that the lobbyists look like bacteria swarming into a cell, or possibly sperm trying to fertilise David Cameron.

An interview with Radio Free Europe

So Viktor Bout got extradited at last. As a result, I was interviewed by Radio Free Europe's Irina Lagunina. They wanted to hear about how the whole Viktor-blogging project got started - I told them about the Defence Energy Support Center files and the T-DODAACs and the like, and the plane spotter websites, and the fact I basically borrowed the idea from the people who were trying to monitor CIA rendition flights.

They wanted to hear about bloggers, and I thanked everyone who took part in Operation Firedump and made the point that all sorts of people across the ideological spectrum had taken part. They asked about my feelings about the extradition, and I said that I'd done the rejoicing when he was arrested and had since been mostly interested in seeing if there was any noticeable change in air movements through the UAE. What really cheered me was when the UAE government kicked out the An-12 operators.

They also wanted to know about the Russian government using VB's companies to move humanitarian aid. Specifically, they referred to something in Doug Farah's book that apparently quotes me on this. Unfortunately, my review copy never turned up so I'm completely in the dark as to what this might be. I said that given the numbers of heavy, ex-Soviet tactical airlifters that he controlled at the peak of his career, it was unlikely that many humanitarian agencies had been able to avoid occasionally dealing with him, and that they were probably right on balance to do the job rather than worry too much. If you need to move 40 tons of drinking water or flour labelled "Gift of the European Union" or whatever before people starve or get cholera, I'm not going to whine about it.

Recombination

In the last month or so I've begun adding blogs to my RSS queue again. I'm surprised by this; I didn't expect to find that the form still had so much energy. Perhaps this isn't so optimistic after all. Meanwhile, watch this space.s

le bateau ivre, port of call: Mogadishu

I wrote about this piece about brewing beer in southern Sudan, and incidentally creating a power station and a water works and a tiny industrial working class.

Well, SAB-Miller did a scenario-planning exercise about this sort of thing and came up with some truly odd answers. In the worst-case scenario:

a market with limited access to water and high energy costs, where people would migrate from areas of water shortage or turbulent weather

their proposed solution involved installing a brewery aboard a ship, so it could sail to wherever the thirsty masses had pitched up. Interestingly, as the story was released through the Grauniad's "Guardian Professional" advertorial division, this is also something SABMiller management was willing to pay good money to tell us about. Time was when big business wanted to keep its apocalyptic fantasies secret.

(Tagged "uncategorized", but should probably be under "uncategorizable".)

local news: power cuts

We keep having power cuts. In eight days, we had three, all of them between 5.30am and 6am, which all lasted most of the morning. I know exactly when they happen, because the smoke alarms start beeping and wake me up. My partner claims she was warmer on a demo than she was in the flat during no.2. As they seem to follow a pattern of happening at the same time every other day - just about when power demand starts to turn back up in earnest - I was wide awake at 0545 today waiting for the plaintive beeps. But no - looking at the chart, the ramp-up is later on a Sunday. Mind you, later in the day the lights flickered repeatedly for half a minute.

So I rang up UK Power Networks' (what used to be EDF Energy Networks, what used to be the London Electricity Board) press office and announced myself as a blogger. And the lights immediately went out...no, actually, they issued the following statement.

UK Power Networks would like to apologise to some customers in the
Holloway Road area of London who have experienced a series of power interruptions over recent weeks.

In the latest incident, power was interrupted to 327 customers at 5.54am today and restored to all customers affected by 11.20am.

The cause of the problem is believed to be an intermittent fault on an underground cable which our engineers are currently trying to trace. This can happen when the heat generated within the cable seals the damaged section, making it difficult to trace

Can it indeed. Let's hope they don't end up needing to do one of these.

hoisted from comments

From comments on this post, Against Viktorfeed:

*sigh* When is it going to get recognised that you can't spot an arms flight just by who used to own the aircraft. Jubba Airways isn't some unknown cargo entity - it's one of the main commercial passenger carriers into Mogadishu and other parts of Somalia. Are they a bit dodgy? Probably. Do they have dodgy planes? Yes. Does the fact that their dodgy planes may have been formerly owned by arms traffickers mean they're arms traffickers themselves? No.

To illustrate this whole *you can't spot an arms flight through the ownership of the plane* thing: Amnesty recently reported that arms were being flown from Bulgaria into Kigali in late 2008, just as Kigali was ramping up support for the CNDP in eastern DRC, on a *standard Air France passenger flight*. Which I'm pretty sure wouldn't have shown up on the 'Viktorfeed'.

www.amnesty.org/en/library/info/ACT30/015/2010/en

The key question is always who owns the cargo, and (for charter flight), who is *chartering* the plane. Not who owns or even operates the plane.

I've never been entirely confident on this point. Thoughts?

killing data.gov.uk, and thinking aloud about mapping the lobbysphere

So the government thinks this is clever. They also think it constitutes a "searchable online database". It is not searchable, nor is it a database. It is a collection of links to department web sites, some of which actually lead to useful documents, some of which lead to utterly pointless intermediary pages, some of which lead to documents in a sensible format, some of which lead to documents in pointlessly wrong formats, and some of which lead to PDF files. It provides no clue how often this data will be released or when or where. The URIs sometimes suggest that they might be predictable, sometimes they are just random alphanumeric sequences. Basically, what he said.

Meanwhile, very few of these documents have made it onto data.gov.uk, the government's data web site (pro-tip: the hint is in the name) which provides all that stuff out of the box. This is not just disappointing - this is actively regressive. Is it official policy to break data.gov.uk?

Anyway, I've been fiddling with NetworkX, the network-graph library for Python from Los Alamos National Laboratory. Sadly it doesn't have a method networkx.earth_shattering_kaboom(). I've eventually decided that the visualisation paradigm I wanted was looking me in the eye all along - kc claffy's Skitter graph, used by CAIDA to map the Internet's peering architecture.

The algorithm is fairly simple - nodes are located in terms of polar coordinates, on a circular chart. In the original, the concept is that you are observing from directly above the north or south pole. This gives you two dimensions - angle, or in other words, how far around the circle you are, and radius, your location on the line from the centre to the edge. claffy et al used the longitude of each Autonomous System's WHOIS technical contact address for their angles, and the inverse of each node's linkdegree for the radius. Linkdegree is a metric of how deeply connected any given object in the network is; taking the inverse (i.e 1/linkdegree) meant that the more of it you have, the more central you are.

My plan is to define the centre as the prime minister, and to plot the ministries at the distance from him given by the weighting I'd already given them - basically, the prime minister is 1 and the rest are progressively less starting with Treasury and working down - and an arbitrary angle. I'm going to sort them by weight, so that importance falls in a clockwise direction, for purely aesthetic reasons. Then, I'll plot the lobbies. As they are the unknown factors, they all start with the same, small node weighting. Then add the edges - the links - which will have weights given by the weight of the ministry involved divided by the number of outside participants at that meeting, so a one-on-one is the ideal case.

When we come to draw the graph, the lobbies will be plotted with the mean angle of the ministries they have meetings with, and the inverse of their linkdegree, with the node size scaled by its traffic. Traffic in this case basically means how many meetings it had. Therefore, it should be possible to see both how effective the lobbying was, from the node's position, and how much effort was expended, from its size. The edges will be coloured by date, so as to make change over time visible. If it works, I'll also provide some time series things - unfortunately, if the release frequency is quarterly, as it may be, this won't be very useful.

Anyway, as always, to-do no.1 is to finish the web scraping - the Internet's dishes. And think of a snappy name.

CSM: a useful tip-off

Following up on this post, I get e-mail from Matthew Turner, who points to an explanation. Communications & Strategy Management Ltd. does indeed have a web site of sorts, at csm-limited.com. It appears to be a political consulting/lobbying firm that does the Tories' demographics/voter-database work, which basically consists of the Richard Murphy mentioned in the past post. The Web site was registered in April, around the same time as the company, through an American proxy registrar. Although the company is registered in somebody's house in Stratford-upon-Avon, its operational address places it at:

The Manor
Coleshill Manor Campus
Birmingham Road
Coleshill
West Midlands
B46 1DL

Coleshill Manor is the headquarters of a variety of Tory organisations, notably the West Midlands Tory party itself and the commercial entity known variously as Constituency Campaigning Services Ltd. or Coleshill Campaigning Services Ltd, which essentially provided the services of the party's central institutions to individual Conservative Associations against a contribution, organising things like junk mail, phone calls, and other jewels of our society.

The purpose of this structure was to render CCS a commercial company in the meaning of the Political Parties, Elections, and Referendums Act - if you gave money to an individual MP's campaign, who then used it to pay for the services of a political consultancy, you would fall under the provisions of the Act and your donation would be both recorded and limited. If, however, you gave it to a commercial company which then sold its services to the campaign, not so much. Here's the Other Taxpayers' Alliance.

Various investigations have occurred into Coleshill - notably that the Tories use large amounts of office space there paid for by various millionaires via the companies based in the building. CCS alone received £1m in donations before the 2005 election. There's also a major call centre there.

Rounding it up, we have here part of a political campaign structure designed to accept external donations outside the legal framework, that is plugged into the Whips' Office at the far end. Neat!

I didn't think I'd still be blogging this six years later

Well, this looks pretty ugly. I have a question. We know that unofficial, non-doctrinal training material was being circulated around the joint services intelligence centre in Chicksands in 2003-2005 - there's an interesting quote about it in the Guardian piece here:

Any public inquiry into the activities of the JFIT would be expected to examine the extent to which it was supervised by military lawyers. It is now known that at least some of the training material used by F Branch at Chicksands between 2003 and 2005 escaped the scrutiny of the training centre's in-house lawyer, Brigadier David Yates, who told the Mousa inquiry that he did not "have the capacity" to check it.

This is important because, as the piece also points out, most of the interrogators were reservists. They would have gone first of all to the Chilwell mobilisation centre to do fitness tests, draw additional kit, get their vaccinations, complete their admin, and to do refresher courses on things like first aid, marksmanship, and anti-terrorist precautions. Then, later, they would have gone for a period of pre-deployment training, which would concentrate on preparing for their specific role in Iraq, before finally shipping out via South Cerney and RAF Brize Norton. It would make sense if the reservist intelligence people were sent to their trade's headquarters, which for most of them would also be their unit's peacetime depot, for their specialised pre-deployment course. (I think I have the process right, but several readers can correct me.)

Now, we also know that the Americans began with the torture in 2002, and that Major-General Geoffrey Miller was transferred from Guantanamo to Iraq with his infamous directive to "Gitmo-ize" the detention camps in the summer of 2003. So, where did this documentation come from?

Better procurement

Via this Wired piece, I see that the US Special Forces are giving up on the various "Land Warrior" projects to load down soldiers with specialised electronic gear - some readers may remember the BBC documentary on one of MOD's efforts in this line and the image of a file of soldiers attempting to move stealthily across Salisbury Plain with enormous objects like big plastic mushrooms lashed on top of their helmets. Instead, they've issued an RFP for software applications running on Android-based devices to achieve the same aims.

The RFP is here, and what an RFP it is too. It is, among other things, clear, and clearly drafted by someone with substantial technical competence. You try finding many contract managers who know what RFC 5740 specifies. From a technical point of view it's pretty demanding: multiple video streams, reliable delivery, in an environment of restricted connectivity - rather you than me. (This 2008 RFP may be part of the explanation.)

However, it's also true that a valid strategy for delivering high bandwidth traffic like video is to shift it from the classic unicast (i.e. one stream per user, from the same source) to a broadcast or multicast route. I wonder if whoever answers it will be an early user of the Stream Control Transmission Protocol, which among other things allows multi-homing at the connection level, so that the same connection between two logical addresses can involve more than one physical source?

This also reminds me of the mid-2000s IP Multimedia Subsystem hype - collaborative whiteboarding was an example use case that came up in literally every vendor presentation, and they would occasionally do demonstrations at conferences, which always turned out to be really awful. Sometimes this was because the server was back in Finland and the endpoints were roaming on a Singaporean operator's 3G network, with hilarious latency consequences. Sometimes it was because IMS just wasn't a very good idea.

not the Thursday music link

It's not Thursday, so it must be time for non-Thursday music link. You know the rules - only covers or remixes are acceptable. Here's DatA's mix of Danger's 19H11:

Sometimes, FaxYourMP Just Isn't Enough

I don't quite know what to make of this:

Q What other sites, remember any particular internet sites you looked at?

A When I was doing research about MPs, I looked at one called theyworkforyou.com and I think another one was called publicwhips [publicwhip.org.uk].

Q So, have you carried out any research to ... about Stephen Timms.

A Yeah, on ... I looked up, I found, I Googled him, I found out he had a website, I found a page about him on theyworkforyou.com ... if you follow that link it shows information about how he voted on different things related to the Iraq war and the build up towards it. I found out that ... he very strongly agreed with the invasion of Iraq and they said very strongly because they worked out all his votes for everything related to that and it came up to something like 99.9% support or something like that.

Q How does that make you feel?

A That made me feel angry because the whole Iraq war is just based on lies and he just voted strongly for everything as though he had no mercy. As though he felt no doubts that what he was doing was right, even though it was such an arrogant thing to do and I just felt like if he could treat the Iraqi people so mercilessly, then why should I show him any mercy?

Q What, what makes you think that it's your place to go and stab him?..

Exactly what is Communication Strategy & Management Ltd?

So I scraped the government meetings data and rescraped it as one-edge-per-row. And then, obviously enough, I tidied it up in a spreadsheet and threw it at ManyEyes as a proof-of-concept. Unfortunately, IBM's otherwise great web site is broken, so although it will preview the network diagram, it fails to actually publish it to the web. Oh well, ticket opened, etc.

Anyway, I was able to demonstrate the thing to Daniel Davies on my laptop, on the bar of the Nelson's Retreat pub in Old Street. This impressed him excessively. Specifically, we were interested by an odd outlier on the chart. Before I get into that, though, here are some preliminary findings.

1 - Clegg's Diary

At first sight, Nick Clegg appears to be unexpectedly influential. His calender included meetings with NATO, the World Bank, the Metropolitan Police, the Gates Foundation, and oddly enough, Lord Robertson of Port Ellen. Not only that, he had one-to-one meetings with all of them. However, he also got The Elders (i.e. retired politicos playing at shop) and the leader of the Canadian opposition, one Michael Ignatieff, Esq. God help us, is Clegg turning out to be a Decent?

2 - Dave from PR's surprisingly dull world

The Prime Minister, no less, meets with some remarkably dull people. In fact, he met quite a lot of people who you'd expect to be left to flunkies while leaving quite a lot of important people to Nick Clegg. He did get BP, Shell, Pfizer, Rupert Murdoch, the TUC general secretary, and Ratan Tata (twice!) as one-on-ones, but he also met a surprising number of minor worthies from Cornwall and vacuous photocalls with people from Facebook.

3 - Francis Maude, evil genius of the coalition

Secretary of State for the Cabinet Office and Paymaster-General, Francis Maude MP, is the surprise hit, as far as I can make out. He seems to have a special responsibility for anything that smacks of privatisation - therefore, the monetary value of meeting him is probably high. Of course, if your evil genius is Francis Mediocritus, you've got problems. No wonder we're in such a mess. All these points are also true of Oliver Letwin.

4 - Communication and Strategy Management Ltd

This is our far outlier. Some of the least significant people on the chart appear to be government whips, which is obviously an artefact of the data set. The data release does not cover intra-governmental or parliamentary meetings, nor does it cover diplomatic activity. Whips, of course, are a key institution in the political system. Given their special role with regard to both the government and parliament, it's not surprising that they appear to be sheltered from external lobbying - access to the Whips' Office would be such a powerful and occult influence that it must be held closely.

So what on earth is Communication and Strategy Management Ltd., a company which had one-on-one access to the Government Chief Whip, the Rt. Hon. Patrick McLoughlin MP, and which according to Companies House was founded on the 11th of April? It has no web site or perceptible public presence. It is located in what looks like a private house, here, not far from Stratford upon Avon:


View Larger Map

Evidently the hub of political influence, but those are the facts. The directors are Elizabeth Ann Murphy and Richard Anthony Cubitt Murphy*, ignoring a company-formation agent who was a director for one day when setting up the company. It's not as if C&SM Ltd is a constituent of McLoughlin's - he's MP for the Derbyshire Dales. Actually, either the directors are related or else there was a cockup, as Murphy's name on the books was amended from Bromley the day after the company was formed and both were born in 1963. The Companies House filing* doesn't give any other information - accounts aren't due for a while - except that the one share issued is held by Norman Younger, who is a partner in the company formation service that was used.

Anyway, the next stop is to learn how this works and put up a nice little dashboard page to help watch the lobbysphere. I'd be happier doing something with python - such as nodebox - but the diagram is already too big to be useful without interactivity, and you can't stick a NodeBox window in a web page.

*Not the Richard Murphy, who is too young.
*WebCheck - it's not an ugly website, it's a way of life...

the House of Lords is not just stranger than you think..

This has me thinking one thing - TheyWorkForYou needs to integrate the text-mining tool researchers used to estimate the point at which Agatha Christie's Alzheimer's disease set in by analysing her books. We could call it WhatHaveTheyForgotten? Or perhaps HowDrunkIsYourMP? Jakob Whitfield pointed me to the original paper, here. It doesn't seem that complicated, although I have a couple of methodological questions - for a start, are there enough politicians with a track record in Hansard long enough to provide a good baseline for time-series analysis?

Instead, we could do a synchronic comparison and look at which politicians seem to be diverging from the average. Of course, some might object that this would be a comparison against a highly unusual and self-selected sample. Another objection might be that the whole idea is simply too cruel. Yet a further objection might be the classic one that there are some things man should not know.

Update: Implemented!

so you want to know who's lobbying?

So I was moaning about the Government and the release of lists of meetings with external organisations. Well, what about some action? I've written a scraper that aggregates all the existing data and sticks it in a sinister database. At the moment, the Cabinet Office, DEFRA, and the Scottish Office have coughed up the files and are all included. I'm going to add more departments as they become available. Scraperwiki seems to be a bit sporky this evening; the whole thing has run to completion, although for some reason you can't see all the data, and I've added the link to the UK Open Government Licence twice without it being saved.

A couple of technical points: to start with, I'd like to thank this guy who wrote an alternative to Python's csv module's wonderful DictReader class. DictReader is lovely because it lets you open a CSV (or indeed anything-separated value) file and keep the rows of data linked to their column headers as python dictionaries. Unfortunately, it won't handle Unicode or anything except UTF-8. Which is a problem if you're Chinese, or as it happens, if you want to read documents produced by Windows users, as they tend to use Really Strange characters for trivial things like apostrophes (\x92, can you believe it?). This, however, will process whatever encoding you give it and will still give you dictionaries. Thanks!

I also discovered something fun about ScraperWiki itself. It's surprisingly clever under the bonnet - I was aware of various smart things with User Mode Linux and heavy parallelisation going on, and I recall Julian Todd talking about his plans to design a new scaling architecture based on lots of SQLite databases in RAM as read-slaves. Anyway, I had kept some URIs in a list, which I was then planning to loop through, retrieving the data and processing it. One of the URIs, DEFRA's, ended like so: oct2010.csv.

Obviously, I liked the idea of generating the filename programmatically, in the expectation of future releases of data. For some reason, though, the parsing kept failing as soon as it got to the DEFRA page. Weirdly, what was happening was that the parser would run into a chunk of HTML and, obviously enough, choke. But there was no HTML. Bizarre. Eventually I thought to look in the Scraperwiki debugger's Sources tab. To my considerable surprise, all the URIs were being loaded at once, in parallel, before the processing of the first file began. This was entirely different from the flow of control in my program, and as a result, the filename was not generated before the HTTP request was issued. DEFRA was 404ing, and because the csv module takes a file object rather than a string, I was using urllib.urlretrieve() rather than urlopen() or scraperwiki.scrape(). Hence the HTML.

So, Scraperwiki does a silent optimisation and loads all your data sources in parallel on startup. Quite cool, but I have to say that some documentation of this feature might be nice, as multithreading is usually meant to be voluntary:-)

TODO, meanwhile: at the moment, all the organisations that take part in a given meeting are lumped together. I want to break them out, to facilitate counting the heaviest lobbyists and feeding visualisation tools. Also, I'd like to clean up the "Purpose of meeting" field so as to be able to do the same for subject matter.

Update: Slight return. Fixed the unique keying requirement by creating a unique meeting id.

Update Update: Would anyone prefer if the data output schema was link-oriented rather than event-oriented? At the moment it preserves the underlying structure of the data releases, which have one row for each meeting. It might be better, when I come to expand the Name of External Org field, to have a row per relationship, i.e. edge in the network. This would help a lot with visualisation. In that case, I'd create a non-unique meeting identifier to make it possible to recreate the meetings by grouping on that key, and instead have a unique constraint on an identifier for each link.

Update Update Update: So I made one.

loose ends tied up

Remember this post from back in September, 2005? (And what a fine month that was.) A Lebanese court has sentenced several people involved to terms of imprisonment, including highly dubious aviation identity Imad Saba and two of his managers. The aircraft captain, who survived, and Saba's representative, who also survived after haranguing the other pilot (who didn't) into taking off with the aircraft several tonnes overloaded and out of balance, were also convicted in absentia and remain on the run.

Shibboleth

Somewhat in the spirit of this XKCD cartoon. There are memes that allow us to tell if other people are likely to be worth speaking to - like biomarkers for language. For example, someone who disbelieves in plate tectonics probably has a wide range of other weird beliefs.

The latest one of these I've noticed is the idea that you have to be unemployed to get housing benefit in the UK. Knobber after horrible knobber shows up talking about claimants "in houses working people couldn't dream of". Wrong, wrong, wrong. So unfortunately, I've come to the parting of the ways with Tim Garton Ash. In an otherwise mostly sensible column:

It's surely not right that people can be worse off if they choose to work than they would be on welfare; or that people on inflated housing benefits make rented accommodation in some areas unaffordable for the working poor.

You don't have to be unemployed to get housing benefit, Tim. I predict bad things.

party like it's 2008

A bit of Viktorfeed. Scheduled for 1845Z, there's a flight from Dubai to Mogadishu under ICAO code JBW708. JBW? That's Jubba Airways, described by Aerotransport.org as "Formed 24/4/1998 by Canadian (Calgary) interests and the Southern Somali Business Groups (50%), in association with Phoenix Aviation. Started operations on 28/5/98."

The aircraft roster consists of two Boeing 737s, one of which belonged to both Phoenix Aviation/AVE and Kam Air, the other to Kam Air twice and East Air, a Tajik company started by Eastok Air, an operation banned in the EU since July, 2007 and which, interestingly, leased aircraft to Iraqi Airways.

brief Iraq post

Weird news from Iraq - apparently one of Sadr's conditions for returning to Iraqi politics is that Allawi and the SIIC are included. There's a turn-up for you. It does sound like the Iranians are the main actors here, and the point of including Allawi is to get minimal consent from the Sunni. Some day there's going to be a good book written on the politics between the US and Iran during the Iraq War.

Afghan links

Here at the Low Expectations Journal we've been rather optimistic recently about Afghanistan - at least relative to our expectations. This week, there's been a piece in the Washington Post that completely contradicts this. However, I would point out that this may not be as significant as all that:

Among the troubling findings is that Taliban commanders who are captured or killed are often replaced in a matter of days.

Abraham Lincoln said that he could make a brigadier into a general in three minutes, but a hundred and ten horses were difficult to replace. Isn't this the whole "Al-Qa'ida's Number Three" argument again, just with the sign reversed to justify pessimism rather than optimism? Surely the question is whether they are finding good replacements. An optimistic report is here. Exum wonders how the paper manages to run two entirely contradictory stories on successive days.

On the other hand, it's not the only case of ending up like the man who has two watches and no longer knows what the time is. Here we have two widely divergent opinions on a basic fact like the rate at which IEDs are discovered. You may recall that "Population Density of Afghanistan: Experts Differ" was actually an accurate headline for a while.

Worryingly, Jeremy Scahill reckons that the negotiations are being sabotaged by the old game of reporting whoever you don't like to the Americans as a Taliban.

Meetings

Mark Ballard of Computer Weekly is trying to get the details of government meetings with the IT industry, and struggling. Among other things, this seems to be yet another use case for an enduring Freedom of Information Act request. It's also one of the reasons why I like the idea of a central contacts register. Back at OpenTech 2009 I said to Tom Watson MP, just after he resigned as a minister, that it wasn't just useful for citizens to be able to find out who officials were contacting - the government itself might benefit from keeping track of who was lobbying it, maintaining a common line-to-take across different departments, and the like. Hey, even the lobbyists might benefit from knowing who else was lobbying.

Of course, there's an argument that the government quite likes having pathological relationships with its suppliers. But that's one of the points where as soon as you get radical enough to understand the situation, you're also too cynical to do anything about it. Watson's been campaigning about this, and the Cabinet Office recently released some data. With the embarrassing bits taken out.

The bulk of it is here, it looks like they're planning to split the disclosure between departments as this only covers ministers in the Cabinet Office (i.e. the PM, DPM, Secretary for the Cabinet Office, Leader of the Commons and the whips). It's also on data.gov.uk but it's going to need reparsing. At least it's not a PDF. It's a bit thin, presumably because the bulk of meetings with external organisations go via officials or bag carrier MPs - DEFRA's is rather chewier.

There's also a list of special advisers by department and salary, which may be handy, and has already informed me that one of William Hague's advisers is none other than Richard Littlejohn's son.

this is what "crowdsourcing" looks like

Shouldn't Crowdsourced New York Apartment Pushing Limits actually be an Onion headline? Either that or the core of a new ResPublica/New School Network collaboration as the Big Society's contribution to solving the housing crisis. We'll crowdsource it! If everyone brings a brick, we'll have...a pile of as many bricks as users who actually bothered showing up, that the two people who actually care about the project will have to use.



(Yes, I know this should be on Stable & Principled, but I'm trying to keep that blog Terribly Serious.)

Meanwhile, genuinely serious and interesting points on the same theme are made in this excellent piece on Park Hill in Sheffield and its redevelopment. It's not as smash-mouth as Owen Hatherley would likely be, but it also makes the point that letting the squatters have their way with it was tried, effectively, and a lot of the work required on the building was basically making good the results. It also strikes me as a good point that it's not, in fact, easier to run away from the scene of a crime on an access deck. Of course, the real point here is that as the society that built it crumbled, they stopped providing proper investigative policing to the people in it and started treating them as the object of mass public-order policing.

convergent mayors

Is Boris Johnson the right's Ken Livingstone? It came to mind as a result of his unexpectedly strong remarks about housing benefit. A lot of Tories disbelieve that Johnson is genuinely committed to the party. Ken spent large chunks of his career either at odds with the Labour Party leadership or outside the party. Johnson is now reprising Livingstone's role in protesting against Thatcher, while also reprising his role a second time around as an alternative version of a government he's fundamentally sympathetic to.

A lot of people remarked that Ken Livingstone, as mayor, was remarkably keen on facilitating the City's interests for someone whose staff included John Ross. Johnson is heavily reliant on the remaining ex-Livingstone officials to keep City Hall's basic functions going. Both of them put a lot of effort into maintaining a public image that is almost a caricature of their party - the whole tedious Shower Jobby act, vs. all the stuff about newts and public transport.

Of course, this overstates a bit. But I do think there's a significant truth here, and I suspect that future Mayors of London are going to have more in common with Ken and the Jobby than they will with the Prime Minister of the day. They will tend to be noisy and brash, given to ranting, and drawn back towards consensus within London by the administrative realities. There is famously no Democratic or Republican way to collect the garbage*. However, they will also tend to operate in permanent tension with the national government up river. This is an expression of the structural factors - you can't position yourself politically by replacing the Underground with a network of cable cars over the streets or abolishing school, so you've got to do so by picking fights with Westminster.

Given that, you're either going to be in the role of unofficial opposition leader, or else aligned with the government of the day's rebels, whoever they may be. Also, it seems that you'll probably end up being a couple of points to the left of your party either rhetorically or operationally. Despite all the yelling, Ken Livingstone was basically following the Blairite "let the bankers rip and then do some redistribution" plan, but with more aggression and nous. It's also true of the Jobby - for all the bullshit, he's not actually changed that much, which puts him some way left of the cuts consensus. Interestingly, this also seems to be true of Bertrand Delanoe and Klaus Wowereit, and perhaps also Michael Bloomberg.

* This argument may no longer seem as convincing as it once did, as there are probably Republicans who want to abolish rubbish collection...

the Pentagon unchecks "internet connection sharing"

The rate of intrusion attempts on US government networks has fallen this year. Obviously, this is going to be a data series dominated by the spikes, so a good botnet between now and Christmas could change that. But it's a nice correction to the constant "cyberwar" bollocks. Also, check out the hilariously .com boom era graphic - streams of ones and zeros! Scary words, like ENCRYPTION SOFTWARE, in fake-LED display fonts!

a slight return from 2004...

Two things: brief piece about the Toyota Hilux, preferred transport to the world guerrilla, and a fatwa against mobile money transfer. You know you've made it when you've been fatwa'd.

More seriously, this huge Guardian piece on Iranian policy in Iraq is well worth reading. It's interesting, to say the least, that the people Sadr wanted to see as guarantors of Iranian good faith were Hezbollah - it would seem they've got a foreign policy these days. Also, there's a sort of disguised alliance between the Iranians and the US. The Americans fought hard to stabilise al-Maliki's government and to build it up as a credible force. It's hard to imagine they really want rid of him now.

And there's this:

It is understood that the full withdrawal of all US troops after a security agreement signed between Baghdad and Washington at the end of 2011 was also sought by Sheikh Nasrallah.

"Maliki told them he will never extend, or renew [any bases] or give any facilities to the Americans or British after the end of next year," a source said....US officials have strongly suggested they would scale back their involvement in Iraq if the Sadrists, who have been a key foe throughout the years of war, were to emerge as a significant player in any government.

But it's their policy to leave:

On the 2011 December withdrawal date, the official said: "Any follow-up engagement with Iraq in relation to troops would be at the request of the government of Iraq. There are no plans to keep troops after December 2011. We are drawing down and all will be out of Iraq."

The piece seems to be heavily influenced by sources in Al Baathi Allawi's entourage. You do wonder if Allawi was only fetched out of the deep freeze in order to press Maliki into making a deal with the Sadrists.

Why are these scumbags so scummy?

John "War Nerd" Dolan got a job, as a lecturer at the American University of Iraq. Hilarity ensued. You bet. It's a tale of un-fantastic right-wing academics, a kind of glaring dullness, a total lack of character, and an endless supply of raw cash. It so happens that John needed that more than anything else, so good luck to him. Read the whole thing - what stands out is the vast gap between the neo-con obsession with The Western Canon! Classicism! Principle! Courage! and the petty, provincial, small-mindedness that people like Joshua "Not The Blogger" Marshall practice in their lives. It's not even the incompetence. It's the style that gives them away.

The other interesting thing in the piece is John Agresto's role. Again and again, he turns up wondering why a string of horrible political thugs treated him with disrespect. Lynne Cheney, his old boss, seems to have been a really awful human being close up. Who knew? But somehow, it never crosses his mind to wonder why this keeps happening every time he associates with the Cheneys or Bill Bennett or some other horrific political gargoyle. It's....full of bastards, just this particular astronaut isn't going to get out of the ship.

I also loved the notion of a neo-conservative as someone who got mugged by reality and now never goes into town for fear of running into reality again. A lesser writer would say that he started carrying a gun in order to shoot reality. However, that would imply some kind of grand, tragic struggle against brute fate. You can't have tragedy without dignity, and that's one thing the administration of the American University of Iraq doesn't have.

This reminded me of two things, or rather the other way around. If you want Mitt Romney to speak, you've got to take a bulk order for his booky wook. Hence the book is a bestseller (for whatever that means in today's book trade). Similarly, 'bagger Sharron Angle's campaign raised $14m and paid $12m right back to the political consultants who organised the donation drive.

The other thing was this documentary series on YouTube about Americans and steroids. Two points come to mind - the enduring role of the quack, and a sort of grinding optimism. And this quote: "Everyone wants to be a monster."

A critical point, though - I'm fairly sure the sheaf of documents one of the doctors waves while reading out a list of horrible side effects that turn out to relate to vitamin C is from an open-access "adverse event reporting system", which basically gathers anything anyone anywhere feels inclined to report. They aren't verified in any way. Anti-vaccine people often abuse this.

Fortunately, someone's done the actual journalism and documented that the ties between multilevel marketing, quackery, and extreme-right politics aren't just style, they're organisational and financial. It goes back a while, too.

new emerging threats

Swinging off my Stable & Principled contributions, this is ridiculously great. Who the hell is this menacing new competitor at the intersection of naval shipbuilding, MoD criticism, and obsession with mobile computing devices?

Oddly enough, I keep adding new blogs to my RSS queue at the moment. The scene burns on. On which theme, time for some music. The Beat vs. Smokey Robinson, so that's not only a cover but also 2-Tone vs. northern soul.

all around the world I've been supporting team blogging proposals

I have mostly been blogging the cuts at Stable & Principled this week. Somebody had to.

How the Government produced a comprehensive spending review without mentioning the monster housing bubble or using the word recession more than once. Smoke and mirrors, and housing benefit cuts. Has Dave from PR got his jets mixed up? Why the Army has quite enough helicopters, thank you, and past statements are no longer operative. Carefully taking no decisions at all on defence. Can the Shirley Porter strategy save the Shower Jobby?

two related stories on Afghanistan

Suddenly the Afghanistan news is full of talking. Petraeus says that ISAF provided one or more significant Taliban leaders with safe conduct to Kabul in order to take part in what sound like "talks-about-talks". The Grauniad was ahead of the story on this one, confirming that contact had been made with the Haqqani network in particular. Note the usage here:

Drawing a parallel with the Northern Irish peace process, the diplomat said: "The Haqqanis know they have to make the transition from the IRA to Sinn Féin."

Dexter Filkins makes the point explicit.

In recent weeks, General Petraeus has increased raids by Special Forces units and launched large operations to clear territory of Taliban militants.

And it seems increasingly clear that he is partly using the attacks to expand a parallel path to the end of the war: an American-led diplomatic initiative, very much in its infancy but ultimately aimed at persuading the Taliban — or large parts of the movement — to make peace with the Afghan government.

Spencer Ackerman sounds faintly bereft, and remarks that:

It would be quite an irony if the chief counterinsurgent prosecuted a hit-em-n-quit-em campaign that helped convince the Taliban that enough is enough

Interestingly, this is actually how his plan in Iraq was originally meant to work out. A combination of counterinsurgency centred on Baghdad, aggressive action against selected groups in the insurgency, and political action was meant to get the violence down to a level at which there would be an opportunity for a negotiated peace between the major factions. It didn't work out like that - to everyone's surprise, mine included, what little peace there is in Iraq came about from below, from local initiatives, while the grand bargain never happened. Iraq still doesn't have a government; the census and the related federalism issues are forever delayed, as are the oil issues.

And the really worrying news on that score is the recent rise of violence at the individual level, as shown by the wave of assassinations against policemen and officials. The danger is that the shaky raft of improvised local deals, which allows the Iraqi politicians and the foreign diplomats to keep arguing without anything too disastrous happening, might break down. In Afghanistan, it looks like the plan is to focus on getting as many enemies on side as possible - and not looking too closely at the raft. This Small Wars Journal piece makes the excellent point that in some ways, the more closely you focus on the whole situation, the less you know - it dissolves into a fractal mass of micro-conflicts. It also practically exudes frustration and the desire for it all to be over. (Who wouldn't.)

Of course, it doesn't do to put too much reliance on the words here. As pointed out in Ackerman's comments, whether you call it counter-insurgency or not, it's still war. However, it's very telling that the "kill team"'s commander was quite so addicted to the rhetoric of the early Bush years. Sean Naylor's piece here quotes him as saying that he wanted to "degrade formations, supply chains and leadership near simultaneously, you’ll cause the enemy in the area to collapse". It's an application of all the stuff about "breaking states through simultaneous strike" that you'll find here.

Also, he was convinced that he had better information than the Taliban because of his vehicles' new computer system. The combination of the hubris of the network-centric warriors, the land submarines, and enthusiastic grassroots participation in torture and war crimes - it's the great smell of the Bush era, all right, and it will clearly be with us quite a lot longer.

RST

Does anyone have any idea why I'm banned from reading ForeignPolicy.com? For the last few days, the three FP blogs I subscribe to haven't been updating, and trying to read this I had to use an anonymous SSL-proxy server. Just for that "test your practical circumvention skills" feeling! I can ping and traceroute to their servers (Amazon EC2 - look at you all cloudy and everything!) but when I send them an HTTP GET they immediately kill the TCP session.

Logistics!

So, the raids on NATO trucks held up when Pakistan suspended the border crossing. A good point is made in comments at Adam Elkus's blog - what about the people who own the trucks?

What indeed. The so-called "transport mafia" played a critical role in the creation of the Taliban in the early 1990s, according to Ahmed Rashid. Back in the 1980s, one of the ways the Soviet-Afghan war transformed Pakistani politics was that an economy grew up to service it. Famously, this is what Osama bin Laden actually did for the mujahedin - his construction firm built the roads up to the border, his organisation received new recruits in Pakistan and passed them on. Logistics. Another element of this war economy was a network of transport firms that trucked the war material the Americans were supplying and the Saudis paying for up from Karachi to the border.

Most of these were close to the politicians who also benefited from the war - the NWFP Islamist parties and the feudal landowners who made up the right of the PPP. In fact, very often, they were actually owned by politicians, or by their proxies. To make sure the money fell in the right places and the trucks went to the right places, the Pakistani army created a new agency, the national logistics cell, which was responsible for divvying up the contracts and organising the operation.

After the war was over, the system stayed in place and became part of the general berserk vision of extending Pakistani and Saudi influence into Central Asia. The military would get to implement the strategic depth concept, and keep recruiting jihadis to use in Kashmir. The jihadis would get to continue their never-ending tour. The Saudis could spread Wahabism and dispose of their malcontents. The ISI would reinforce its special role in politics. And the transport mafia would benefit from what appeared to be enormous economic opportunities trading through Afghanistan into what had been the Soviet Union.

In fact, when the Pakistanis came to pick a proxy in Afghanistan after 1992, the choice was between Gulbuddin Hekmatyar and the project of creating a new movement. The ISI wanted to stick with Hekmatyar, who they had originally sent into Afghanistan in the mid-70s. However, the other beneficiaries of the war weren't satisfied with him - in the first Bhutto government, the logistics mafia and its friends were very powerful indeed. The key figure was the Interior Minister, the former chief of the Frontier Corps who had recruited the first generation of mujahedin in the 70s.

From a left-wing point of view, a crucial factor here was that the whole imperialist vision of caravans of trucks trading across the Hindu Kush as far as Siberia was a form of economic development that went straight to the traditional powers in Sindh via their new investments in the war economy. A stereotype view might be to say that the PPP was a mixture of Benazir and Bhutto - mass protest politics, and the feudal world. It was this intersection between internal Pakistani class and regional politics, grand strategic visions, and tactical opportunism that led them to support a group of Afghans based in Spin Boldak. Later, during the wars of the 90s, the Taliban repeatedly benefited from transport supplied by the NLC.

So you've got to wonder if setting fire to a load of trucks isn't overreaching a bit. The role of transport and route security just can't be overstated here; the irony in this McClatchy piece is intense.

For nearly a decade, the U.S. has spent hundreds of millions of dollars trying to cut off the remote, high altitude mountain trails Taliban forces use to smuggle weapons and fighters into Afghanistan. Now, the U.S. military is turning its attention to the border crossing.

"More and more we've realized that they are not coming through the passes, they're just coming through the . . . gate," said one U.S. government official in Afghanistan who spoke on the condition of anonymity so he could candidly discuss the unfolding plan to focus on the border crossing.

On the other side of the border, here's the guy who charges $1,200 a truck for safe passage. People are starting to notice; the US Host Nation Trucking contract amounts to 10% of Afghan GDP, paid to companies controlled by the Afghan government's relatives.

Obviously, there are a lot of people in Pakistan who would be delighted to set fire to trucks owned by a Northern Alliance defence minister's son, but as far as I know a major dispute between Afghanistan and Pakistan is precisely about whether cargo can move through Pakistan in foreign-owned vehicles. This strongly suggests the movement on the Pakistani side is controlled by the same old, same old people. Key quote:

Until now, the diplomat said, protection of the route had not been needed because the delivery rate had been remarkably efficient given the length and rough nature of the route from the port of Karachi.

I suspect that if they want it delivered, it will be delivered. I've even heard it suggested that some of the cargo burned was insured in advanced, which if true would be impressively sick - it's not often you get to have your own foreign policy and pull off an insurance fire at the same time.

two links about freedom of information

I do worry that a lot of the continued drive for government data release is really about the hope that the public will find useful sunday-for-monday press release fodder that will help spin the cuts. (By comparison, you try finding a full list of cuts.)

Did you know New Labour was spending your money on FRUIT FLIES? It reminds me a little of those Americans who started a Twitter feed to mock the 'baggers complaining about stimulus fund projects and found that hordes of people believed every one of their randomly generated line-items. If you sort the list by size, you'll find that the biggest organisations in it are the bits of HMRC that handle customer, or rather taxpayer, facing operations - processing tax returns, answering phone calls, dealing with visitors in person. All the stuff the guff squad gets het up about usually turns out to account for two AO 1s, half a desk, and a 10% share of the office dog's time.

Meanwhile, Angus Gascoigne points out a new Microsoft Dynamics CRM-based product that lets you "manage, track, and respond to Public Records Act and Freedom of Information Act requests in a timely and responsive manner". I'm actually quite positive about this. Incompetence is as effective a barrier to freedom of information as secrecy - in fact, it's arguably better. There are no legal bounds to incompetence. Also, quite a lot of FOI legislation, notably the UK's, has exclusions based on cost, which means that it's always in the interests of the government to maximise the costs of retrieving any given document. This is what the ISP industry used to call "strategic incompetence" in their OFCOM filings against BT - of course, we have to offer unbundling, but sadly your letter has gone missing yet again...

Making disclosure part of an automated workflow strikes me as nothing but a good idea - it's hard to do things the other way than the default.

learning the wrong lessons from an interesting chart

(Via here.)

Apparently the interesting bit is:

the extent to which the public vastly overestimates the prosperity of lower-income Americans. The public thinks the 4th quintile has more money than the median quintile actually has. And the public thinks the 5th quintile has vastly more wealth than it really has...You can easily see how this could have a giant distorting effect on our politics. Poor Americans are simply much, much, much needier than people realize and this is naturally going to lead to an undue slighting of their interests.

The other interesting bit is the political breakdown. If you look at the second chart, which represents what the sample thought would be an ideal distribution, two things become obvious - one, the ideals are not very different, two, they are all significantly more egalitarian than the reality. People who admitted to voting for George Bush wanted to redistribute wealth quite radically - even when you compare their preferences with their illusory beliefs about the distribution, they want a very significant change. Compare them with reality, well...

The rest is pretty obvious - the least egalitarian group is those earning more than $100,000, the most are those earning less than $50,000, women are more egalitarian than men as a group, but no more so than declared John Kerry voters.

The problem is clearly much wider than their estimates of the lowest quintile's wealth - in fact, although the people studied were unaware of quite how bad things were, they are clearly very well aware of inequality and they want it to change. And this sweeps right across the board. Even if they were unaware of the full poverty of the poor, they were well aware of the rich.

What they need evidently isn't Blairism - scrape a bit off the top and pay it out to the poorest, with all kinds of interlocking and perverse conditions drawn up by the Yglesians of this world. This is why universalism is important - it's possible that the whole discourse of "targeting" as applied to social policy reinforces the delusion that the poor are actually rich.

the participatory panopticon strikes back

Now that I'm on last.fm I can have one of these!



I can't help thinking of this in Gini/Lorenz terms - am I right in thinking there's interesting information to be had in how the line-of-best-fit of the plot skews around the 45 degree mark?

replacing myself with a rather complicated PHP web application, it seems

Here's a question. Having seen the Google's new "Priority Inbox" feature and also John Graham-Cumming's POPFile application, both ways of using a Bayesian classifier to guess which e-mail you will want to read first and to file it automatically, I was wondering if anyone had applied the same idea to RSS. I've recently started to add new blogs to my reader again, and it struck me that reading them took up enough time that it might be useful to prioritise and classify them automatically. It might even be yet another project I probably won't find the time to finish.

Searching the web, though, I was surprised to find quite a lot of similar projects that didn't seem to have many users or for that matter to be in active development. It actually looks like this is one of the problems that almost all developers at one point or another feel the need to tackle. But nobody's made it stick. Somebody even had their RSS feeds delivered by e-mail and used POPFile itself, but that's silly. I can think of a couple of reasons - one is that the use case might be fundamentally flawed. If it wasn't for surprises, the blogosphere would be pretty dull - otherwise you might just read Martin Kettle's column or watch TV. If you could have a feed of blog posts that you were guaranteed to read, would you want to read them? Of course, you could introduce some sort of random element, perhaps promoting some proportion of the posts least likely to be read, but that would defeat the point.

One feature which I didn't see anywhere was a social element. I could certainly see a use for an application that classified RSS items into groups, and let multiple users contribute to the same group. I mark some of the items as "Telco 2.0", and therefore train the classifier to filter things relevant to the company into that bucket. But other T2 people have opinions about what is relevant to the company, and they might benefit from mine as well. Obviously, if we use the same classification profile we'll get the same results - interestingly, we'll get the same results in some sense even if we're not all reading the same blogs. So I'd like to be able to have shared group filters.

Does anyone know of an application that does this, preferably without letting some random website see everything I read? Points for integration with other RSS readers, notably either Akregator or Firefox/Sage. I'd be OK with a web page served on localhost (or on a server I control). At the moment, this is in the lead, but it strikes me as being rather more heavyweight than is ideal.

Thursday music link...on Monday

Thursday music link on Sunday didn't happen because I was out of the country and offline for 2.5 whole days. So here goes, and thanks for contributions so far. They said they wanted to make the girls dance, but that was widely considered unsuccessful. This suggests there's been some progress.

I'll show you thursday music link...(on Sunday)

How about some music? The aim of this feature is to make D^2's thursday music links look silly, and the only rule is that everything has to be performed by someone other than the original. Call it an agonising reappraisal. Here's the Clash with Janie Jones, doing "(Get Up I Feel Like Being A) Sex Machine" in 1982.

baaack...

It seems horribly fitting that, with the Tories back in, Wigan have started winning the league again. I mean, not that there's anything wrong with them. It's just the effect of the years when they won literally everything, year in, year out. And Maurice Lindsay, of course.

If we have to have Wigan though, I think the Wigan who played last night are about as good as you could get. The real giveaway of how good they were was how good St Helens were. There is still no club like them for style - they started playing catch-up in the first half and would have outplayed most sides, but against Wigan they just didn't quite convert the chances they created. To be honest, that's usually a sign the other side were better... Even the fact Wigan missed so many kicks and used three goal kickers (oddly, not including Paul Deacon, who was available and playing a damn good game) didn't stop them.

Even Keiron Cunningham's last spell in the game didn't change anything. I think they're going to win and keep on winning.

the frontier

Am I right in thinking this is a form of "superempowerment", of the NATO forces on the border, the Taliban, and the Pakistani Frontier Corps on the other side?

Pakistani authorities say that the checkpoint guards tried to alert the US helicopters that they had strayed into Pakistani territory by firing in the air, but the US pilots mistook this action for a hostile attack and blew away the checkpoint.

Any one of them can trigger a violent response from the other, which rapidly flips the whole situation into a higher energy state, with consequences at least up to the operational level. Of course, the FATA are only sovereign territory in a very special and restricted sense of the word "sovereign" - but arguably, optional sovereignty is a useful political tool, permitting the Pakistani state to a) tolerate the jihadis in some parts of the country when that is useful, b) tolerate the Americans in the same places when useful, and also c) assert sovereignty to push back on the Americans when useful, in the light of this.

This is actually roughly what Gallagher and Robinson meant with the "crumbling frontier" 50 odd years ago - zones of ambiguous sovereignty were important because they provided reasons for imperial expansion, reasons against it, and a way for peripheral political actors to use the empire for their own ends.

This is an article about a statistic

(Inspiration here.)

A numerical variable was today reported to have either increased or decreased or remained constant. Depending on which one, this may represent a record value for this variable, a dramatic rise or fall since whichever point in time is required to show a dramatic rise or fall, or nothing whatsoever. In a development which is probably entirely unrelated, although there is no way the business editor will admit this and publish this bit, the FTSE-100 share index rose or fell slightly on the news.

Speaking to this website, a spokesman for a lobby which claims to care about the current value of this variable said that the integer demonstrated clearly that the lobby was right. The spokesman said that the government must act, that the government must immediately stop acting, or perhaps that the current value of the variable showed that although insufficient, the government's policy was a step in the right direction. In any case, it demonstrated the enduring relevance of their members' concerns.

Reached for comment, the Ministry of Variable said that it was going to take tough action on the number. Friends of the minister said he or she fully understood their concerns, but that he or she would not be stampeded into action. However, the minister will say, modernisers would not be held back in the comfort zone by variable interests. The Shadow Minister for Variable said that the government was relying on a fundamentally flawed measurement and that their own preferred measure showed that the variable should be significantly higher or lower. He or she accused National Statistics of twisting their measurement of the variable to suit the government of the day. The Minister's office retorted that they would take no lessons on variable from a party that had allowed the value of variable to rise to record levels, fall to record levels, or stagnate at a constant level when they were last in office.

The Campaign For or Against Variable said that the public were in danger and the precautionary principle should be applied. "So-called statisticians claim that this level of variable is perfectly safe, but how can anyone really know? Also, the weekly level of variable has been recorded as being as high or low as X in the last six months, when the statisticians say there is only a 5% chance of this. So how come it's happened once in 24 weeks?"

The mean value of variable over the last 20 years is Y, and it typically varies as much as Z year-on-year. Over the long term, variable in the UK is typically A% higher or lower than the average of OECD countries, EU countries, or the world. On the basis of variable's distribution and standard deviation, this week's value could be expected about every B years, and therefore this news is either important or pure noise. To be more accurate, the variable should perhaps be given as a percentage of GDP, as a per-capita value, a median value rather than a mean, or as a percentage of some total or broader average. It's very likely that this may be explained better by drawing a graph. Unfortunately, this paragraph was edited out of the final article, or quite possibly the author never bothered to write it in the first place.

Both the people we spoke to in the street, because they looked likely to say something sensational, who recognised variable but didn't bother us with comments like the paragraph above said they were deeply frightened and baffled by the issue.

a glimpse of the press in action: Johnson family edition

Sorry about this, but yet more journobashing follows. And I warn you that the actual information in this story is media-bollocks, but it may be interesting all the same. Zoe Williams interviews Rachel "Boris Johnson's Sister" Johnson, editor of the Lady.

Rachel Johnson has been editing the Lady for almost a year. In that time, it has generated a spectacular amount of chat – gossip columns, a massive article in the Sunday Times, a full hour on Channel Four, and now a book, A Diary of the Lady: My First Year as Editor (which, incidentally, is a total romp. I don't know why I'm talking like this. I seem to have caught something from the book, a contagious mannered poshness).

That would be another Moretti Moment, I think. It gets better, a bit. But the problem with this piece is a classic case of burying the lede. You have to get through 16 paragraphs of stuff like this:

Johnson is possessed of a great deal of charisma, which makes her seem incredibly beautiful in a way that surprises you afterwards, because in photos she looks like a pleasant, bossy, female version of her brother, Boris...I don't really see it as anything to do with journalism, or editorial in-fighting, or the fortunes of the magazine, but more a series of small skirmishes that are so closely, unflinchingly described that they are magnetic, the way a mother and a daughter fighting in a shoe shop is magnetic, even though you know exactly what they're going to say...Amusingly, every time she leaves off from her detached self-flagellation and is unpleasant about anyone but herself, especially anyone in the Lady offices, she finishes by telling you how beautiful they are, or, if that absolutely won't stand, willowy...

...before the remaining readers who haven't yet scooped out their lobes with the kitchenware to escape all this hideousness arrive at some content, yer actual Shannon information.

It is pretty obvious that this just isn't a family trying to revive a flagging magazine, it's one that wants a bit more heft in the world. They want the mayor of London's sister on speed dial, they want an editor who has a picture of herself talking to David Cameron on her noticeboard. "They just want a seat at the table, don't they?" She shrugs. "The magazine's not a power in the land, though, is it? It could be again. What I think would be quite nice would be to have an alternate offering to the Spectator, which looks lovely. We're not intellectual, but we don't talk down." "You want it to be the Spectator's wife?" "Yes! Exactly!" This wasn't at all my point, but it's not the first time I have to submit to her superior conversation-management. Her appointment, the subsequent sackings, the documentary, none of this has anything to do with a magazine. I feel bad for her that she's sweating the numbers, but then it occurs to me that maybe the intoxication of power makes up for it.

So the point is to convert the paper into a new right-wing organ? Now that's actually interesting and informative. A new right-wing organ edited by the Mayor of London's sister (and that of the MEP for the South East and MP for Orpington). That's also actually interesting and informative, especially as a lot of Tories apparently don't really accept Boris Johnson as a genuine Tory, and tension is picking up between Johnson's administration in City Hall and David Cameron's government upriver in Whitehall.

And you can see how this could work. The owners obviously aren't particularly obsessed with the circulation or the display-ad revenue, or they would hardly have let it go on in its current form all these years. Or, for that matter, appoint an amateur catblogger whose chief qualification is being the Mayor of London's sister as editor. The point of an opinion mag isn't circulation, but influence, or propaganda - it's about influencing the stuff that gets into the newspapers that do have a real circulation. The paper does fit rather well with both the Johnsons' public image and the aesthetic culture of the current Tories. And the rather grand offices in Covent Garden look like a great place for an "event".

Also, getting your sis appointed editor is a lot cheaper than buying the paper. Further, the fact she brought her husband (the communications director of the National Trust, would you believe it) to the job interview is interesting.

So, if we were going to reconstruct this as journalism, we might move this couple of sticks up to the lede, just ahead of the rather good point that despite all the "chat" the paper's circulation has gone nowhere. In fact, we might make the point that rather than "chat", in its political context, this is the real substance of the Johnson family - relentless self-publicity and self-promotion transmitted through a total determination to attend the opening of every abscess in town, to say nothing of envelopes, and to get in front of every camera that isn't marked "Westminster City Council". We might also set to editing the rest of the story with a two-handed Viking broadsword (Hack! Slash! Cleanse! Flense!), but there's them as likes that guff.

a glimpse of the press in action: Ed Miliband edition

Has anyone else found that the Guardian's coverage of the Labour conference made them want to throw up? Here's Patrick "Unseasonably Mild" Wintour:

But his efforts to show he can lead a united, re-energised party committed to redefining the centre ground of politics look likely to be hampered by the expected decision of his defeated brother, David, to leave frontline politics to give him the chance to lead on his own terms.

Ed Miliband is said to be still trying to persuade him to remain at the top of Labour politics, and insisted there was "no psychodrama" between them. David Miliband has been offered the shadow chancellorship, but friends say he does not want it. More broadly, they say, he fears that if he remains the brothers will be ground down by rumours of splits, jealousies and factionalism that disfigured nearly a decade of Tony Blair's premiership.

So David Miliband's decision to resign is both going to "hamper...his efforts to show he can lead a united, re-energised party" and also prevent "splits, jealousies, and factionalism". Right. Also, the actual news in the story - that David Miliband isn't going to serve as shadow chancellor - has been buried, because it's more important to show that Wintour still gets given talking points by Blairite press officers. The piece attributes statements to "friends", "aides", his "team", or to nobody at all ("it is said") some eight times in 23 paragraphs. Also, there are four "he will say" statements - i.e. "I have been given a copy of the speech, like the other reporters, and like them I am trying to retail this as if it were a secret".

Meanwhile, we get a vast quantity of stuff about David Miliband's wife and his hurt feelings (no less than three Wintour pieces use the same quote about her). ""David is giving Ed some space to carry on torching the house we built." comes up in some form multiple times as well.

So much for Wintour. As usual, if it was news you were after you'd have done better to read the paper's second string political reporters, notably Allegra Stratton. But the rot is not entirely contained. Here's Patrick Barkham.

For his three days as Labour leader, Ed Miliband has worn purple ties every day: first a gravitas-imbued deep purple and then a heavily textured lilac affair yesterday that was far closer to blue than red....Miliband's purple policy looks very deliberate given his scattergun approach to ties in the past. Mary Riddell, the commentator, has noted that during the intensive pre-Copenhagen summitry last year, Red Ed liked nothing better than to buy "garish ties from street traders" in New York. Was he stocking up on purple in preparation for his immaculately-tied leadership tilt?

Honestly. There's a scene in Nanni Moretti's film Caro diaro where the protagonist, stuck in Rome in August with not much to do, has gone to see Henry: Portrait of a Serial Killer based on a review in one of the newspapers. Having walked out of the movie, he tracks down the critic and confronts him in his bed, shaking him awake and forcing him to listen as he reads out the review. He cowers with embarrassment at each sentence. Reading this slab of dreck (so he wore a different colour tie each day, and this is a departure from a "scattergun approach"?) I felt like doing something similar. At least on the Daily Hell you're allowed to do an Alan Smithee and have pieces like this attributed to DAILY MAIL REPORTER when the management insists on them.

There's more, and worse. Jon Henley is sent out to do that classic piece of vacuous psuedo-reporting, a vox pop. In order to grasp the temper of the peuple de gauche, he gets sent to Pangbourne of all places. Even there, though, he struggles to find the rage he was sent for.

"We don't vote on stuff like that any more, do we?" said Dan Perkins, 31, a geography teacher supervising a group of schoolchildren..."Judi Green, 34, said she thought that was "brilliant. All to the good; really quite refreshing. The sign of a new era, even."

He pays a visit to the local working men's club:

In the working men's club – "the only working men's club for millionaires in the country", one winking lady member suggested – the lunchtime drinkers, of an earlier generation, weren't quite so sure.

Oh really?

"They'll use it against him, the tabloids and that." ...Ann Willoughby, a widow from the former council estate who would, personally, "have liked to see John Prescott get it, because he's really one of us", agreed Ed Miliband's family arrangements could be "a stick for his enemies to beat him with. They'll try anything."...Her son Mark, 42, an HGV driver, reckoned simply that "none of that stuff bothers me, and nor should it bother anyone". Jane Turner, a single mother of 43 on her bike, said she couldn't care less either. ...it would be "wrong to judge someone on that, in 2010"....felt strongly that it was "far better for a politician to be honest and up-front than pay lip service to a faith you don't have"...

In fact, the only people he encountered with any objection to Ed Miliband were two self-declared conservatives, one of whom turned out to be William Hague's sister. But he could always try back at the office. Here's Madeleine Bunting:

What's really at stake here is not a few details about his family life but an accumulation of characteristics that speak to the cliche of a metropolitan liberal elite. It's part of why the Blairs came to be so distrusted and Miliband will have to work hard to head off the damage that some of these associations could generate.

This is in a piece about the fact that Ed Miliband is not married. So, what are the other characteristics in that "accumulation"? She doesn't say, except that ("metropolitan") he lives in London, like all the other 650-odd MPs and Madeleine Bunting. I wonder if he might be a rootless cosmopolitan or a sinister left-wing intellectual? Has anyone heard if such might be the case? Worse yet, apparently the reason why the Blairs came to be so distrusted is that they weren't married. Who knew?