a Christmas gift for you: UK Wired edition

I think I'm going to cut the UK Wired out of my diet. They recently got an award, and the mag is usually a handsome book, but it's not good enough. Typically there's a bit less than one genuinely interesting piece per issue, which is nowhere near enough to make the cover price worthwhile. The best stuff is reserved for the US edition - you know the names. And there's far too much filler. A big part of the mag trade is making the inevitable filler part of the whole aesthetic; the best magazines are always good at this. The Face's nibs and odds-and-sods were almost a decent fanzine in themselves, as well as providing spacers between the real stuff and a support-scaffold for the ads. But UK Wired is riddled with advertorial - constant gadget porn, but nothing you could call a product review.

There's a strong argument that there are plenty of publications that do gadget reviews. We used to say on MCI that "we are not a gadget mag". Indeed. So chop it out, unless it really is advertorial rather than editorial that happens to be shit. And if it's paid for, it's polite to say so. Also, given the bulk and kind of ads they carry, they really have no right to be playing daft games with filler. In the latest issue, there are a total of two ads for anything I expect ever to be able to afford and both of them are for web hosting services.

It's a pity, though, that having used one of those advertisers' e-mail hosting and picked endless spam out of Fistful from the other's compromised hosts, I wouldn't use either of them in a fit. This is the tone of the whole thing - the design is fancy, but it's not structural. It's there to look at, not through, or better, with. The writing tends to be tech-y, not techy. The piece on synthetic biology in this month's paper, for example, could have got into a national newspaper, it's that lightweight. (On the other hand, the piece on re-designing Mecca's infrastructure to keep pilgrims from getting killed was genuinely interesting, but crammed into a corner.) The photography is good, and the production is luscious, but here's the problem: high gloss is ads, not content. The columnists are reliably disappointing: Warren Ellis doesn't say anything you couldn't get from his blog, and anyway, if you're going to hire him, why not hire him to do what he does for real? Some people would buy it for Ellis cartoons alone. And there's the guy whose career is based on having marketed MS Internet Explorer, a product you get forced on you when you buy a new PC, and that you have to get rid of by surgery before it explodes messily. That's like marketing the appendix.

But it wouldn't be so bad if it wasn't for David Rowan's risible leader column. This month, he decided to cover, or at least mention, Dave from PR's trip to Shoreditch to say how great Internet start-ups were. Unfortunately, all he could find to say was that the industry needed:

"tax incentives, networking events, and an ability to hire and fire more easily to attract people like Le Meur back".

Well, the first adds up to "gimme subsidy", the second is vacuous (and self-interested), and the third is pernicious. Most new businesses die within three years of launch and their employees go down with the ship, so by definition this can't be all that important. But if you work for the contract cleaners, I suspect this is significantly more important to you. There's another hiring constraint, though, that really is relevant here: immigration restrictions. We know much bigger companies have been complaining through the CBI about the new numerical cap on immigration. Strange, no mention of that, especially as it's the current government's own, direct, and specific fault. (And we could also be snarky about the fact Loic Le Meur's business puts on a conference every year in Paris, that well-known paradise of hardcore libertarianism.)

If Rowan had wanted to cover the Cameron trip, though, there was a good story in there for a real journalist. Rumours included that staffers from No.10 had called up BT and suggested that they might want to put in free WLAN "from Old Street to the Olympic Park", with no suggestion that there would be any funding for this. Another suggests that they asked if BT would consider moving their enormous R&D centre from Martlesham Heath to the East End, a hilariously enormous project and one that takes no account of the major operational infrastructure BT has at MHRC.

Less sarcastically, if you're a community broadband project, you're not allowed to offer service to any kind of business if you're using BT passive infrastructure. And you've got to pay business rates on your network as soon as it's built, while the incumbents can wait until it's in service and earning money. There's an OFCOM consultation on the small business telecoms market that's been hanging fire for ages. Journalism!

a Christmas gift for you

I approve of this message. What was the BBC development hierarchy thinking? As Vowl says, it wasn't even so much the content, awful though it was, but the quality.

I approve of this message. What was the BBC development hierarchy thinking? As Vowl says, it wasn't even so much the content, awful though it was, but the quality. Airport documentaries: might have been funny, in 1998. Stelios hasn't actually been in charge of EasyJet in years, and IIRC he doesn't own it any more either. An Asian character who's obsessed with hip-hop and constantly talks about "bitches": well, Ali G was funny, in 1996, and he's somebody else's material anyway. Sacha Baron Cohen should sue but he probably doesn't want to associate himself with this shite. Stealing jokes is one thing, but stealing ones that will soon be old enough to join the Army is pathetic.

Also, if you're going to poke fun at crappy low-cost airlines' grasping, self-publicising executives, surely Michael O'Leary's endless grandstanding and bullshitting must be a seam of comedy gold...unless you're Matt Lucas and David Williams, in which case you're clearly too scared he might sue, so the other bucket shop is still Irish but has to look like Aer Lingus.

Something else: production values. Obviously something posing as a cheap docusoap has to look cheap, but once you spend a certain amount of effort pretending to be shit, the face grows to fit the mask. I didn't actually see any sets falling over, but perhaps I wasn't paying attention. Perhaps I don't watch enough TV, but was this the worst slab of dreck broadcast in the last 10 years? Further, you, me, and everyone else is going to be rolling out to defend the BBC enough times in the next twelve months that we'll all get even sicker of it than we did during the Hutton inquiry, and this isn't going to help.

an infallible scheme for redesigning Britain

This paper in PLoS One is fascinating (if heavily blogged already). Basically, BT let some researchers from MIT, Cornell, UCL, and their own R&D division have an anonymised slice through their call-detail record (CDR) pile, the database from which phone bills are calculated. The scientists filtered out all the numbers that only made or accepted calls, in order to get rid of the call centres and spammers, and drew the rest as a massive directed multi-graph network. The conclusions are fascinating; in human terms, Wales isn't a meaningful unit, and neither is England. Scotland, however, forms a well defined sub-graph.

Instead, Wales splits into three geographic tiers with very little interconnection. These regions don't respect the border at all - not surprisingly, the northern tier is completely integrated with Liverpool and Manchester and the central tier with the West Midlands. South Wales is clearly identified, with a sharply defined border along the water between it and the West Country. There's also a well-defined western border to Yorkshire, and interestingly also between the West and South Ridings but not between them and the North Riding. Essex is an extension of London, but Kent is distinct. So is Norfolk.

In fact, England isn't really identifiable on the maps: surprisingly, the administrative units that fit best to the BT data are the EU regions much hated by 'kippers. More broadly, if it's got a recognisable accent, it's a recognisable presence on the graph - although the big exception is Yorkshire. There's even a territory for people with no recognisable accent, a sort of motorway crescent to the west of London which is described as a "tech corridor" - in fact, if you were to draw all the Formula One teams' workshops on the map, they would essentially all fall within it, as would Vodafone, O2, Cable & Wireless, and 3UK's headquarters, Aldermaston, Eidos, Surrey Satellite Tech, chunks of BAE and Thales, and Electronic Arts UK, so perhaps they have a point. In the end, though, this potentially interesting zone - Ballardia? - gets lumped in with the Cameroonian central-southwest.

a glimpse of imagery diplomacy

This is one of the most interesting stories in the Wikileaks cable dump. The Saudis use the existence of the French national imagery satellite capability, and David Ignatius's column in the Washington Post, to resist efforts by the Americans to stop them using US arms and satellite data provided for use on Al-Qa'ida for other goals of foreign policy, notably trying to encroach on Yemeni territory. Of course, the UK isn't allowed to do that.

Back to 2006

Bérube sez:

So these days, when I talk to my scientist friends, I offer them a deal. I say: I’ll admit that you were right about the potential for science studies to go horribly wrong and give fuel to deeply ignorant and/or reactionary people. And in return, you’ll admit that I was right about the culture wars, and right that the natural sciences would not be held harmless from the right-wing noise machine.

Ah..I said back in February 2007:

the modern global Right has operationalised postmodernism as a system of power

The Googles tells me I actually said it as far back as April, 2006 in a thread at Chris Lightfoot's.

Quoted without comment

"Richard sent me photos of his private parts before I'd even met him," says the redhead. "I thought this was very odd for a politician."

dis.gustin.ng

Yahoo! might be going to shut down del.icio.us, the link-sharing and bookmarking website it bought back in 2005 or thereabouts. (They might sell it, too.) This is awful - it's one of the most useful things on the Web and it's a key link in the production chain for everything I've written in the last six years for this blog, Fistful of Euros, Stable & Principled, Telco 2.0, and God knows what else. As well as providing a bookmarks file you can use anywhere, it also provided a huge quick-reference handbook of stuff other people found useful. During 2004, not only did I start using it, but this blog started to provide a list of RSS items from my account and several other blogs in the sidebar.

Yahoo! never did much with del.icio.us - they managed to retire the original domain name and redirect it to delicious.com, just as link-shorteners became fashionable, they made the web site more ugly, and they tried to impose some sort of horrible terms of service amendment by asking users to sign in with Yahoo! user IDs. I had nothing against this, but when I saw the lengthy new ToS document, I didn't bother reading it - it could only be evil, and therefore I refused. Bizarrely, they never even tried putting adverts on the home page, despite being the world leader in display-style Internet advertising, and neither did they ever try to get me or anyone else to subscribe, although they got me and hordes of others to pay for Flickr accounts.

You'll note that this doesn't include any new features or anything interesting at all. Also, they never did anything about spam accounts, so a lot of the social functionality became useless as "links for you" were always spam. However, they couldn't kill a basically useful product. If they sell it, though, it might survive or it might die - look what happened with Technorati.

A common theme about Yahoo! is that although the company drifts strategically, and every now and then gives the Chinese secret police confidential data about dissidents, the engineers are pretty good. True - they released lots of cool and useful stuff. Pipes, YQL, Term Extractor, YUI hackdays. Similarly, the Firefox extension for del.icio.us is very good indeed. It provides a full-text search over your tags, something the web site itself doesn't, and it can provide offline access to your bookmarks if you need that.

So here's a tip. The FF extension lets you work with your bookmarks offline and without signing in - so it must store them on your local machine. In fact, it uses quite a lot of Firefox's bookmarking functionality. And when you sign out, it asks if you want to keep your bookmarks in Firefox. You can add more bookmarks before you sign in again. Therefore, there's a way to slurp your data out of Yahoo! before it all gets deleted. Obviously they'll stop maintaining the plugin at some point. But once your data is stored as browser bookmarks, it can't be too far from being exported to an OPML file, at which point it could be imported anywhere else. Is dis.gustin.ng available?

repetition, repetition, repetition

This would do as a HOWTO start a war.

The document, drawn up by John Williams, press adviser to the then foreign secretary, Jack Straw, spells out ways to soften up the media, including "critics like the Guardian". Under the heading Not taking the UN route, Williams wrote: "Our argument should be narrow, and put with vigour – Iraq is uniquely dangerous."

In his memo, he said drafts of the dossier at the time had no "killer fact" which "proves" that "Saddam must be taken on now, or this or that weapon will be used against us." When Blair was launching the dossier three weeks later, he told parliament that intelligence had "established beyond doubt" that Iraq had WMDs.

Williams wrote: "Our target is not the argumentative interviewer or opinionated columnist, but the kind of people to whom ministerial interviews are a background hum on the car or kitchen radio. We must think Radio 5. Although the big Radio 4 programmes have to be done, we must not let them set themselves up as judge and jury."

Repeat, repeat, repeat. Here's Grant Shapps applying the same technique:

"We are still saying someone could have rent of £21,000-a-year paid by the taxpayer. How many could afford to pay that?"

Give him some credit - he even let on what he was doing.

psa

Anyone going?

lobby

Oh yes, so the IBM ManyEyes people fixed their computer.



I've got much more data now - I still need to do the four (key) departments that release in PDF format, and flush the existing stuff to replace the records with ones with standardised dates, but that should give you an idea. Hit the button in the visualisation with a network on it to redraw the force-directed graph.

in case of Russian invasion, break glass

Shock! NATO is secretly reviewing its plans to defend the Baltic states and Poland in the event of a war with Russia, a Wikileaked cable reveals. Interestingly enough, the details of this are already public - Gazeta Wyborcza published them on the 5th of November, appropriately enough, detailing that the NATO Response Force would be first in followed by up to 9 divisions from NATO states with the biggest contributions from the US, the UK, and Germany, using Gdynia and Swinoujscie as the main reinforcement ports and, of course, the NATO navies to clear the troops' way across the seas. On the 11th of November (again, pretty damn appropriate) Jean-Dominique Merchet's Secret Défense covered the story at its new home. All the leak really adds is that the planning exercise was in large part motivated by the need to get the new NATO members on side with the "reset" of US-Russia relations.

Korea: logistics, and propaganda

In my continuing fit of doom about Korea, this isn't helping - a US Military Sealift Command reserve freighter full of Maritime Prepositioning System kit is practising offloading it all in a Korean port. Supposedly, when they're finished they'll put it all back aboard and sail away. If you believe that, though...

The MPS is the US military's way of saving time shipping stuff around; they basically keep all the gear for an Army or Marine brigade packed in a ship somewhere strategic. Instant force, just add soldiers, who can come by air. This has a nasty logistics sound to it. Meanwhile, there is a real danger of war, says a Korean strategist from CSIS. Serious politicians are saying things like "reunification is drawing near" and that the Japanese military might be sent to look for people abducted by North Korea. That last one, from the Japanese prime minister, has an even nastier propaganda sound to it.

The Chinese envoy has been to Pyongyang, while the Foreign Ministry has had a pop at the US commander in chief in the Pacific, Admiral Mullen. This could be good news in the sense that Chinese engagement might warn off anyone from doing anything dangerous. The US Deputy Secretary of State is going to Beijing soon with a delegation, followed by Robert Gates next month.

keeping your leaks leaky

Image number four here has a certain additional spice, doesn't it? What a week. As well as WikiLeaks being the website they couldn't hang, 4chan became a geopolitical actor, thus fulfilling my prediction that in the future, trolls would be considered a strategic resource like oil. The BBC interviewed a builder from Leeds thinking he was a Liberal MP, but as it turned out, it didn't matter - the real MP did exactly what the fake one said. And of course there was the case of James Naughtie, demonstrating that the BBC really is the voice of the nation. Speak for England! Prince Charles got a nasty surprise from the students, which startled the mainstream media into actually covering the demonstrators' main grievance for once.

A lot has been written about how Wikileaks is staying online and I don't propose to add to it - this piece on CNET and this one from Renesys should tell you all you need. If you're looking for mirrors the list is currently here and there's a mirror of the mirror list here.

However, there are a couple of good technical points to be made here that I've not seen elsewhere.

First of all, Wikileaks is a website designed to be easily cloned. If you look at it, each page is a self-contained file with a flat URI - there are no signs of a query string, and each cable released has a unique ID within the same directory. This is important because it means that the process of creating a mirror is just to copy all the files into the /public_html/ directory of another web server. On a Unix-like system, it could be a one-liner command (the site doesn't actually let you do this the quickest way) - the utility wget is capable not only of traversing the directory and downloading all files, but also of changing links within them to point to the same filename in the target directory. The -m or --mirror option activates the options -N -r -l inf --no-remove-listing, which will in order ensure you only download material you haven't already loaded, that wget will get everything in the target directory or directories, and that any directory listings will be preserved. -p requires that everything needed to make up a page, such as a photo of Julian Assange, will be retrieved. -k turns on link conversion.

It might be enough to do: wget -N -p -k wikileaks.wherever /home/you/public_html/

So it's easy to create a mirror, and it's trivial to keep it up to date - you could just run your script as a cron job to grab whatever gets released every day. Anyone thinking of a really controversial Internet project should, IMHO, consider design-for-cloning to be a useful pattern. The clone count is now in the thousands.

Second technical point: what a horrible idea Mastercard SecureCode (and its pal Verified by Visa) is. I already hated it before this - it's a password, that should be a strong password because it's financial, but that I don't use that often and therefore can't remember, and it trains you to accept the idea of typing confidential information into a random web site you didn't ask for. Essentially all phishing requires you to type your bank details into something that you didn't ask for. Forcing the public to type their bank details into some random website they didn't ask for is howling insane. Right?

Also, the failure case is horrible - you get to reset the password by disclosing a whole lot of confidential information into the same random website you didn't ask for, so an attacker who managed to inject a frame into the original merchant's website could fake a failed payment and harvest all the information they would need to empty your bank account. And the service support when they imposed it on me was dire, especially as the SecureCode web site went down part way through the process.

But it's worse than that. An important part of the way card payments are accepted on the Web is that, as is also true in shops, you interact with the merchant, whose bank interacts with the wider infrastructure. So you should know who you're dealing with. Further, the bank should at least have some idea who its merchants are - they are customers after all - and restrict access to the system to them. And there's more than one bank that provides merchant service, so there are no single points of failure.

The SecureCode (and its Visa twin) websites, though, are customer-facing, so they have to accept traffic from the whole of the Internet. And all the Mastercard payments from the Web have to go via the SecureCode website. So you have a critical operational function, that is a single point of failure, and that is exposed to every last dog on the Internet. It's only surprising that somebody didn't bring it down earlier, especially when things like Bees with machine guns! are available.

not the Thursday music link

It's not Thursday, so it must be time for a not-Thursday music link. Special "one hand washes the other" edition.

Admin notice: Wikileaks

Need to know how to reach Wikileaks data? All the mirrors, alternative URIs, IP addresses, instructions, etc you could want are here. You might want to grab a local copy of the page itself, from this URI.

not about the Americans

The key fact to remember about the Wikileaks cable dump is this: it's not about the Americans. There's not been much in there that says something huge about US policy, which is why con-wissy types are so happy to deny it any significance. What there has been is something for everybody - a major purpose of diplomacy is to get political information, and leaking a ton of US diplomatic cables provides something for every host country to enjoy.

Here's Italy's delivery, for example. Not that anything about Berlusconi is shocking any more, but it's certainly interesting that he has a very personal special relationship with Russia. That throws an interesting light on the era of the "3Bs", Bush, Blair, and Berlusconi. Modern thinkers all, they also all thought they had special access to Russia.

Here's important confirmation that the Saudis are a major force pushing for military dictatorship in Pakistan, and probably in so far as they support Nawaz Sharif they are only using him as a pretext for military rule. This also tends to confirm that the Saudi influence sphere is a real factor in Afghanistan still.

Here's something for Belgium.

Here's something for us; Mervyn King was a key actor in insisting on cuts and a Con-Dem coalition, and specifically in terrorising Nick Clegg with "it's worse than we thought" stories.

Here's something else for us: there was a major ruck in the intelligence special relationship about the disclosure of imagery gathered by U2s operating from Akrotiri to the Lebanese, Israeli, and Turkish governments. It seems that the Brown government was trying to impose serious conditions on operations from Akrotiri.

Something for the Americans: Robert Gates is a major barrier to starting more wars.

This is interesting, although you've probably already read it.

And of course there's going to be a bank sometime in the near future.

Statistics efforts are coalescing here.

Leave your favourite leak in the comments.

Update: The Grauniad metadata file claims to contain the date, source, tags, and destination of each cable but the destinations are missing.

Moulded from birth. Forged by hatred. Honed in ignorance.

I'm beginning to worry seriously about Korea. There's the wikileaked cable suggesting that Chinese tolerance is running out. There's more recent confirmation. This after the initial non-reaction. Even if Peter Foster is right that the Chinese position hasn't changed that much, it still looks like something has changed in the deterrent balance.

On the other side, Joint STARS has been deployed. You know to start worrying when the ugly grey kit comes out. The US Navy has put 2 carriers and their reinforced task groups off Korea, including a ballistic-missile defence destroyer (USS Paul Hamilton) and four Ticonderoga class cruisers. In all there are something over 900 vertical launch missile tubes on surface ships alone, as well as 70 or so F/A-18s. The Jimmy Carter is in the area, but we don't know which other submarines are, or what percentage of the cruisers' VLS tubes are full of Tomahawks as opposed to SM-3 air defence missiles, Harpoon ship-to-ship missiles, or ASROC antisubmarine ones. And the US Navy has chosen this moment to send 30,000 tonnes of jet fuel to Korea. They do move this stuff around, but it's surely an odd moment to move the jet fuel if you weren't preparing for war. There are also two Marine groups in the area, so chuck in 16 Harriers and a bit shy of a brigade of Marines.

Unlike, say, Iran in 2007, US carrier availability is currently high. They have more ships to send if required.

The South Koreans have been as good as promising to retaliate hugely if there is another attack. They've sacked the defence minister and replaced him with a serving general. People are throwing D'Annunzio-style demonstrations for war. General upcranking is going on. So you can probably see why I'm worried. The whole Japanese navy is at sea, probably in part to get their Aegis missile destroyers deployed on their anti-missile radar picket patrol line early. And there's that unexpected uranium enrichment.

So it's probably high time to worry. Here's more worry: an excellent piece in the Small Wars Journal by US Army Colonel David S. Maxwell, on the problems of either occupying North Korea or just coping with the upshot of a collapse. I hadn't been aware of the degree to which the state ideology is based on the anti-Japanese guerrilla years. In comments, Maxwell says that what worries him more than the prospect of guerrilla war in post-North Korea is a warlord scenario, more Afghanistan than Iraq. Rather, it would be more like the worse-case scenarios for the end of the Soviet Union, given some of the kit that would available.

Maxwell's policy recommendation is to start at once with a propaganda drive to persuade the middle levels of the North Korean state not to go guerrilla and not to sell any highly enriched uranium they may have hanging around, and to come up with a plan for reunification led by Koreans and secured by all-party talks. That's all very sane, but it's not going to be of much help if someone fires artillery into Seoul tomorrow night. So from a British point of view, the best advice I could give would be "get on a plane and go and do an Attlee".

There are also PowerPoint slides to go with that. Hence the title - it could almost be a motto for the blog.

constant levels of outrage

In this thread at Charlie Stross's, it occurred to me that social outrage is a constant, but that its content is infinitely variable. You could almost call it the principle of the conservation of outrage - outrage can neither be created nor destroyed, but only transferred from one object to another. Addiction to drugs or drink has transitioned from being a sin to being a medical condition. Mental illness is doing something similar. Sexuality, for several whole generations, is a ship that has sailed.

But you'd be a fool to imagine that the outrage has gone anywhere. It worries me that, for example, the revival of what Paul Krugman calls New Austerian economics is really explained by the need to be outraged at somebody - the surplus of outrage has been directed at the victims of financial misfortune, who are always in ready supply. Of course, the fact that it went that way is interesting in itself and tells us something about the functions the pool of available outrage performs.

what could possibly go wrong?

So someone's trying to raise $150,000 to buy a satellite from the bankruptcy of TerreStar, in order to "Connect Everyone". I admire the aim, but I'm concerned that this is going to be a round of forgetting that a lot of perfectly good GSM operators are doing just that. Also, I can't find any reference to what they intend to use for the customer-premises equipment except that "we're building an open source low cost modem", which would be better if it came with a link to the source repo, right, or at least some requirements documentation? I'm also a little concerned that the team includes this guy:

Fabian is a NYC based Swiss wanna-be-entrepreneur who spends all his time trying to make meaningful connections between ourselves and business.

(and I chose charitably) but not anyone whose potted bio mentions being an RF engineer.

Actually, I think that it would be more worthwhile to start off with the low-cost open source satellite radio, as this may be the difficult bit and would be highly reuseable in other projects. A lot of Indian or African GSM people would find a cheap satellite radio very useful for their backhaul requirements. Depending on the spec it could be used with things like the amateur radio AMSATs, the transponders on the ISS, and the spare US Navy FLTSATCOMs. USRP is way too expensive at the moment (they cost more than a cheap netbook) so that one's out.

review of a movie that doesn't exist yet

I think most of my readers also read Patrick Lang's blog, but I think this guest post is the best thing yet written on the Taliban/SIS/McChrystal/Petraeus fake sheikh affair. Really, there's a great movie to be made here - the multiplicity of motives, the ironic contrast between the absurd story and the deadly serious interests and emotions that drive it forward, the eternal ambiguity of the relationship between the manipulator and the manipulated.

The ISI comes out of it as being dastardly clever, but in a deeply futile way. They succeed in preventing a dangerous outbreak of peace and sanity, but what have they gained? The wars grind on, the butcher's bill ticks up, the fantasy of a Pakistani empire of trucks and pipes across the Hindu Kush is as far away as ever, the Indians continue with their industrialisation across the other border.

The Americans come out of it as being well-meaning but naive. After all, they only get into this story because they want peace. So does the real Taliban leader. They both share a sort of big, stupid nobility.

The British do almost as badly as the ISI; not only do they end up being the dupes of the piece, they do so without the saving grace of having good intentions. They're as naive as the Americans but more underhanded. SIS gets involved purely as a way of sucking up to the Americans and putting one over its real enemies, GCHQ, Her Majesty's Forces, MI5, and the main-line Foreign Office diplomats. The Government is desperately keen on the project for similarly base reasons - to suck up to the Americans, to grab at an opportunity to solve its problem in Afghanistan, and of course to embarrass the Labour Party. Of course, it would have been a brilliant political fix had it come off - but the master manipulator is not Bismarck but William Hague.

The fake sheikh, meanwhile, is a classic example of the Pinocchio/Hauptmann von Kopenick theme - the puppet of bigger forces who becomes a power in his own right. Without his successful performance, of course, none of the many expectations curling around the tale have a hope of happening. His agency is real, and his character expands to fill the role. The fact that the whole project is an exercise in theatre is interesting in itself - a film within the film. The actors in the film are, of course, puppets of the script and the direction, and it is a work of fiction. The enduring purpose of the theatre and the cinema, however, is that works of fiction have real influence on their audiences. Like the fake sheikh.

After all, the grocer of Quetta (not a bad title) is the only character in the drama who successfully pursues his interests. He gets some interesting time off away from his bazaar stall, and even gets rich. You could play this as the ordinary man who succeeds in making fools of the powerful who insist on involving him in their schemes, or perhaps as a microcosm of all the people who are getting rich off the continued war, Mother Courage rather than Kopenick. Alternatively he could be killed off, casting the whole thing as an utterly bleak tragedy. However, arguably the classic in this vein is The Third Man and that sticks with the tragicomic.

the intersection of Wikileaks and Viktorfeed

If Bout agreed to cooperate with the Justice Department in a plea agreement sparing him life in prison, “we’ll certainly want to know more from him about the circumstances of those Iraq contracts,” said a law-enforcement official with detailed knowledge of the criminal case against Bout. The official was referring to large U.S. military contracts in 2003 and 2004 in which Bout’s cargo companies were used as subcontractors to deliver military supplies to U.S. forces in Iraq. “We’d want to understand if our officials knew exactly who they were dealing with. If U.S. officials knew they were dealing with Bout, that’s uncomfortable news.” Could U.S. officials face prosecution? “The conspiracy laws are broad,” the official said without elaboration.

Well, that would be fun. I'm also interested by the suggestion of a plea bargain. From here. This is something I don't think Irina from RFE/RL quite grasped.

There are some interesting Viktor Bout-related cables in the Wikileaks dump - this one suggests that part of his defence against extradition was to suggest he was in Thailand as part of a "government to government submarine deal". This one details a visit by US diplomats to Ras al-Khaimah airport, where they viewed some rotting Il-76s and made enquiries about what may have been Viktor's maintenance base. This was in any case around about the same time that the UAE gave the Antonov 12 operators the boot.

I'm surprised there aren't more, although there may be more releases yet. The relevant tags would be AE for the UAE and perhaps some others - NEA for the Near East desk is one.

Update: Moar!

False dichotomy watch

We've seen plenty of this sort of stuff before:

These networks such as the UnCut movement or the student movement with outstanding micro-organisations such as the UCL Occupation (which has received over 60,000 hits on its blog in a little under a week) who have so dynamically organised yesterday, today and going forward will inevitably be more flexible and effective than organisations with generic ‘leaderships’ such as major businesses, the police or even the National Union of Students.

Well, if your benchmark of effectiveness is the NUS... Mere snark, though. This particular Internet prairie fire does seem to be spreading nicely and doing more than tweeting. However, whenever someone starts going on like this, I do tend to suspect what they mean is "...more flexible and effective than organisations with people with funny accents who are train drivers an stuff".

I've said before, though, that I suspect that a lot of this network organising is structurally suited to negativity. Look at the 'baggers, for example. The classic examples political science types use, like open-source software projects, tend to be very different to the implementations in politics - rather than trying to recruit masses of people, they're usually driven by a hard core of the obsessed, or of people whose job it is. Order is difficult, mayhem is easy. Specifically, you can contribute significantly to mayhem by putting in an hour here or there.

On the other hand, though, it's not as if we're likely to run out of rage. This is 'bagger lesson one. Negative tactics and the expression of inchoate rage are not without value. Nigel Stanley gets it right - it's a false dichotomy. Getrennt marschieren, vereint schlagen, and we'll all get there in the end.

the lack of liberals

So I went to protest the London Lib Dems' conference, held in the late Blairite magnificence of Haverstock School NW3. Arriving punctually, what did I find?



No Liberals. In fact, not only had they vanished from the Haverstock, it turned out they'd given up on the whole idea of having a conference and punted it to February. Now you know why the party's colour is yellow. Not only were they afraid of the general public, but their reputation is now so toxic that nobody wants to give them house room.

Meanwhile, here's a prominent London Lib Dem in action.

"Right at this moment of financial peril to the nation is perhaps not the moment to introduce mandatory pay audits."

Just two years ago, the Liberal Democrat MP backed mandatory measures, saying: "A voluntary audit system for private industry is hardly worth the paper it's printed on. We need to know when the government actually plans to step in if progress isn't made."

The Liberal Democrat manifesto pledged to introduce fair-pay audits for all but the smallest companies.

Today Featherstone said: "It was a different world two years ago – financially and in terms of pressures on business. We are in a completely new landscape now … Much more of partnership working, no longer government dictates, this is absolutely the time to make voluntary pay-reporting work."

Two years ago? Two years ago was December, 2008 - hardly a moment of expansive prosperity. Banks were falling like seagull shit. People were trying to estimate what the absolute minimum level of cash balances was that could prevent the bankruptcy of the entire GM and Ford supply chains. Flocks of great empty ships were gathering in Falmouth harbour and off Singapore, forever delayed by the drying up of trade credit.

Also, check out that last sentence, a real classic of pseudo-Blair verbiage without enough verbs.

public service announcement

Sometimes, somebody does a web site listing protests against the cuts so you don't have to.

fail

Odd thought, working on NetworkX visualisations of lobbying activity. So much of whether the finished image is any use or not is an aesthetic decision, nothing to do with the actual graph metrics in any formal or programmatic way.

Randomly fiddle with the scaling factor, or increase the multiplier you apply to the nodes' link degree, and things look different - and possibly better. Also, more information may not be better - I can't help but think it was a lot easier to derive conclusions before I gave the various ministries node weights to reflect their relative power and divided the ministry weighting on each link by the number of organisations taking part in the meeting, which should be an improvement.

Also, after sweating quite a bit over this, does anyone know if NetworkX even lets you position nodes using polar coordinates? This looks useful.



This is far from ideal, but it should give some idea. This one is using the standard force-directed spring layout and just the data I have from June, with the nodes coloured by their degree in the chart and sized by their meeting count. Downing Street (the large purple plook) is fixed in place and the others adjusted. I rather like the fact that the lobbyists look like bacteria swarming into a cell, or possibly sperm trying to fertilise David Cameron.

An interview with Radio Free Europe

So Viktor Bout got extradited at last. As a result, I was interviewed by Radio Free Europe's Irina Lagunina. They wanted to hear about how the whole Viktor-blogging project got started - I told them about the Defence Energy Support Center files and the T-DODAACs and the like, and the plane spotter websites, and the fact I basically borrowed the idea from the people who were trying to monitor CIA rendition flights.

They wanted to hear about bloggers, and I thanked everyone who took part in Operation Firedump and made the point that all sorts of people across the ideological spectrum had taken part. They asked about my feelings about the extradition, and I said that I'd done the rejoicing when he was arrested and had since been mostly interested in seeing if there was any noticeable change in air movements through the UAE. What really cheered me was when the UAE government kicked out the An-12 operators.

They also wanted to know about the Russian government using VB's companies to move humanitarian aid. Specifically, they referred to something in Doug Farah's book that apparently quotes me on this. Unfortunately, my review copy never turned up so I'm completely in the dark as to what this might be. I said that given the numbers of heavy, ex-Soviet tactical airlifters that he controlled at the peak of his career, it was unlikely that many humanitarian agencies had been able to avoid occasionally dealing with him, and that they were probably right on balance to do the job rather than worry too much. If you need to move 40 tons of drinking water or flour labelled "Gift of the European Union" or whatever before people starve or get cholera, I'm not going to whine about it.

Recombination

In the last month or so I've begun adding blogs to my RSS queue again. I'm surprised by this; I didn't expect to find that the form still had so much energy. Perhaps this isn't so optimistic after all. Meanwhile, watch this space.s

le bateau ivre, port of call: Mogadishu

I wrote about this piece about brewing beer in southern Sudan, and incidentally creating a power station and a water works and a tiny industrial working class.

Well, SAB-Miller did a scenario-planning exercise about this sort of thing and came up with some truly odd answers. In the worst-case scenario:

a market with limited access to water and high energy costs, where people would migrate from areas of water shortage or turbulent weather

their proposed solution involved installing a brewery aboard a ship, so it could sail to wherever the thirsty masses had pitched up. Interestingly, as the story was released through the Grauniad's "Guardian Professional" advertorial division, this is also something SABMiller management was willing to pay good money to tell us about. Time was when big business wanted to keep its apocalyptic fantasies secret.

(Tagged "uncategorized", but should probably be under "uncategorizable".)

local news: power cuts

We keep having power cuts. In eight days, we had three, all of them between 5.30am and 6am, which all lasted most of the morning. I know exactly when they happen, because the smoke alarms start beeping and wake me up. My partner claims she was warmer on a demo than she was in the flat during no.2. As they seem to follow a pattern of happening at the same time every other day - just about when power demand starts to turn back up in earnest - I was wide awake at 0545 today waiting for the plaintive beeps. But no - looking at the chart, the ramp-up is later on a Sunday. Mind you, later in the day the lights flickered repeatedly for half a minute.

So I rang up UK Power Networks' (what used to be EDF Energy Networks, what used to be the London Electricity Board) press office and announced myself as a blogger. And the lights immediately went out...no, actually, they issued the following statement.

UK Power Networks would like to apologise to some customers in the
Holloway Road area of London who have experienced a series of power interruptions over recent weeks.

In the latest incident, power was interrupted to 327 customers at 5.54am today and restored to all customers affected by 11.20am.

The cause of the problem is believed to be an intermittent fault on an underground cable which our engineers are currently trying to trace. This can happen when the heat generated within the cable seals the damaged section, making it difficult to trace

Can it indeed. Let's hope they don't end up needing to do one of these.

hoisted from comments

From comments on this post, Against Viktorfeed:

*sigh* When is it going to get recognised that you can't spot an arms flight just by who used to own the aircraft. Jubba Airways isn't some unknown cargo entity - it's one of the main commercial passenger carriers into Mogadishu and other parts of Somalia. Are they a bit dodgy? Probably. Do they have dodgy planes? Yes. Does the fact that their dodgy planes may have been formerly owned by arms traffickers mean they're arms traffickers themselves? No.

To illustrate this whole *you can't spot an arms flight through the ownership of the plane* thing: Amnesty recently reported that arms were being flown from Bulgaria into Kigali in late 2008, just as Kigali was ramping up support for the CNDP in eastern DRC, on a *standard Air France passenger flight*. Which I'm pretty sure wouldn't have shown up on the 'Viktorfeed'.

www.amnesty.org/en/library/info/ACT30/015/2010/en

The key question is always who owns the cargo, and (for charter flight), who is *chartering* the plane. Not who owns or even operates the plane.

I've never been entirely confident on this point. Thoughts?

killing data.gov.uk, and thinking aloud about mapping the lobbysphere

So the government thinks this is clever. They also think it constitutes a "searchable online database". It is not searchable, nor is it a database. It is a collection of links to department web sites, some of which actually lead to useful documents, some of which lead to utterly pointless intermediary pages, some of which lead to documents in a sensible format, some of which lead to documents in pointlessly wrong formats, and some of which lead to PDF files. It provides no clue how often this data will be released or when or where. The URIs sometimes suggest that they might be predictable, sometimes they are just random alphanumeric sequences. Basically, what he said.

Meanwhile, very few of these documents have made it onto data.gov.uk, the government's data web site (pro-tip: the hint is in the name) which provides all that stuff out of the box. This is not just disappointing - this is actively regressive. Is it official policy to break data.gov.uk?

Anyway, I've been fiddling with NetworkX, the network-graph library for Python from Los Alamos National Laboratory. Sadly it doesn't have a method networkx.earth_shattering_kaboom(). I've eventually decided that the visualisation paradigm I wanted was looking me in the eye all along - kc claffy's Skitter graph, used by CAIDA to map the Internet's peering architecture.

The algorithm is fairly simple - nodes are located in terms of polar coordinates, on a circular chart. In the original, the concept is that you are observing from directly above the north or south pole. This gives you two dimensions - angle, or in other words, how far around the circle you are, and radius, your location on the line from the centre to the edge. claffy et al used the longitude of each Autonomous System's WHOIS technical contact address for their angles, and the inverse of each node's linkdegree for the radius. Linkdegree is a metric of how deeply connected any given object in the network is; taking the inverse (i.e 1/linkdegree) meant that the more of it you have, the more central you are.

My plan is to define the centre as the prime minister, and to plot the ministries at the distance from him given by the weighting I'd already given them - basically, the prime minister is 1 and the rest are progressively less starting with Treasury and working down - and an arbitrary angle. I'm going to sort them by weight, so that importance falls in a clockwise direction, for purely aesthetic reasons. Then, I'll plot the lobbies. As they are the unknown factors, they all start with the same, small node weighting. Then add the edges - the links - which will have weights given by the weight of the ministry involved divided by the number of outside participants at that meeting, so a one-on-one is the ideal case.

When we come to draw the graph, the lobbies will be plotted with the mean angle of the ministries they have meetings with, and the inverse of their linkdegree, with the node size scaled by its traffic. Traffic in this case basically means how many meetings it had. Therefore, it should be possible to see both how effective the lobbying was, from the node's position, and how much effort was expended, from its size. The edges will be coloured by date, so as to make change over time visible. If it works, I'll also provide some time series things - unfortunately, if the release frequency is quarterly, as it may be, this won't be very useful.

Anyway, as always, to-do no.1 is to finish the web scraping - the Internet's dishes. And think of a snappy name.

CSM: a useful tip-off

Following up on this post, I get e-mail from Matthew Turner, who points to an explanation. Communications & Strategy Management Ltd. does indeed have a web site of sorts, at csm-limited.com. It appears to be a political consulting/lobbying firm that does the Tories' demographics/voter-database work, which basically consists of the Richard Murphy mentioned in the past post. The Web site was registered in April, around the same time as the company, through an American proxy registrar. Although the company is registered in somebody's house in Stratford-upon-Avon, its operational address places it at:

The Manor
Coleshill Manor Campus
Birmingham Road
Coleshill
West Midlands
B46 1DL

Coleshill Manor is the headquarters of a variety of Tory organisations, notably the West Midlands Tory party itself and the commercial entity known variously as Constituency Campaigning Services Ltd. or Coleshill Campaigning Services Ltd, which essentially provided the services of the party's central institutions to individual Conservative Associations against a contribution, organising things like junk mail, phone calls, and other jewels of our society.

The purpose of this structure was to render CCS a commercial company in the meaning of the Political Parties, Elections, and Referendums Act - if you gave money to an individual MP's campaign, who then used it to pay for the services of a political consultancy, you would fall under the provisions of the Act and your donation would be both recorded and limited. If, however, you gave it to a commercial company which then sold its services to the campaign, not so much. Here's the Other Taxpayers' Alliance.

Various investigations have occurred into Coleshill - notably that the Tories use large amounts of office space there paid for by various millionaires via the companies based in the building. CCS alone received £1m in donations before the 2005 election. There's also a major call centre there.

Rounding it up, we have here part of a political campaign structure designed to accept external donations outside the legal framework, that is plugged into the Whips' Office at the far end. Neat!

I didn't think I'd still be blogging this six years later

Well, this looks pretty ugly. I have a question. We know that unofficial, non-doctrinal training material was being circulated around the joint services intelligence centre in Chicksands in 2003-2005 - there's an interesting quote about it in the Guardian piece here:

Any public inquiry into the activities of the JFIT would be expected to examine the extent to which it was supervised by military lawyers. It is now known that at least some of the training material used by F Branch at Chicksands between 2003 and 2005 escaped the scrutiny of the training centre's in-house lawyer, Brigadier David Yates, who told the Mousa inquiry that he did not "have the capacity" to check it.

This is important because, as the piece also points out, most of the interrogators were reservists. They would have gone first of all to the Chilwell mobilisation centre to do fitness tests, draw additional kit, get their vaccinations, complete their admin, and to do refresher courses on things like first aid, marksmanship, and anti-terrorist precautions. Then, later, they would have gone for a period of pre-deployment training, which would concentrate on preparing for their specific role in Iraq, before finally shipping out via South Cerney and RAF Brize Norton. It would make sense if the reservist intelligence people were sent to their trade's headquarters, which for most of them would also be their unit's peacetime depot, for their specialised pre-deployment course. (I think I have the process right, but several readers can correct me.)

Now, we also know that the Americans began with the torture in 2002, and that Major-General Geoffrey Miller was transferred from Guantanamo to Iraq with his infamous directive to "Gitmo-ize" the detention camps in the summer of 2003. So, where did this documentation come from?

Better procurement

Via this Wired piece, I see that the US Special Forces are giving up on the various "Land Warrior" projects to load down soldiers with specialised electronic gear - some readers may remember the BBC documentary on one of MOD's efforts in this line and the image of a file of soldiers attempting to move stealthily across Salisbury Plain with enormous objects like big plastic mushrooms lashed on top of their helmets. Instead, they've issued an RFP for software applications running on Android-based devices to achieve the same aims.

The RFP is here, and what an RFP it is too. It is, among other things, clear, and clearly drafted by someone with substantial technical competence. You try finding many contract managers who know what RFC 5740 specifies. From a technical point of view it's pretty demanding: multiple video streams, reliable delivery, in an environment of restricted connectivity - rather you than me. (This 2008 RFP may be part of the explanation.)

However, it's also true that a valid strategy for delivering high bandwidth traffic like video is to shift it from the classic unicast (i.e. one stream per user, from the same source) to a broadcast or multicast route. I wonder if whoever answers it will be an early user of the Stream Control Transmission Protocol, which among other things allows multi-homing at the connection level, so that the same connection between two logical addresses can involve more than one physical source?

This also reminds me of the mid-2000s IP Multimedia Subsystem hype - collaborative whiteboarding was an example use case that came up in literally every vendor presentation, and they would occasionally do demonstrations at conferences, which always turned out to be really awful. Sometimes this was because the server was back in Finland and the endpoints were roaming on a Singaporean operator's 3G network, with hilarious latency consequences. Sometimes it was because IMS just wasn't a very good idea.

not the Thursday music link

It's not Thursday, so it must be time for non-Thursday music link. You know the rules - only covers or remixes are acceptable. Here's DatA's mix of Danger's 19H11:

Sometimes, FaxYourMP Just Isn't Enough

I don't quite know what to make of this:

Q What other sites, remember any particular internet sites you looked at?

A When I was doing research about MPs, I looked at one called theyworkforyou.com and I think another one was called publicwhips [publicwhip.org.uk].

Q So, have you carried out any research to ... about Stephen Timms.

A Yeah, on ... I looked up, I found, I Googled him, I found out he had a website, I found a page about him on theyworkforyou.com ... if you follow that link it shows information about how he voted on different things related to the Iraq war and the build up towards it. I found out that ... he very strongly agreed with the invasion of Iraq and they said very strongly because they worked out all his votes for everything related to that and it came up to something like 99.9% support or something like that.

Q How does that make you feel?

A That made me feel angry because the whole Iraq war is just based on lies and he just voted strongly for everything as though he had no mercy. As though he felt no doubts that what he was doing was right, even though it was such an arrogant thing to do and I just felt like if he could treat the Iraqi people so mercilessly, then why should I show him any mercy?

Q What, what makes you think that it's your place to go and stab him?..

Exactly what is Communication Strategy & Management Ltd?

So I scraped the government meetings data and rescraped it as one-edge-per-row. And then, obviously enough, I tidied it up in a spreadsheet and threw it at ManyEyes as a proof-of-concept. Unfortunately, IBM's otherwise great web site is broken, so although it will preview the network diagram, it fails to actually publish it to the web. Oh well, ticket opened, etc.

Anyway, I was able to demonstrate the thing to Daniel Davies on my laptop, on the bar of the Nelson's Retreat pub in Old Street. This impressed him excessively. Specifically, we were interested by an odd outlier on the chart. Before I get into that, though, here are some preliminary findings.

1 - Clegg's Diary

At first sight, Nick Clegg appears to be unexpectedly influential. His calender included meetings with NATO, the World Bank, the Metropolitan Police, the Gates Foundation, and oddly enough, Lord Robertson of Port Ellen. Not only that, he had one-to-one meetings with all of them. However, he also got The Elders (i.e. retired politicos playing at shop) and the leader of the Canadian opposition, one Michael Ignatieff, Esq. God help us, is Clegg turning out to be a Decent?

2 - Dave from PR's surprisingly dull world

The Prime Minister, no less, meets with some remarkably dull people. In fact, he met quite a lot of people who you'd expect to be left to flunkies while leaving quite a lot of important people to Nick Clegg. He did get BP, Shell, Pfizer, Rupert Murdoch, the TUC general secretary, and Ratan Tata (twice!) as one-on-ones, but he also met a surprising number of minor worthies from Cornwall and vacuous photocalls with people from Facebook.

3 - Francis Maude, evil genius of the coalition

Secretary of State for the Cabinet Office and Paymaster-General, Francis Maude MP, is the surprise hit, as far as I can make out. He seems to have a special responsibility for anything that smacks of privatisation - therefore, the monetary value of meeting him is probably high. Of course, if your evil genius is Francis Mediocritus, you've got problems. No wonder we're in such a mess. All these points are also true of Oliver Letwin.

4 - Communication and Strategy Management Ltd

This is our far outlier. Some of the least significant people on the chart appear to be government whips, which is obviously an artefact of the data set. The data release does not cover intra-governmental or parliamentary meetings, nor does it cover diplomatic activity. Whips, of course, are a key institution in the political system. Given their special role with regard to both the government and parliament, it's not surprising that they appear to be sheltered from external lobbying - access to the Whips' Office would be such a powerful and occult influence that it must be held closely.

So what on earth is Communication and Strategy Management Ltd., a company which had one-on-one access to the Government Chief Whip, the Rt. Hon. Patrick McLoughlin MP, and which according to Companies House was founded on the 11th of April? It has no web site or perceptible public presence. It is located in what looks like a private house, here, not far from Stratford upon Avon:


View Larger Map

Evidently the hub of political influence, but those are the facts. The directors are Elizabeth Ann Murphy and Richard Anthony Cubitt Murphy*, ignoring a company-formation agent who was a director for one day when setting up the company. It's not as if C&SM Ltd is a constituent of McLoughlin's - he's MP for the Derbyshire Dales. Actually, either the directors are related or else there was a cockup, as Murphy's name on the books was amended from Bromley the day after the company was formed and both were born in 1963. The Companies House filing* doesn't give any other information - accounts aren't due for a while - except that the one share issued is held by Norman Younger, who is a partner in the company formation service that was used.

Anyway, the next stop is to learn how this works and put up a nice little dashboard page to help watch the lobbysphere. I'd be happier doing something with python - such as nodebox - but the diagram is already too big to be useful without interactivity, and you can't stick a NodeBox window in a web page.

*Not the Richard Murphy, who is too young.
*WebCheck - it's not an ugly website, it's a way of life...

the House of Lords is not just stranger than you think..

This has me thinking one thing - TheyWorkForYou needs to integrate the text-mining tool researchers used to estimate the point at which Agatha Christie's Alzheimer's disease set in by analysing her books. We could call it WhatHaveTheyForgotten? Or perhaps HowDrunkIsYourMP? Jakob Whitfield pointed me to the original paper, here. It doesn't seem that complicated, although I have a couple of methodological questions - for a start, are there enough politicians with a track record in Hansard long enough to provide a good baseline for time-series analysis?

Instead, we could do a synchronic comparison and look at which politicians seem to be diverging from the average. Of course, some might object that this would be a comparison against a highly unusual and self-selected sample. Another objection might be that the whole idea is simply too cruel. Yet a further objection might be the classic one that there are some things man should not know.

Update: Implemented!

so you want to know who's lobbying?

So I was moaning about the Government and the release of lists of meetings with external organisations. Well, what about some action? I've written a scraper that aggregates all the existing data and sticks it in a sinister database. At the moment, the Cabinet Office, DEFRA, and the Scottish Office have coughed up the files and are all included. I'm going to add more departments as they become available. Scraperwiki seems to be a bit sporky this evening; the whole thing has run to completion, although for some reason you can't see all the data, and I've added the link to the UK Open Government Licence twice without it being saved.

A couple of technical points: to start with, I'd like to thank this guy who wrote an alternative to Python's csv module's wonderful DictReader class. DictReader is lovely because it lets you open a CSV (or indeed anything-separated value) file and keep the rows of data linked to their column headers as python dictionaries. Unfortunately, it won't handle Unicode or anything except UTF-8. Which is a problem if you're Chinese, or as it happens, if you want to read documents produced by Windows users, as they tend to use Really Strange characters for trivial things like apostrophes (\x92, can you believe it?). This, however, will process whatever encoding you give it and will still give you dictionaries. Thanks!

I also discovered something fun about ScraperWiki itself. It's surprisingly clever under the bonnet - I was aware of various smart things with User Mode Linux and heavy parallelisation going on, and I recall Julian Todd talking about his plans to design a new scaling architecture based on lots of SQLite databases in RAM as read-slaves. Anyway, I had kept some URIs in a list, which I was then planning to loop through, retrieving the data and processing it. One of the URIs, DEFRA's, ended like so: oct2010.csv.

Obviously, I liked the idea of generating the filename programmatically, in the expectation of future releases of data. For some reason, though, the parsing kept failing as soon as it got to the DEFRA page. Weirdly, what was happening was that the parser would run into a chunk of HTML and, obviously enough, choke. But there was no HTML. Bizarre. Eventually I thought to look in the Scraperwiki debugger's Sources tab. To my considerable surprise, all the URIs were being loaded at once, in parallel, before the processing of the first file began. This was entirely different from the flow of control in my program, and as a result, the filename was not generated before the HTTP request was issued. DEFRA was 404ing, and because the csv module takes a file object rather than a string, I was using urllib.urlretrieve() rather than urlopen() or scraperwiki.scrape(). Hence the HTML.

So, Scraperwiki does a silent optimisation and loads all your data sources in parallel on startup. Quite cool, but I have to say that some documentation of this feature might be nice, as multithreading is usually meant to be voluntary:-)

TODO, meanwhile: at the moment, all the organisations that take part in a given meeting are lumped together. I want to break them out, to facilitate counting the heaviest lobbyists and feeding visualisation tools. Also, I'd like to clean up the "Purpose of meeting" field so as to be able to do the same for subject matter.

Update: Slight return. Fixed the unique keying requirement by creating a unique meeting id.

Update Update: Would anyone prefer if the data output schema was link-oriented rather than event-oriented? At the moment it preserves the underlying structure of the data releases, which have one row for each meeting. It might be better, when I come to expand the Name of External Org field, to have a row per relationship, i.e. edge in the network. This would help a lot with visualisation. In that case, I'd create a non-unique meeting identifier to make it possible to recreate the meetings by grouping on that key, and instead have a unique constraint on an identifier for each link.

Update Update Update: So I made one.

loose ends tied up

Remember this post from back in September, 2005? (And what a fine month that was.) A Lebanese court has sentenced several people involved to terms of imprisonment, including highly dubious aviation identity Imad Saba and two of his managers. The aircraft captain, who survived, and Saba's representative, who also survived after haranguing the other pilot (who didn't) into taking off with the aircraft several tonnes overloaded and out of balance, were also convicted in absentia and remain on the run.

Shibboleth

Somewhat in the spirit of this XKCD cartoon. There are memes that allow us to tell if other people are likely to be worth speaking to - like biomarkers for language. For example, someone who disbelieves in plate tectonics probably has a wide range of other weird beliefs.

The latest one of these I've noticed is the idea that you have to be unemployed to get housing benefit in the UK. Knobber after horrible knobber shows up talking about claimants "in houses working people couldn't dream of". Wrong, wrong, wrong. So unfortunately, I've come to the parting of the ways with Tim Garton Ash. In an otherwise mostly sensible column:

It's surely not right that people can be worse off if they choose to work than they would be on welfare; or that people on inflated housing benefits make rented accommodation in some areas unaffordable for the working poor.

You don't have to be unemployed to get housing benefit, Tim. I predict bad things.

party like it's 2008

A bit of Viktorfeed. Scheduled for 1845Z, there's a flight from Dubai to Mogadishu under ICAO code JBW708. JBW? That's Jubba Airways, described by Aerotransport.org as "Formed 24/4/1998 by Canadian (Calgary) interests and the Southern Somali Business Groups (50%), in association with Phoenix Aviation. Started operations on 28/5/98."

The aircraft roster consists of two Boeing 737s, one of which belonged to both Phoenix Aviation/AVE and Kam Air, the other to Kam Air twice and East Air, a Tajik company started by Eastok Air, an operation banned in the EU since July, 2007 and which, interestingly, leased aircraft to Iraqi Airways.

brief Iraq post

Weird news from Iraq - apparently one of Sadr's conditions for returning to Iraqi politics is that Allawi and the SIIC are included. There's a turn-up for you. It does sound like the Iranians are the main actors here, and the point of including Allawi is to get minimal consent from the Sunni. Some day there's going to be a good book written on the politics between the US and Iran during the Iraq War.

Afghan links

Here at the Low Expectations Journal we've been rather optimistic recently about Afghanistan - at least relative to our expectations. This week, there's been a piece in the Washington Post that completely contradicts this. However, I would point out that this may not be as significant as all that:

Among the troubling findings is that Taliban commanders who are captured or killed are often replaced in a matter of days.

Abraham Lincoln said that he could make a brigadier into a general in three minutes, but a hundred and ten horses were difficult to replace. Isn't this the whole "Al-Qa'ida's Number Three" argument again, just with the sign reversed to justify pessimism rather than optimism? Surely the question is whether they are finding good replacements. An optimistic report is here. Exum wonders how the paper manages to run two entirely contradictory stories on successive days.

On the other hand, it's not the only case of ending up like the man who has two watches and no longer knows what the time is. Here we have two widely divergent opinions on a basic fact like the rate at which IEDs are discovered. You may recall that "Population Density of Afghanistan: Experts Differ" was actually an accurate headline for a while.

Worryingly, Jeremy Scahill reckons that the negotiations are being sabotaged by the old game of reporting whoever you don't like to the Americans as a Taliban.

Meetings

Mark Ballard of Computer Weekly is trying to get the details of government meetings with the IT industry, and struggling. Among other things, this seems to be yet another use case for an enduring Freedom of Information Act request. It's also one of the reasons why I like the idea of a central contacts register. Back at OpenTech 2009 I said to Tom Watson MP, just after he resigned as a minister, that it wasn't just useful for citizens to be able to find out who officials were contacting - the government itself might benefit from keeping track of who was lobbying it, maintaining a common line-to-take across different departments, and the like. Hey, even the lobbyists might benefit from knowing who else was lobbying.

Of course, there's an argument that the government quite likes having pathological relationships with its suppliers. But that's one of the points where as soon as you get radical enough to understand the situation, you're also too cynical to do anything about it. Watson's been campaigning about this, and the Cabinet Office recently released some data. With the embarrassing bits taken out.

The bulk of it is here, it looks like they're planning to split the disclosure between departments as this only covers ministers in the Cabinet Office (i.e. the PM, DPM, Secretary for the Cabinet Office, Leader of the Commons and the whips). It's also on data.gov.uk but it's going to need reparsing. At least it's not a PDF. It's a bit thin, presumably because the bulk of meetings with external organisations go via officials or bag carrier MPs - DEFRA's is rather chewier.

There's also a list of special advisers by department and salary, which may be handy, and has already informed me that one of William Hague's advisers is none other than Richard Littlejohn's son.

this is what "crowdsourcing" looks like

Shouldn't Crowdsourced New York Apartment Pushing Limits actually be an Onion headline? Either that or the core of a new ResPublica/New School Network collaboration as the Big Society's contribution to solving the housing crisis. We'll crowdsource it! If everyone brings a brick, we'll have...a pile of as many bricks as users who actually bothered showing up, that the two people who actually care about the project will have to use.



(Yes, I know this should be on Stable & Principled, but I'm trying to keep that blog Terribly Serious.)

Meanwhile, genuinely serious and interesting points on the same theme are made in this excellent piece on Park Hill in Sheffield and its redevelopment. It's not as smash-mouth as Owen Hatherley would likely be, but it also makes the point that letting the squatters have their way with it was tried, effectively, and a lot of the work required on the building was basically making good the results. It also strikes me as a good point that it's not, in fact, easier to run away from the scene of a crime on an access deck. Of course, the real point here is that as the society that built it crumbled, they stopped providing proper investigative policing to the people in it and started treating them as the object of mass public-order policing.

convergent mayors

Is Boris Johnson the right's Ken Livingstone? It came to mind as a result of his unexpectedly strong remarks about housing benefit. A lot of Tories disbelieve that Johnson is genuinely committed to the party. Ken spent large chunks of his career either at odds with the Labour Party leadership or outside the party. Johnson is now reprising Livingstone's role in protesting against Thatcher, while also reprising his role a second time around as an alternative version of a government he's fundamentally sympathetic to.

A lot of people remarked that Ken Livingstone, as mayor, was remarkably keen on facilitating the City's interests for someone whose staff included John Ross. Johnson is heavily reliant on the remaining ex-Livingstone officials to keep City Hall's basic functions going. Both of them put a lot of effort into maintaining a public image that is almost a caricature of their party - the whole tedious Shower Jobby act, vs. all the stuff about newts and public transport.

Of course, this overstates a bit. But I do think there's a significant truth here, and I suspect that future Mayors of London are going to have more in common with Ken and the Jobby than they will with the Prime Minister of the day. They will tend to be noisy and brash, given to ranting, and drawn back towards consensus within London by the administrative realities. There is famously no Democratic or Republican way to collect the garbage*. However, they will also tend to operate in permanent tension with the national government up river. This is an expression of the structural factors - you can't position yourself politically by replacing the Underground with a network of cable cars over the streets or abolishing school, so you've got to do so by picking fights with Westminster.

Given that, you're either going to be in the role of unofficial opposition leader, or else aligned with the government of the day's rebels, whoever they may be. Also, it seems that you'll probably end up being a couple of points to the left of your party either rhetorically or operationally. Despite all the yelling, Ken Livingstone was basically following the Blairite "let the bankers rip and then do some redistribution" plan, but with more aggression and nous. It's also true of the Jobby - for all the bullshit, he's not actually changed that much, which puts him some way left of the cuts consensus. Interestingly, this also seems to be true of Bertrand Delanoe and Klaus Wowereit, and perhaps also Michael Bloomberg.

* This argument may no longer seem as convincing as it once did, as there are probably Republicans who want to abolish rubbish collection...

the Pentagon unchecks "internet connection sharing"

The rate of intrusion attempts on US government networks has fallen this year. Obviously, this is going to be a data series dominated by the spikes, so a good botnet between now and Christmas could change that. But it's a nice correction to the constant "cyberwar" bollocks. Also, check out the hilariously .com boom era graphic - streams of ones and zeros! Scary words, like ENCRYPTION SOFTWARE, in fake-LED display fonts!

a slight return from 2004...

Two things: brief piece about the Toyota Hilux, preferred transport to the world guerrilla, and a fatwa against mobile money transfer. You know you've made it when you've been fatwa'd.

More seriously, this huge Guardian piece on Iranian policy in Iraq is well worth reading. It's interesting, to say the least, that the people Sadr wanted to see as guarantors of Iranian good faith were Hezbollah - it would seem they've got a foreign policy these days. Also, there's a sort of disguised alliance between the Iranians and the US. The Americans fought hard to stabilise al-Maliki's government and to build it up as a credible force. It's hard to imagine they really want rid of him now.

And there's this:

It is understood that the full withdrawal of all US troops after a security agreement signed between Baghdad and Washington at the end of 2011 was also sought by Sheikh Nasrallah.

"Maliki told them he will never extend, or renew [any bases] or give any facilities to the Americans or British after the end of next year," a source said....US officials have strongly suggested they would scale back their involvement in Iraq if the Sadrists, who have been a key foe throughout the years of war, were to emerge as a significant player in any government.

But it's their policy to leave:

On the 2011 December withdrawal date, the official said: "Any follow-up engagement with Iraq in relation to troops would be at the request of the government of Iraq. There are no plans to keep troops after December 2011. We are drawing down and all will be out of Iraq."

The piece seems to be heavily influenced by sources in Al Baathi Allawi's entourage. You do wonder if Allawi was only fetched out of the deep freeze in order to press Maliki into making a deal with the Sadrists.

Why are these scumbags so scummy?

John "War Nerd" Dolan got a job, as a lecturer at the American University of Iraq. Hilarity ensued. You bet. It's a tale of un-fantastic right-wing academics, a kind of glaring dullness, a total lack of character, and an endless supply of raw cash. It so happens that John needed that more than anything else, so good luck to him. Read the whole thing - what stands out is the vast gap between the neo-con obsession with The Western Canon! Classicism! Principle! Courage! and the petty, provincial, small-mindedness that people like Joshua "Not The Blogger" Marshall practice in their lives. It's not even the incompetence. It's the style that gives them away.

The other interesting thing in the piece is John Agresto's role. Again and again, he turns up wondering why a string of horrible political thugs treated him with disrespect. Lynne Cheney, his old boss, seems to have been a really awful human being close up. Who knew? But somehow, it never crosses his mind to wonder why this keeps happening every time he associates with the Cheneys or Bill Bennett or some other horrific political gargoyle. It's....full of bastards, just this particular astronaut isn't going to get out of the ship.

I also loved the notion of a neo-conservative as someone who got mugged by reality and now never goes into town for fear of running into reality again. A lesser writer would say that he started carrying a gun in order to shoot reality. However, that would imply some kind of grand, tragic struggle against brute fate. You can't have tragedy without dignity, and that's one thing the administration of the American University of Iraq doesn't have.

This reminded me of two things, or rather the other way around. If you want Mitt Romney to speak, you've got to take a bulk order for his booky wook. Hence the book is a bestseller (for whatever that means in today's book trade). Similarly, 'bagger Sharron Angle's campaign raised $14m and paid $12m right back to the political consultants who organised the donation drive.

The other thing was this documentary series on YouTube about Americans and steroids. Two points come to mind - the enduring role of the quack, and a sort of grinding optimism. And this quote: "Everyone wants to be a monster."

A critical point, though - I'm fairly sure the sheaf of documents one of the doctors waves while reading out a list of horrible side effects that turn out to relate to vitamin C is from an open-access "adverse event reporting system", which basically gathers anything anyone anywhere feels inclined to report. They aren't verified in any way. Anti-vaccine people often abuse this.

Fortunately, someone's done the actual journalism and documented that the ties between multilevel marketing, quackery, and extreme-right politics aren't just style, they're organisational and financial. It goes back a while, too.

new emerging threats

Swinging off my Stable & Principled contributions, this is ridiculously great. Who the hell is this menacing new competitor at the intersection of naval shipbuilding, MoD criticism, and obsession with mobile computing devices?

Oddly enough, I keep adding new blogs to my RSS queue at the moment. The scene burns on. On which theme, time for some music. The Beat vs. Smokey Robinson, so that's not only a cover but also 2-Tone vs. northern soul.

all around the world I've been supporting team blogging proposals

I have mostly been blogging the cuts at Stable & Principled this week. Somebody had to.

How the Government produced a comprehensive spending review without mentioning the monster housing bubble or using the word recession more than once. Smoke and mirrors, and housing benefit cuts. Has Dave from PR got his jets mixed up? Why the Army has quite enough helicopters, thank you, and past statements are no longer operative. Carefully taking no decisions at all on defence. Can the Shirley Porter strategy save the Shower Jobby?