The intersection of electronic warfare and mall management

Here's something interesting. You may remember this story from back in November about the CIA spy network in Lebanon that met at a Pizza Hut they codenamed PIZZA, and which was rolled up by a joint Hezbollah-Lebanese military intelligence investigation. The key detail is as follows:

U.S. officials also denied the source's allegation that the former CIA station chief dismissed an email warning that some of his Lebanese agents could be identified because they used cellphones to call only their CIA handlers and no one else.
...
Lebanon's security service was able to isolate the CIA informants by analyzing cellphone company records that showed the numbers called, duration of each call and location of the phone at the time of the call, the source said.

Using billing and cell tower records for hundreds of thousands of phone numbers, software can isolate cellphones used near an embassy, or used only once, or only on quick calls. The process quickly narrows down a small group of phones that a security service can monitor.

If the top paragraph is true, it would have been catastrophically ill-advised. Even somebody special, like a CIA agent under diplomatic cover, has a relatively large number of weak ties to normal people. This is the reverse of the small-world principle, and is a consequence of the fact that the great majority of people are real human beings rather than important persons. As a result, things like STELLAR WIND, the illegal Bush-era effort to analyse the whole pile of call-detail records at AT&T and Verizon in the hope that this would find terrorists, face a sort of Bayesian doom. We've gone over this over and over again.

However, phone numbers that only talk to special people are obviously suspicious. Most numbers with a neighbourhood length of 1 will be things like machine-to-machine SIMs in vending machines and cash points, but once you'd filtered those out, the remaining pool of possibles would be quite small. It is intuitive to think of avoiding surveillance, or keeping a low profile, but what is required is actually camouflage rather than concealment.

There are more direct methods - which is where electronic warfare and shopping mall management intersect.

Path Intelligence, a Portsmouth-based startup, will install a network of IMSI-catchers, devices which act as a mobile base station in order to identify mobile phones nearby, in your shopping centre so as to collect really detailed footfall information.

Similarly, you could plant such a device near that Pizza Hut to capture which phones passed by and when, and which ones usually coincided. Alternatively, you could use it in a targeted mode to confirm the presence or absence of a known device. Which makes me wonder about the famous Hezbollah telecoms network, and whether it was intended at least in part to be an electronic-intelligence network - as after all, nothing would be a better cover for a huge network of fake mobile base stations than a network of real ones.

Meanwhile, this year's CCC (like last year's) was just stuffed with GSM exploits. It really is beginning to look a lot like "time we retired that network".

Protection....

Update: I originally didn't want to publish this because I didn't think it was good enough, but I hit the wrong button. Anyway, Alistair Morgan read it and thinks one of the premises of the whole thing is wrong. Namely, the weapons were going in the same direction as the drugs, not the other way around. Well, at least the story moved on a bit, but this renders mostly useless a whole additional post I put together from reading a lot of crazy-but-interesting stuff out of the bottom of the Internet. Also, despite the Jessie J reference there's better music at the bottom if you get that far.

So, Alistair Morgan's twitter feed frequently hints at "cocaine, weapons, and Ireland" as well as police corruption as being factors involved in the case of his brother, Daniel Morgan, the private detective murdered in 1987, probably by people who were since employed by News International. It's often been said that Morgan was on the point of publishing some sort of huge revelation when he was killed, but nobody knows what it was beyond his brother's hints based on what the police told him at the time.

Since the eruption of the phone-hacking scandal, a number of sidelights have come up which linked the News of the World, its cadre of ex-police gumshoes, and its contacts inside the police force. Notably, it seems to have spied on the former Army intelligence agent-handler, Ian Hurst, on an NGO, British-Irish Rights Watch (because documents of theirs were on Hurst's computer when they hacked it), and perhaps on the chief of police, Sir Philip Orde. It would have been hard for people working for the press not to have covered at least one Northern Irish story in the last 20-odd years simply because it was such a news staple, but it's worth noting their interest.

The War Economy of Northern Ireland

So, what might link Morgan, cocaine, weapons, Ireland, and policemen? There are some fairly well-known stylised facts or stereotypes about the economy of the Troubles. The IRA mostly funded itself from money collected in the United States, from bank robberies, and from unofficial taxes it collected in the North. It also got contributions from friendly countries, specifically Libya. The Loyalists didn't have a reliable source of their own money abroad like NorAid, and so specialised in protection and drugs. Both sides also got involved in smuggling across the border as a commercial exercise.

That's a glib summary 'graf; obviously, I collect a revolutionary tax for the struggle, you impose fines on drug dealers and dishonestly stick to some of the money, and they are merely thugs operating a protection racket. Traditionally, both Sinn Fein and the British tended to stereotype the Loyalists as basically criminal and the IRA as proper insurgents - there may be some truth in there, but the distinction is one of emphasis and degree and also of propaganda rather than of kind.

Having obtained money, they both needed to convert some of it into arms. The IRA got a famous delivery in the 80s from Libya in its role as Secret Santa, and also often bought guns in the US over the counter and smuggled them back. I don't know how well characterised the sources of Loyalist arms are, which of course gives me license to speculate.

Permanently Operating Factors

Now for the cocaine, which has often been known to land in bulk quantities on the wilder, less populated bits of the Atlantic coast that also offer good harbours. This is a rare combination, as people live near ports. Two of the best bits on that score are northwest Spain and southwest Ireland. Having landed, you can move it on anywhere in the UK-Ireland common travel area without much more trouble. Since the creation of the Schengen area, Galicia is even better for this because there is such a choice of markets you can reach without a customs inspection. But in 1987 this was an un-fact, so you might as well go to Ireland.

This transit trade had important consequences - notably the rise of Martin "The General" Cahill, the assassination of Veronica Guerin, and probably a substantial chunk of the Irish property bubble via the laundering of profits and also by the boost to those ol' animal spirits the drug provides.

Imagine, then, that an important criminal actor supplying the London market with cocaine also had access to a reliable surplus of weapons. There is the potential for trade here.

However, it's not that simple - the famous Libyan shipment would have fit in a couple of shipping containers, and it kept the IRA going up until peace was signed, with a fair bit left over to be buried in concrete by the international commissioners on decommissioning. It is very unlikely that any plausible flow of arms to Northern Ireland would have paid for the flow of cocaine into the South-East.

We Don't Need Your Money, Money, Money, We Just Wanna Make The World Dance...


There's something else going on - Diego Gambetta would have already pointed out that you need to understand the trade in protection. To sell protection, you need weapons, which are the capital equipment of the business of private protection. In so far as the buyers in the UK were paying in guns as well as cash, they were arguably expressing a protector-protectee relationship. While on our territory, we protect you, and license you to provide protection. This was also reciprocated. In accepting them, were the sellers of the cocaine undertaking to protect it in transit on their own territory?

Another way of looking at this, which Gambetta would also approve of, would be in terms of costly signalling. Being both a supplier and a protector is a powerful position, but it might be worth letting the other side have it as a guarantee or hostage, to signal that you didn't intend to break the agreement and deal with some other supplier. This makes even more sense given that you still have a regular supply of guns you could cut off or use against them, and therefore both parties have something to lose.

Now, Gambetta's work mostly deals with Sicily, where a very important protection supplier has often been irrelevant. London is a very different society from this point of view. Whatever you think of the police, you can't just ignore them as a factor. In some other societies, the police might be protection consumers, but here, police corruption usually takes the form of policemen selling protection. (In a sense, the more effective the police, the more tempting this will be. Nothing sells like the good stuff.)

So, gazing down on this complex, neo-medieval exchange of cash, credit, and protection, there is a sort of Sun King whose permission is required for any protection contract to be signed. It's like a feudal society. My liege lord is only so, because he is the King's subject, and the King at least theoretically owes duties to the Emperor, or later, directly to God. Our buyer is in a position to offer protection for his end of the business because he enjoys protection supplied by the police.

Who were the recipients, the sellers? They might have been drug dealers who needed to buy protection from one or other paramilitary group. They might have been drug dealers who wanted to build up enough arms that they could stop buying protection, or rather, change protector. Or they might have been paramilitaries who sold protection to the drugs trade. The distinction is surprisingly unimportant.

So, to put the pieces together, there was some group of South-East London villains importing cocaine from transit providers in Ireland, who were also exporting weapons in the opposite direction as part of an exchange of protection for their common business. This required buying protection from the police. Where did the weapons come from? And why is News International involved?

it flies! well, that's what it was bloody well designed to do

The question isn't so much "did Eric Pickles eat all the pies?", it's "who paid for the pies, and how many did he declare in the register of members' interests?". TBIJ is on an absolute tear on Tory lobbying stories at the moment, and the combination of photo and caption for the Eric Pickles one is masterly.

But this story reveals more than it says. So, four cabinet ministers accepted donations to their private offices since May, 2010. Those would be William Hague, George Osborne, Liam Fox, and Michael Gove, or to put it another way, most of Atlantic Bridge and the core of the neo-conservative group within the Conservative Party. I do not think this is a coincidence.

Curiously, it seems that if you get donations to your private office you don't also get them to your constituency party branch and vice versa, with the exceptions of George Osborne and Michael Gove, who would have more jam on it, wouldn't they?

Pickles, for his part, received zero, which makes perfect sense. You can't eat money, and as for spending it on unofficial advisers, that only makes sense if you ever take advice from other people and the Bradford food-mountain has always known he's right.

Meanwhile, Lord Astor of Hever turns up as a trustee of the Bridge and an pal of the Werritty-funding SAS walt, Iraq contract hunter, and intimate of mercenaries Tim Spicer and Anthony Buckingham.

I think I've said before that Astor of Hever came out of the Lobster Project proof of concept script as being a surprisingly important gatekeeper - although in himself, he isn't a major node, people who meet him also tend to get one-to-one meetings with the most important ministers. His weighted network degree, a measurement of how many links in the lobbying network involve him adjusted for how many people took part in the meetings, is 0.125, pretty low (78th in the league), but his gatekeepership metric is 2.533, the third highest overall and the very highest score for a minister with UK-wide responsibility. (I discount the gatekeepership numbers for Scottish and Welsh ministers, as their role is partly to represent Scottish and Welsh interests and they are structurally heavily lobbied.)

The gatekeepership metric in Lobster is the ratio of the average weighted network degree of those who lobbied a given minister to the average of all lobbies, to the ratio of that minister's network degree to that of an average minister, thus capturing the degree to which meeting that minister was associated with meeting more or less important ones while taking into account the fact that some ministerial jobs are more important than others. If it is greater than 1, you're likely to get a boost, if less, you're being heard out.

A limitation is that obviously, the Prime Minister can't help you meet a more important minister, so it doesn't yet deal with the situation where you meet the PM to get your word across and are then referred to a junior minister for action. I accept that this is a problem, although you would expect that it is easier to lobby the small fry, so the metric is nevertheless useful. However, at a network degree of 0.125, Lord Astor is not affected by this phenomenon.

OK, so we have a prediction - other ministers involved with the Werritty/Fox/Atlantic Bridge case will demonstrate unusually high gatekeepership. Step forward Gerald Howarth MP, Minister for International Security Strategy, who achieves a gatekeepership of 2.36, the fourth highest overall and the second highest UK-wide, on a network degree of 1.2. That's some pull, when you note that he's a significant node in terms of quantity.

Lobster detected a sinister network of influence! How awesome is that?

Adventures in radial graph visualisation - Sindy edition

The Independent on Sunday has another really excellent piece on Liam Fox including a very good network visualisation. See the power of the radial graph paradigm, and understand why I want Lobster to look like that! kc claffy wasn't wrong.

Liam Fox: Not Fit For Purpose

OK, so "Not All That" Foxy Liam Fox is in trouble.

"He is an odd bloke," said one fellow minister. "He has fingers in so many pies that you kind of think one of them will land him in trouble somewhere along the line."

Another Tory MP said Fox's tendency to name-drop and brag about his close friendships with Republicans in the US, media magnates such as David and Frederick Barclay (owners of the Daily Telegraph), and his endless globe-trotting, even before he entered the cabinet, has made many bristle and help explain why he has plenty of enemies in the Tory party and in Whitehall. "I think you either roll with the bluster or find it repellent," said a Tory MP.

Ah, one of them. Anyway. Part of the problem is this famous meeting where his bestie Adam Werritty just happened to turn up. What was on offer? Well, a product called Cellcrypt, whose makers were trying to sell it to the MoD to stop evilly-disposed persons from eavesdropping on British soldiers' phone calls back to the UK. (Note: this is going to be long. Technical summary: voice encryption apps for GSM-style mobile networks can guarantee that your call will not be overheard, but not that your presence cannot be monitored, and not necessarily that the parties to your calls cannot be identified.)

Back in the early days of Iraq, the CPA permitted one mobile phone operator in each of its three zones to set up. The British zone, CPA-South/Multinational Division South-East, let the Kuwaiti national telco, MTC (now Zain and busy running Mo Ibrahim's old Celtel business into the ground) set up there with a partner some of us may have heard of. It's from Newbury and it's not a pub or an estate agency and its logo is a big red comma...funny how Vodafone never talked that particular investment up, innit? Anyway. Later the Iraqi government did a major tender for permanent licences and Orascom got most of it, but that's another story.

One thing that did happen was that soldiers took their mobiles with them to Iraq, and some of them pretty soon realised that buying a local SIM card in the bazaar was much cheaper than making roaming calls back to the UK. Either way, lots of +44 numbers started showing up in their VLR, the big database that keeps track of where phones are in a GSM network so it can route incoming calls.

Pretty soon someone who - presumably - worked for the MTC-Voda affiliate and whose purposes were not entirely aligned with Iraq The Model realised that you could use the VLR to follow the Brits (and the Yanks and the Danes and the Dutchmen and Kiwis and all sorts of contractors) around. Not only that, you could ring up their families in the UK and make threats with the benefit of apparently supernatural knowledge.

This obviously wasn't ideal. Efforts were made to mitigate the problem; soldiers were discouraged from using local GSM networks, more computers and public phones were made available. The eventual solution, though, was to get some nice new ruggedised small-cell systems from companies like Private Mobile Networks Ltd., which basically pack a small base station and a base station controller and a satellite backhaul terminal into a tough plastic box of a suitably military colour. You open it up, unfold the antenna, turn on the power, and complete some configuration options. It logs into the mobile operator who's providing service to you via the satellite link.

Now, because radio signals like all radiation lose intensity with the inverse square of the distance, you'll be vastly louder than everyone else. So any mobile phone nearby will roam onto your private mobile network and will be in the UK for mobile phone purposes, a bit like the shipping container that's technically in Egypt at the end of Four Lions. And none of this will touch any other mobile network that might be operating in your area. Obviously you can also use these powers for evil, by snarfing up everyone else's traffic, and don't for a moment think this isn't also done by so-called IMSI catchers.

You're not meant to do this, normally, because you probably don't have a licence to use the GSM, GSM/PCS, or UMTS frequencies. But, as the founder of PMN Ltd. told a colleague of mine, the answer to that is "we've got bigger tanks".

So, where were we? Well, the problem with trying to do...something...with Cellcrypt is that it doesn't actually solve this problem, because the problem wasn't originally that the other side could listen to the content of voice calls. Like all telecoms interception stories, it was about the traffic analysis, not the content. Actually, they probably could listen in as well because some of the Iraqi and Afghan operators may not have been using up-to-date or even *any* air interface encryption.

But if you're going to fix this with an encryption app like Cellcrypt, you've got to make sure that every soldier (and sailor and diplomat and journo and MoD civilian) installs it, it works on all the phones, and you absolutely can't make calls without it. Also, you've got to make sure all the people they talk to install it.

And the enemy can still follow you because the phones are still registering in the VLRs!

So, there's not much point relying on OTA voice encryption to solve a problem that's got nothing to do with the voice bearer channel. However, bringing your own small cell network certainly does solve the problem, elegantly, and without needing to worry about what kind of phones people bring along or buy locally.

And the military surely understand this, as by the time of the famous meeting, they'd already started deploying them. Also, back when this was a big problem, 19 year-old riflemen usually didn't have the sort of phones that would run a big hefty application like Cellcrypt, which also uses the mobile data link and therefore would give them four figure phone bills.

To sum up, Werritty was helping someone market gear that the MoD didn't need, that was hopelessly unfit for purpose, wouldn't actually do what the MoD wanted, and would cost individual soldiers a fortune, by providing privileged access to the Secretary of State for Defence.

discovering the axis of barking

The Grauniad Dabatlog has produced a rather fancy network visualisation of the sources cited in Anders Behring Breivik's personal manifesto/horse-shit compendium. This is great as I now don't need to worry that I perhaps should have made one. It's very pretty and you can click on stuff, and see that some of the sources are thinktanks and some of them are newspapers, and well, it's very pretty and you can click on stuff. It also comes with a piece by Andrew Brown reprising his "Don't be beastly to the creationists!" shtick but with Melanie Phillips, for some reason.

Unfortunately it's almost completely intransparent, and gives little indication of what data is being visualised or on what basis, and there is really no obvious conclusion to draw from it. But did I mention pretty and click? If forced to take a view, I would reckon that the underlying data is probably a matrix of which sources appear together with others and the layout algo is a force-directed graph (aka the default in pretty much any visualisation toolkit), probably weighted by appearance count. There's some sort of proprietary metric called "linkfluence" which appears to be given by(indegree/outdegree)*len(neighbourhood) or words to that effect.

As a result, the only information I got from it was that he linked to Wikipedia, the BBC, and big news sites a lot. Well yes; Wikipedia, bbc.co.uk, etc, generate a hell of a lot of web pages and people read them a lot. Obviously, to say the least, you need to normalise the data with regard to sheer bulk, or you'd end up concluding that Google (or Bing or Yahoo) was his inspiration because he did a lot of web searches, or that he was a normal man twisted by SMTP because he used e-mail.

In fact, I thought they actually did that until I realised that RSS.org is about the other RSS, the Indian extreme-right movement, not the popular Internet syndication standard. Harrowell fail. Anyway, it does show up rather nicely that the groups "European nationalists", "Counter-Jihad", and "American Right-Wing" overlap. However, I feel there's something missing in the characterisation of MEMRI and various other sites as just "Think Tanks" as if they were just like, say, IPPR.

Also, an emergent property of the data is that there is an Axis of Barking running vertically through it: the nearer you are to the top of the diagram, the more extreme and crazy. MEMRI, FrontPage, Gates of Vienna, Melanie Phillips are near the top; the Wikipedia article on the Russo-Turkish War of 1878 is at the bottom. And the MSM is somewhere in the middle. (Although I do wonder if they allocated the sources to groups before or after running the force-directed graph.)

It seems to be one of those command the exciting world of social media with just one click! things.

Anyway, upshot. I want to avoid Project Lobster producing a diagram like this one. It's too impressionistic and fluffy and reliant on basically aesthetic reasoning. (I think we've had this point before.) Of course, that's partly the difference between the underlying data sets; it was at least thinkable if unlikely that there would be no grouping in Breivik's sources, while presumably political lobbying is nonrandom and subject to intelligent design.

Elsewhere, a reader passed this along which I need to actually watch (isn't video time consuming?). There's a shindig in Warsaw in late October. And I want this on a T-shirt.

your call could not be connected - please check the number and try again

The Obscurer has possibly the first intelligent article on the whole "turn off their Facebook! that'll learn em!" furore. Notably, they interviewed one-man UK mobile industry institution Mike Short. Go, read, and up your clue. I especially liked that the piece provided some facts about the 7th July 2005 terrorist incident and the mobile networks.

There is only one reported case of a UK network being closed by police. During the 7/7 London suicide bombings, O2 phone masts in a 1km square area around Aldgate tube station were disconnected for a number of hours.

Police have an emergency power to order masts to be put out of action known as MTPAS – Mobile Telecommunication Privileged Access Scheme. The move has to be approved by Gold Command, by the officers in highest authority during a major incident, and is designed to restrict all but emergency service phones with registered sim cards from making calls. But a shutdown can have dangerous knock-on effects. Short says that phones within the Aldgate zone automatically sought a signal from live masts outside it, overloading them and causing a network failure that rippled out "like a whirlpool".

On the day, other networks were simply overloaded as Londoners sought reassurance and information. Vodafone alone experienced a 250% increase in call volumes

MTPAS is the GSM-land equivalent of the old fixed phone Telephone Preference Scheme (not to be confused with the new one that blocks cold-callers), which permitted The Authorities to turn off between 1% and 90% of phone lines in order to let official traffic through. As far as I know, the Met never asked for it and it was City of London Police who initiated it without asking the Met or anyone else, and in fact O2 UK's network had been keeping up with demand up to that point, before the closure caused the cascade failure Short describes.

The significance of O2 is that it used to be "Surf the Net, Surf the BT Cellnet" and some residual gaullist/spook reflex in the government tried to keep official phones on what was then one of two British-owned networks.

Anyway, this weekend seems to have the theme "The Intersection of Charlie Stross and the August 2011 Riots". Charlie's talk at USENIX is sensibly sceptical about some tech dreams as they apply to networking.

This leaves aside a third model, that of peer to peer mesh networks with no actual cellcos as such – just lots of folks with cheap routers. I’m going to provisionally assume that this one is hopelessly utopian, a GNU vision of telecommunications that can’t actually work on a large scale because the routing topology of such a network is going to be nightmarish unless there are some fat fibre optic cables somewhere in the picture. It’s kind of a shame – I’d love to see a future where no corporate behemoths have a choke hold on the internet – but humans aren’t evenly distributed geographically.

Especially as the theoretical maximum bandwidth of one fibre is about the same as the entire radio spectrum. And the point about routing table size and complexity is a very good one, especially as it's assumed that the routers aren't CRS-1s but rather Linksys fifty quidders or mobile phones.

However, one thing the liberation technologists should take away from the riots is that you shouldn't get hung up on bandwidth. It's great to be able to post the photos on Flickr, but it's more useful to have your own secure voice and messaging. When the Egyptian government relented on its GSM cut-off, the Egyptian Twitter feeds lit up with calls for more people to this or that exit of Tahrir Square or medical supplies to the clinic or (and I remember this) that a lost child was waiting at the press tent.

It was what NANOG users would call operational content. There was of course no need whatsoever for it to go via a Bay Area website - all Twitter provided was the one-to-many element, very important, and the publicity on the Web. The latter is a nice-to-have feature, the former, critical. Text, or even voice, is not a high bandwidth application and doesn't necessarily need access to the global Internet.

So yes - perhaps there is in fact quite a bit of angular momentum to be had in a mobile mesh-WLAN client as an instrument of democracy, as long as you're willing to accept that it's not the sort of thing that can be exclusive to people who agree with you. But then, that's the test of whether or not you actually believe in democracy.

Something else, between Charlie's USENIX talk and the riots. Isn't one of the biggest disappointments, from a police point of view, the performance of CCTV? No doubt it will help put some of the rioters in jail. But it didn't prevent the riots and neither did it seem to help quell them much. It's possible that the whole idea that potential surveillance (like the original panopticon) is a policing influence isn't as strong as it's made out to be.

Another point; not all crimes are punished or even taken notice of. This is obvious. Less obvious is that the degree to which the police ignore crime is an important political fact. Is it possible that CCTV, by forcing them to make at least a token response to everything that passes in camera range, actually contributed to using up the police strength? In a riot, the police aim is to demonstrate public, mass control. They are usually willing to ignore quite a lot of individual criminality in the process. It's possible that surveillance culture and technology are opposed to strategy.

taliban mobile!

Did you know ISAF has been carrying out air missions to destroy Taliban radio towers? You do now, thanks to Thomas Wiegold's blog. Specifically, Task Force Palehorse includes UAE Apache Longbow attack helicopters and American Kiowa Warrior reconnaissance helicopters, plus (according to comments) German ELINT specialists. And they go out and identify Taliban radio networks, and kill them.

There's much interesting stuff for German-speakers in comments, notably that the technologies include old fashioned VHF, pirate GSM, and possibly other systems as well, that the relays are often solar-powered, and that the Taliban are significant users of IMSI-catchers - fake GSM/UMTS base stations used to monitor mobile phone activity.

So are the Germans, in order to prevent leakage from their own camps. The British have been using ruggedised, highly portable small cells for some time to stop soldiers using the Afghan GSM networks, for fear both of security leaks and also that (as in Iraq) their relatives in the UK might get nasty phone calls.

the giant squid meets the criminal octopus as the last Tory sings his swan song

Peter Oborne's piece on post-Murdoch Britain is interesting, although mostly for the sheer otherness of his thinking. He's at least got the good sense or moral minima required to end up on the right side of the debate, but he gets there through some truly odd reasoning. Can anyone remember even one instance when any of the News International outlets ever ran an editorial arguing that a republican form of government was desirable? But he ascribes "a powerful republican agenda" to Rupert Murdoch.

I'd argue that Murdoch has a powerful Republican agenda, as in the American political party, but not a republican one. The only newspaper that professes republicanism is the Guardian, or perhaps the Andersonstown News, which isn't the same kind of republicanism and is interested in a different republic. The only way I can get sense out of it is to assume that he's talking about Australia, which seems a bit of a distant concern. Further, David Miliband knows Polly Toynbee socially, and we're asked to believe that this is worth mentioning in the same breath as the whole Murdoch system of government, as he describes it. Also, he seems to think Michael Gove, Eurabia-pushing ex-News International executive, is still a credible cabinet minister.

It's like reading a leader written by a giant squid. It's intelligence. But not as we know it.

Of course, he's right that what has been revealed is a system, a sort of parallel government. One thing Oborne is extremely unlikely to ever say is that there was a qualitative shift in the late 2000s in how it worked, although I think he's vaguely aware of it.

Labour leaders in the 1990s reacted to the power of the Murdoch system by trying to accommodate themselves to it. This is as good a way as any to define "The Project" - an effort to accommodate the party to the realities of the Murdoch system and to manage a transactional, bargaining relationship with it. The key figures in this effort - Peter Mandelson, Jonathan Powell, and Alistair Campbell - prided themselves precisely on their ability to manage the relationship, to negotiate a degree of freedom of action. The terms of business between NI and Labour can easily be criticised - would a Martian journalist dropped on a street corner between 1997 and 2007 have noticed that the Sun supposedly "supported Labour"? I think not. The paper didn't actually call for a vote against, but it did pour abuse on Labour ministers, opposed basically all its policies except invading Iraq, and offered the Tories a sympathetic hearing.

By the end of the Blair era, this relationship was strained - the original breakdown might be traced to Iraq, in fact, and the exit of Alistair Campbell from No.10. In fact, what was under way was more fundamental than that. We might call it The Project 2.0.

This would involve the Conservatives rather than Labour, and critically would go much further than the original Project. Rather than a transactional, bargaining relationship with the Tories, mediated by powerful media managers belonging to the party, the Project 2.0 foresaw something very different. Whereas Alistair Campbell's role was as "Emily" Blair's bulldog, guarding the gate at the interface between the prime minister's office and the press, alternately wagging and barking as seemed expedient, this new project would integrate News International's men into the machinery of government.

Part of the legacy the original Project was the enlarged status of the government's press officers at every level. In fact, it is unfair to blame this on New Labour. It is a great strategic trend that has been going for many years. Sir Bernard Ingham is not, I think, remembered for presenting an even-handed account of the Thatcher government's record or acting with total equanimity towards critics and sycophants alike. Sir Gerald Kaufman, Joe Haines, Philip De Zueleuta - they all served their prime ministers in the dark art of propaganda and earned a reputation of sinister efficiency and great influence. Arguably, this goes all the way back to Lloyd George's secretariat. Labour's innovation was to bring in outsider professionals, and much else was up to the force of personality of those involved.

Another important trend was their integration with the core executive, the hard centre of the state centred around the prime minister, the Cabinet Office, the Treasury, the intelligence services, and the Ministry of Defence. At the top level, these strands all wind together in the prime minister's office (which, I notice, has grown a domain name, pmo.gov.uk, recently). As a result, the prime minister's official spokesman now has a sort of parallel management-information system that runs into the departments via their press officers, in much the same way the Treasury's MIS extends into the departments through the system of public-service agreements. The two phenomena are quite closely linked, in fact - departments' priorities are fixed through the comprehensive spending round and the PSAs, with reference to the political/press strategy defined on the PMOS's network. Then, their success or failure is monitored via the Treasury reporting system and the No.10 policy unit, and the political response to it coordinated through the PMOS net.

One Blairite contribution to this which is unique was its militarisation. The Blair years turned out to be ones of war, and the emerging press-management network was heavily used in support of the wars, with the result that intelligence was increasingly redistributed from the intelligence-administrative complex into the press-political management system. At the same time, the military's public affairs function was integrated into the system, with a common line for the civilian and uniformed spokesmen. Alistair Campbell's invention of the Coalition Information Centre (a still under-reported creation) gave this an international dimension.

The Project 2.0 consists, then, of reversing the process. Rather than the politicians selecting press managers to control the interfaces between the political management network and the media, News International selected them and made recommendations to politicians, who incorporated them into key locations in the system - the very top at No.10, the Metropolitan Police, ACPO, and to be frank, who knows where else? We know that George Osborne recommended Andy Coulson to David Cameron, and that Rebekah Brooks thinks she recommended him to Osborne. Dick Fedorcio claims he can't remember who recommended Neil Wallis to him, but he did admit that the motivation was to influence No.10 Downing Street, rather than to influence the press. (And who reckons Tom Watson wouldn't have asked him if Brooks made the recommendation if he didn't know the answer?)

Where Campbell bargained with NI and Mandelson cooperated with them, the situation in May, 2010 was more like Field-Marshal Montgomery's remark that "I banned all talk of Army Co-Operation. There were not two plans, Army and Air, but one, Army-Air. When you are one entity you cannot cooperate."

Now, Oborne is as I said, sort of aware of all this. A few weeks ago he surfaced the fact that there is an identifiable Murdoch caucus in the Tory Party, around George Osborne and Michael Gove, opposed to a Telegraph one. It is common knowledge that David Cameron's first priority as Tory leader was to short-circuit the party's internal procedures in order to flush in a lot more new candidates, the so-called A-list. One wonders if the list was pre-approved. And there seems to be an identifiable historical break point - when Michael Howard ran for prime minister in 2005, he brought in a pretty ugly character to run a pretty ugly campaign, Lynton Crosby, but at least he appears to have picked him himself.

My final question, then - with the dilution of the Tory Party with new candidates, and the integration of Murdoch's political officers into key nodes, does David Cameron actually exist politically? Is he, y'know, a thing? What fuckery is this?

Vaz discovers the network

Dick Fedorcio is taking a little trip to Westminster to be quizzed. By Keith Vaz's committee. How times change - Keith Vaz as the standard bearer of public integrity. It's got to be more convincing to get your second chance after 10 years in the wilderness though.

Meanwhile, has anyone else noticed that the "#AskEdM" hashtag is a shamelessly obvious exercise in trolling? Think about it - how many people who aren't either political obsessives following up a story or just looking for a fight, or else full-time rightwing howler monkeys, ever hang out in Paul Staines's comments threads with all the coup fantasies and racebaiting and ZaNuLiebour and EUSSR? Nobody. Really.

So why not lure the torysphere out in the open, somewhere the majority can actually see them behaving in this horrible way? Like Twitter.

And if they notice it, they'll get angry and look an even more repellent herd of shit-smeared zombies. It's brilliant. As a side benefit, if Edelmans have managed to rig some network of semiautomated talking-points distribution bots, they're bound to show up there so they can be identified. And if you were looking for a list of horrible spamming arsewits, Mike Gigglers trying to be hipsters but stuck at the brown-dwarf stage, ZaNuLiebour dittoheads, and other membrane fauna...well, here's your chance to populate your blacklist. I added dozens. It was a lot of fun.

The only annoying thing is that as far as I know, although Twitter has "lists" that let you group other users and subscribe to their collected output (yeah, like XMPP Collection Nodes) it doesn't have the inverse operation. So you can't easily replicate the Team Cymru Bogons BGP feed and automate the process, even if we've already got a serviceable darknet.

Elsewhere, I put together a quick network diagram of the top Met-NI interactions. Nothing very surprising except that the commissioner is always sought after, and that John Yates appears to have an independent following with them (surprise!). Of course, your man Dick was probably at all the meetings in this chart.



Finally, some music.

You (Feat. Nina Sky) - Creep (Hannah Holland Remix) by Hannah Holland

a city wired for sound

Am I right in thinking that Andy Hayman's testimony yesterday fingered Met press chief Dick Fedorcio? Hayman admitted he'd regularly had dinner with News International executives while he was meant to be investigating them. He mentioned that he had done this in the company of the head of communications of the Met, presumably with his approval, although Hayman was also acting in his capacity as ACPO media lead.

Fedorcio has had the same job since 1997. He was named by Nick Davies as having been present in the meeting where the Met demanded to know why Dave Cook was being followed by News International private detectives, and apparently intervened with senior police officers to get them to go easy on NI. Surely the guy in charge of police-press-political relations is a key figure in a scandal that's all about relations between the press, the police, and politics?

Like the key News International men, Alex Marunchak and Greg Miskiw, there's no sign of him. The Home Affairs committee, and indeed anyone else who wants the truth about this, must call Fedorcio without delay. Oh, and is Greg Miskiw in the UK?

Second point. Yesterday's New York Times claims that Miskiw and others on the NOTW were able to locate mobile phones by paying £500 a shot to a corrupt police officer. That is to say, this policeman had access to the lawful intercept systems that are part of all GSM and UMTS cellular networks, or at least he could task people who did. ETSI Specification 01.33 defines this as a standard element of all GSM networks and the corresponding 3GPP TS 33.106 does so for UMTS ones.

If this is so, they could certainly also get pen-register information - lists of calls to and from given phone numbers - and even tap the calls themselves.

This is a massive violation of the UK's critical national infrastructure security, of the Regulation of Investigatory Powers Act, and of the Data Protection Act. News International, their police contact, and the police force responsible (not necessarily the Met) should all be prosecuted.

There is an urgent need to audit the lawful interception systems' logs, among other things to find out if there are other unauthorised users out there. International standards foresee a detailed audit trail as part of these systems in order to preserve the legal chain-of-evidence. If the Interception Request message was submitted in proper form from the police to the telcos, the operators are legally in the clear, but if I was in charge of their network security I'd suspend processing the requests until such an audit was carried out as we now know that an unknown but significant percentage of them are illegal.

Thank fuck we didn't build that giant national ID card database.

Third point. Not that anyone will answer this, but were any of the Prime Minister's designated deputies for nuclear retaliation subject to illegal telecoms surveillance?

Fourth point. Circling back to the Defence Vetting Agency and Andy Coulson, the vetting procedure as described on the DVA Web site states that in some cases, the decision may be taken to issue a security clearance subject to risk management measures taken by the department involved. In these cases, the DVA will disclose information to the sponsoring department that it would usually keep confidential. Did they make such a recommendation to the Prime Minister's office, and if so, what was the information?

a few thoughts on the exciting world of social media

So how do you get from Shoreditch to the South Bank? Well, as Tom from Boris Watch pointed out, you take a number 243 bus. Or you wait 15 years - one way or another. Or you practice, baby. Or you get a haircut. Anyway, so much for taking the opportunity to reuse what I thought was quite a good joke. If journalism is the first draft of history - an ill-thought out exercise in speed-typing riddled with basic factual errors that aspires one day to be edited into ideological propaganda - and blogging is the first draft of journalism, then Twitter is evidently a lot of old cock.

I wanted to flag this article on how the tube map is a lie. Apparently, people tend to underestimate how long their routes will take when using the Beck map, not just tourists but natives (or as we call 'em round here, slightly less recent immigrants) too.

I think the explanation, and the fix, are as follows. Beck's key insight was to analogise the system to an electrical circuit and draw a schematic diagram of it, showing the key components (stations) and how they interconnect. However, the problem is that we expect a map to show geographical information whereas the Beck map shows logical information.

The fix is, I think, to adopt cold potato routing. The Internet normally uses hot-potato routing - networks hand over traffic to each other at the first possible interconnection point, trying to get rid of it as soon as possible. This has some advantages - it avoids the situation where traffic for Network B is routed into Network A, carried across it, and then carried back towards its source because the furthest interconnect point has failed.

Occasionally this causes a pathological equilibrium - consider a network with customers on both coasts of the US and interconnections with another similar network. Under hot-potato routing, traffic from a customer of A on the East Coast to a customer of B on the West Coast could get routed into B on the East Coast, back out to A, and eventually into B on the West Coast.

Cold-potato routing is the opposite. You carry the traffic as far towards its destination as you can yourself, then hand it off. Roughly, cold is more efficient but hot is more robust. Basically, the recommendation from this would be to avoid changes as far as possible, including changing between modes of transport - which includes, of course, getting onto the tube in the first place. When everything breaks down - every five minutes - of course you can revert to hot potato to route around the break.

You'll note that Tom's solution gets it in a one-r.

the 1980s considered as a battle in the air war over Germany

Thinking about my last post brought one of the ideas in this one to mind, especially given today's front page. That is, was the miners' strike a strategic bombing campaign?

You what? But consider the strategy the NUM adopted. The basic idea was to concentrate on the supply of coal to the steel industry - hence the battle of Orgreave. The point of this was to force British Steel, as it then was, to idle production. That would, they hoped, cause the steel managers and the downstream industries that consumed steel to put pressure on the government to settle. It might even bring out the steelworkers on strike.

The other option was to concentrate on the other big coal consumer, the electricity industry. Power cuts would hit the economy generally, and would hit consumers directly, unlike cuts in supply to the steelworks.

An important difference between the two was that much of the steel industry's coal was delivered as coke, whereas the power stations received coal directly from the mines. (Other differences included the fact that the power sector had more options and that power cuts might have unfavourable political consequences because they affected the public directly.) This created a number of critical network nodes between the coal and steel industries. The NUM hoped to target these and therefore send the crisis cascading through the downstream industries until the adversary cracked and gave in or the population rioted and got rid of them.

This really is very close to the whole package of airpower theory, or for that matter John Robb's global guerrillas concept. As readers will be aware, I'm sceptical of both. Anyway, why didn't it work?

Arguably, the big problem with this as a strategy was that the government didn't actually care about what happened to the downstream industries. For the government, even the maximum degree of trouble the miners could inflict on the steelmaking and metalworking economy was a price they were willing to pay.

A question to the reader: What is it that the Tories value most?

meet Project Lobster

My lobbying project has been entered in the Open Data Challenge! Someone posted this to the MySociety list, with rather fewer than the advertised 36 hours left. I was at a wedding and didn't read it at the time. After my partner and I had tried to invent a tap routine to the back end of Prince's "Alphabet Street" and had got up at 8am to make it for the sadistic bed & breakfast breakfast and gone back to help clean up and drink any unaccountably unconsumed champagne, and the only thing left to look forward to was the end of the day, I remembered the message and noted that I had to get it filed before midnight.

So it was filed in the Apps category - there's an Ideas category but that struck me as pathetic, and after all there is some running code. I pushed on to try and get something out under the Visualisation category but ManyEyes was a bit broken that evening and anyway its network diagram view starts to suck after a thousand or so vertices.

As a result, the project now has a name and I have some thin chance of snagging an actual Big Society cheque for a few thousand euros and a trip to Brussels. (You've got to take the rough with the smooth.)

The most recent experiment with the Lobster Project - see, it's got a name! It's got you its grips before you're born...it lets you think you're king when you're really a prawn...whoops, wrong shellfish - was to try out a new centrality metric, networkx.algorithms.centrality.betweenness_centrality. This is defined as the fraction of the shortest paths between all the pairs of nodes in the network that pass through a given node. As you have probably guessed, this is quite an inefficient metric to compute and the T1700 lappy took over a minute to crunch it compared to 7 seconds to complete the processing script without it. Perhaps the new KillPad would do better but the difference is big enough that it's obviously my fault.



As far as I can see, though, it's also not very useful. The results are correlated (R^2 = 0.64) with the infinitely faster weighted graph degree. (It also confirms that Francis Maude is the secret ruler of the world, though.)

The NX functions I'm really interested in, though, are the ones for clique discovery and blockmodelling. It's obvious that with getting on for 3,000 links and more to come, any visualisation is going to need a lot of reduction. Blockmodelling basically chops your network into groups of nodes you provide and aggregates the links between those groups - it's one way, for example, to get department level results.

But I'd be really interested to use empirical clique discovery to feed into blockmodelling - the API for the one generates a python list of cliques, which are themselves lists of nodes, and the other accepts a list of nodes or a list of lists (of nodes). Another interesting option might be to blockmodel by edge attribute, which would be a way of deriving results for the content of meetings via the "Purpose of meeting" field. However, that would require creating a list of unique meeting subjects and then iterating over it creating lists of nodes with at least one edge having that subject, and then shoving the resulting list-of-lists into the blockmodeller.

That's a lorra lorra iteratin' by anybody's standards, even if, this being Python, most of it will end up being rolled up in a couple of seriously convoluted list comps. Oddly enough, it would be far easier in a query language or an ORM, but I've not heard of anything that lets you do SQL queries against a NX graph.

Having got this far, I notice that I've managed to blog my enthusiasm back up.

Anyway, I think it's perhaps time for a meetup on this next week with Who's Rob-bying.

OpenTech washup, and an amended result

So it was OpenTech weekend. I wasn't presenting anything (although I'm kicking myself for not having done a talk on Tropo and Phono) but of course I was there. This year's was, I think, a bit better than last year's - the schedule filled up late on, and there were a couple of really good workshop sessions. As usual, it was also the drinking conference with a code problem (the bar was full by the end of the first session).

Things to note: everyone loves Google Refine, and I really enjoyed the Refine HOWTO session, which was also the one where the presenter asked if anyone present had ever written a screen-scraper and 60-odd hands reached for the sky. Basically, it lets you slurp up any even vaguely tabular data and identify transformations you need to clean it up - for example, identifying particular items, data formats, or duplicates - and then apply them to the whole thing automatically. You can write your own functions for it in several languages and have the application call them as part of the process. Removing cruft from data is always incredibly time consuming and annoying, so it's no wonder everyone likes the idea of a sensible way of automating it. There's been some discussion on the ScraperWiki mailing list about integrating Refine into SW in order to provide a data-scrubbing capability and I wouldn't be surprised if it goes ahead.

Tim Ireland's presentation on the political uses of search-engine optimisation was typically sharp and typically amusing - I especially liked his point that the more specific a search term, the less likely it is to lead the searcher to a big newspaper website. Also, he made the excellent point that mass audiences and target audiences are substitutes for each other, and the ultimate target audience is one person - the MP (or whoever) themselves.

The Sukey workshop was very cool - much discussion about propagating data by SMS in a peer-to-peer topology, on the basis that everyone has a bucket of inclusive SMS messages and this beats paying through the nose for Clickatell or MBlox to send out bulk alerts. They are facing a surprisingly common mobile tech issue, which is that when you go mobile, most of the efficient push-notification technologies you can use on the Internet stop being efficient. If you want to use XMPP or SIP messaging, your problem is that the users' phones have to maintain an active data connection and/or recreate one as soon after an interruption as possible. Mobile networks analogise an Internet connection to a phone call - the terminal requests a PDP (Packet Data Profile) data call from the network - and as a result, the radio in the phone stays in an active state as long as the "call" is going on, whether any data is being transferred or not.

This is the inverse of the way they handle incoming messages or phone calls - in that situation, the radio goes into a low power standby mode until the network side signals it on a special paging channel. At the moment, there's no cross-platform way to do this for incoming Internet packets, although there are some device-specific ways of getting around it at a higher level of abstraction. Hence the interest of using SMS (or indeed MMS).

Their other main problem is the integrity of their data - even without deliberate disinformation, there's plenty of scope for drivel, duplicates, cockups etc to get propagated, and a risk of a feedback loop in which the crap gets pushed out to users, they send it to other people, and it gets sucked up from Twitter or whatever back into the system. This intersects badly with their use cases - it strikes me, and I said as much, that moderation is a task that requires a QWERTY keyboard, a decent-sized monitor, and a shirt-sleeve working environment. You can't skim-read through piles of comments on a 3" mobile phone screen in the rain, nor can you edit them on a greasy touchscreen, and you certainly can't do either while looking out that you don't get hit over the head by the cops.

Fortunately, there is no shortage of armchair revolutionaries on the web who could actually contribute something by reviewing batches of updates, and once you have reasonably large buckets of good stuff and crap you can use Bayesian filtering to automate part of the process.

Francis Davey's OneClickOrgs project is coming along nicely - it automates the process of creating an organisation with legal personality and a constitution and what not, and they're looking at making it able to set up co-ops and other types of organisation.

I didn't know that OpenStreetMap is available through multiple different tile servers, so you can make use of Mapquest's CDN to serve out free mapping.

OpenCorporates is trying to make a database of all the world's companies (they're already getting on for four million), and the biggest problem they have is working out how to represent inter-company relationships, which have the annoying property that they are a directed graph but not a directed acylic graph - it's perfectly possible and indeed common for company X to own part of company Y which owns part of company X, perhaps through the intermediary of company Z.

OpenTech's precursor, Notcon, was heavier on the hardware/electronics side than OT usually is, but this year there were quite a few hardware projects. However, I missed the one that actually included a cat.

What else? LinkedGov is a bit like ScraperWiki but with civil servants and a grant from the Technology Strategy Board. Francis Maude is keen. Kumbaya is an encrypted, P2P online backup application which has the feature that you only have to store data from people you trust. (Oh yes, and apparently nobody did any of this stuff two years ago. Time to hit the big brown bullshit button.)

As always, the day after is a bit of an enthusiasm killer. I've spent part of today trying to implement monthly results for my lobby metrics project and it looks like it's much harder than I was expecting. Basically, NetworkX is fundamentally node-oriented and the dates of meetings are edge properties, so you can't just subgraph nodes with a given date. This may mean I'll have to rethink the whole implementation. Bugger.

I'm also increasingly tempted to scrape the competition's meetings database into ScraperWiki as there doesn't seem to be any way of getting at it without the HTML wrapping. Oddly, although they've got the Department of Health's horrible PDFs scraped, they haven't got the Scottish Office although it's relatively easy, so it looks like this wouldn't be a 100% solution. However, their data cleaning has been much more effective - not surprising as I haven't really been trying. This has some consequences - I've only just noticed that I've hugely underestimated Oliver Letwin's gatekeepership, which should be 1.89 rather than 1.05. Along with his network degree of 2.67 (the eight highest) this suggests that he should be a highly desirable target for any lobbying you might want to do.

lobbyists lobbying for lobbying

Best lobby metrics (lobbylyzer? lobster, for LoBbying Social Topology ExploreR?) result yet. Today I implemented my gatekeeper vs. flakcatcher metric - it averages the edge weights of all the neighbours of a minister, and returns a ratio of the difference between this and the average weight and the difference between the minister's weighting and the average. The principle is that if you lobby a minister, and then get access to another, your lobbying effort should gain or lose impact depending on the difference between the minister's own importance and the average. In the null hypothesis, where no minister is better or worse as a lobbying target than predicted by their department and their individual rank, you'd get the difference between the minister's weighting and the average.

If some ministers are gatekeepers, though, you would see a greater boost to your efforts at influence than the null case. Similarly, if some of them are flak catchers who mainly exist to turn away lobbying efforts, and you happened on one of those, you'd get a lesser boost. This metric should be greater than unity if the minister is a gatekeeper, 1 if they are perfectly mediocre, and less than unity if they are a flak catcher.

Interestingly, the Scotland and Wales Offices score highly. The highest value recorded is for David Jones MP, Parliamentary Under-Secretary of State for Wales, with 2.75. His closest rival is David Mundell MP, from the Scotland Office, on 1.94. The highest scoring Cabinet minister is the Scottish Secretary, Michael Moore. Some of this is down to the magic of low expectations. Nobody thinks these departments are great offices of state, not even the Welsh or the Scots - real power there has long since shifted to the devolved administrations. So if you meet any other minister, you're likely to do better. But the gatekeeper metric should handle this, as it measures influence relative to the structural difference between the minister and the average weighting. Arguably, this is a valid measurement. These ministries' role really is as a gatekeeper, something like a diplomatic representation in both directions.

So, who's the all-UK champ? It turns out to be Mark Harper MP, Minister for Political and Constitutional Reform, with an impressive 1.65, contrasting with his network degree of 0.08. Hilariously, one of his lobbies turns out to be the UK Public Affairs Council, the lobbyists' trade union, which wanted to see him in July, 2010 on the pressing matter of "lobbying".

He's followed home by Education PUSS Tim Loughton MP on 1.228, Defence PUSS Lord Astor on 1.2, Education's Lord Hill on 1.1, Justice PUSS Crispin Blunt MP with 1.09, and the sinister intellectual force that is Oliver Letwin MP, Minister of State for Government Policy, on 1.05. Francis Maude, whose horrific rise to national influence has been tracked with interest, turns out to be quite the flak catcher, on 0.43 - or is it that he claims to be a figure of authority in his own right? Does the buck stop there? After all, and as expected, once you meet the prime minister you can't really go anywhere but down.

Of course, what everyone will surely want to know is who gets the wooden spoon. Step forward Andrew Stunell MP, PUSS in the Department for Communities and Local Government, with a mighty 0.21 to go with his network degree of 0.3. Stunell held a large number of meetings around the country, notably in London, Bristol, and Bradford, as "Big Society Roundtables" with a wide range of community organisations. It would appear that nobody was more shortchanged than these. Meeting Mr. Stunell reduced one's average lobbying impact by a smacking 80%. Such was the coalition's contempt for, among other organisations, Operation Black Vote, the Stephen Lawrence Trust, and basically everyone in Bradford who showed up. The list is here.

It will surprise nobody that meeting a coalition minister would increase the UK Public Affairs Council's members' influence by 65%, but cut that of council tenants, Muslims, blacks, single mothers, young people (to list just a few) by 80%. But it's worth making it hideously explicit. And here's a lesson from all this obscure science that is easy enough to operationalise: if you see Andrew Stunell coming towards you through the Strangers' Bar with a smile on his face, don't make eye contact, don't shake hands, don't offer him your business card. Run. Spill a pint. Create a diversion. Trigger the fire alarm. Do not, in any circumstances, lobby him.

Here's the really sad bit. Stunell's dance card, from TWFY.

Voted very strongly against introducing foundation hospitals.
Voted strongly against Labour's anti-terrorism laws.
Voted very strongly against the Iraq war.
Voted very strongly for an investigation into the Iraq war.
Voted moderately against allowing ministers to intervene in inquests.
Voted a mixture of for and against greater autonomy for schools.
Voted moderately against replacing Trident.
Voted very strongly for the hunting ban.
Voted moderately for more EU integration.
Voted very strongly against introducing ID cards.
Voted very strongly for laws to stop climate change.
Voted very strongly against a stricter asylum system.
Voted moderately for removing hereditary peers from the House of Lords.
Voted strongly for a wholly elected House of Lords.
Voted very strongly for equal gay rights.
Voted moderately for a transparent Parliament.

I remember the Lib Dems. Do you? I wonder if Andrew Stunell remembers Andrew Stunell.

Routine maintenance

After this post and the outstanding response to it, I've just been working on the lobby project's underpinnings, specifically to backport some data cleaning from the analyser script into the original scraper, and to fix the one-edge-per-row version of the scraper. As a result I've had to flush the datastore and also search out some URIs that have changed. So far we've recreated 931 out of 1,721 meetings, although we're getting the dreaded "Execution status: run interrupted by a timeout". Actually, we've got 1,233 meetings back, and we've got rid of some crap. Anyone wanting the dataset can get it from the Scraperwiki API here or here for linkwise rather than meetingwise (coming soon) as either json-dict or csv. A full SQL syntax is available.

With luck, there will also be some more data quite soon. On the analysis score, notably, this and also this seem useful. The first estimates the value of a node based on its edges, which is fundamentally what I'm trying to achieve, and the second finds the cliques in the network a given node belongs to.

Regarding visualisation issues, I think one of my mistakes last time out was to visualise the data as a multi-graph - i.e. a structure with zero or more links between each node, permitting the existence of multiple links between the same pair of nodes. This invariably means a lot of links. The nature of the data - multiple meetings are absolutely central to the whole project, and lobbies meet ministers at different times and on different issues - enforces an underlying multigraph structure. But it would be possible to condense it for visualisation purposes - if we rolled up all links between the same nodes into one, we could tot up their weights and perhaps show that in the visualisation, as a thicker line for example.

lobby: update

I'm beginning to make some progress with the lobbying project. Last week I got it spitting out data; in mid-week, I optimised the process of loading the meetings from the ScraperWiki API into NetworkX. Hint: the obj_hook keyword argument in python's json.load() function is really useful!

This weekend it's producing information about lobbies, ministers, and government departments. I've got implementations nearly ready for a couple more dimensions of data - providing each actor's network degree by month, and trying to measure the extent to which ministers act as gatekeepers or flak-catchers. The first of those involves reimplementing a bit of NetworkX - you can't ask for node properties excluding certain edges by attribute, or at least you can't do so without creating a new subgraph, which seems ugly. The second, at the moment, counts the edges of a node if they have a higher weight than that of the node itself and expresses the sum of those edges' weights as a percentage of the total meetings that minister had. That doesn't take any account of time, yet.

I'm thinking of using Google App Engine to deploy it, running the data generator as a cron job and using the bulk uploader utility to slurp the results.

As a taster, the biggest single private interest lobbying Government is Barclays Bank, followed by Shell, the World Bank, the London Stock Exchange, BP, RBS, BAE, Standard Chartered, Lloyds, and Ratan Tata. This may not be that surprising. Neither is it very surprising, if somehow comforting in an old-fashioned way, that the two biggest lobbies of all are the Confederation of British Industry and the TUC, which is achieving about two-thirds the lobbying effort of the CBI and about twice that of Barclays. I was surprised to find that lobby 26 is Facebook, above Tesco, Microsoft, or UNISON. (Google is far, far down the list.) The highest placed individual trade union is the CWU at 24, between HSBC and the Electoral Commission. The littlest lobby is a nursery school in Leeds that got herded into a Big Society meeting with Nick Hurd MP.

I'm not so sure about using this model to assess the ministers, as we're using a priori weightings on them. But the decision to lobby a given minister must contain some information about the lobbyist's perceptions of their power and influence. Britain's most lobbied minister is Chris Grayling MP, Minister of State for Employment, who achieves a weighted degree of 4.2, not far off twice the prime minister. David Willetts, Vince Cable, Nick Clegg, and Francis Maude are the next four before the prime minister. They range between 2.8 and 2.6 with the PM on 2.3. Britain's least influential minister appears to be Baroness Warsi, minister without portfolio, on a score of 0.057.

BIS is the most lobbied department on 12.42, followed by the Department for Work and Pensions on 9.65, the Treasury on 7.065, the Cabinet Office on 5.62, and the DCLG on 3.825. Delight to the econophysicists (are they still around?): the distributions seem to show a nice power-law relationship! Which tells us what precisely? Well....not much except that it's a social network and they usually have them!

There were 2,073 nodes, either ministers or lobbies, in the graph at the last data upload. 2,848 interactions between them were analysed.

Does anyone have any ideas for other metrics that might be interesting?

I cannot begin to theorise what may have caused such a catastrophic malfunction

So, a free, jetlagged afternoon by the pool in Palo Alto, after this experience. What to do? Obviously, hack on some code. I dragged out the lobby analyzer project and got it to actually spit out ministers, lobbyists, and MPs, with their weighted degrees in the network, onto the command line. The conclusions are dreadful and confirm all my preliminary work. We are being ruled by Francis Maude and David Willetts. They both have significantly higher scores than the Prime Minister, with Willetts topping the poll. Of course, this is using the idea the wrong way up, but presumably the lobbyists' choice of who to lobby contains information about their perception of ministerial importance and influence.

BritiLeaks: the best website that doesn't exist yet

Absolutely, certainly, without a doubt the most important bit in the News of the World case:

Last Friday, a high court judge ordered NoW to make available Mulcaire's notes to the growing list of people suing the paper. Justice Geoffrey Vos, who is in charge of the hacking cases, ordered "rolling disclosure" to all claimants.

Hundreds of thousands of emails will now be handed over to alleged victims.

E-mails. Thousands of them. Hundreds of thousands of them. Just once, to have such means at my disposal! To have as many planes as this!... I don't know how the claimants' briefs intend to handle this epic dump, but I can't think of any more interesting political document going. Does anyone know if any of this stuff will end up in the public domain?

It's no surprise, in the light of that, that they're serving the people, with the people's currency, although if I read this right, it doesn't necessarily stop the disclosure. And, of course, there may well be more claims now a precedent has been set. There is of course also the police operation, but I don't hold out too much hope there.

And then there's this. Yet another story showing the police in a very bad light, in the week I had to thank'em. It doesn't show the industry in a great one either, as four out of five carriers seem to have treated the message from the police as a legalistic excuse to do nothing. Actually, three out of four - there's no mention of 3UK at all. I wonder why? O2 gets the prize, having apparently decided unilaterally to inform all its subscribers who were affected. The Guardian gets it right here, making the point that:

It also means many of the victims were deprived of the chance to check the call data, which is kept by the phone companies for only 12 months, and that they had no opportunity to change their pin codes or to assess the damage done by the interception of their messages.

This. RIPA III requires telecommunications data retention for 12 months, no more and no less. It looks horribly like there was an effort to ensure Screws' comedy STELLAR WIND wasn't disclosed until the logs were purged at the end of the canonical 12 months.

Of course, this won't help them any with the people who were spied upon last year. It looks even more as if the police accepted promises from News International that they would behave in future. Will they get their act together and sue, already?