Even more AV

More AV comment, from a colleague:

I voted "no". Why?

1. Arrow's theorem. Getting democracy "right" is mathematically impossible.
2. Elections are not there to select the "right" government, but to confer legitimacy on that which is elected, and to enable truly awful ones to be removed. FPTP is good enough.
3. Fiddling with the electoral maths is thinking too small. If you want to reform the system, and have a referendum on it, then a much more comprehensive reallocation of power between centre and edge is needed.

I'm not as impressed with Arrow's theorem as a lot of people are - perhaps because I'm just more comfortable with better rather than perfect than my correspondent here is. Actually, having just been reading the wikipedia discussion, I have a couple of fundamental disagreements - notably that I don't really have a problem with the notion that there is a potential case in which one "dictatorial" voter or coalition of voters decides the election. For a start, the whole point of an election is that the candidates try to secure the support of enough voters to win. Secondly, if who turns out to be the critical voter is random with respect to the demographic makeup of the electorate and not known in advance, there's an argument that this would in fact be fair, in that it wouldn't privilege any interest-group over any other. As it would be impossible to target them for campaigning purposes, politics would have to operate as if the system was formally fair in Arrow's sense. Given the entrenched biases in the current system, this would in fact be a move towards justice.

Anyway, there's nothing in Arrow to say that we should prefer a system that is pathological when it's operating normally to one that is usually better but might have odd corner cases. It's also ironic that economic libertarians get so hung up on Arrow when the same logic about Pareto efficiency is a well-known problem for free-market economics (making any given market more free doesn't necessarily move the macroeconomy closer to full allocative efficiency).

rewind to 2006

The log says AFRICOM wants to know what's up with B.727 msn 22045, ex of Irbis Air and now with "Mega" of Kazakhstan. Last heard of in Shymkent in January 2010, before that in Djibouti in 2008. I've no new information to add.

Even more AV

OK, so I'd just about reasoned myself around to voting no. But every time I get there I run into some nightmarish turdpool of mendacity from the no campaign. I may just not be able to stomach doing anything that George Osborne, John Reid, Robert Edmiston, Dan Hodges, and "Tom" "Newton"-"Dunn" agree with. I'd abstain, but that's no solution.

To summarise yet again, the arguments are as follows, net-net:

Yes: It's crap, and it doesn't afford an immediate kick at the Lib Dems, but there might be some angle to be had from it and anyway everyone hates the Lib Dems.
No: Inchoate lashout.

I'm actually quite keen on policy based on gut hatred - there have been quite a few occasions over the last ten years when allowing yourself to be guided by tribal loyalty, practiced trigger-movements, and aesthetic revulsion would have put you in the right when a lot of people got it wrong based on reason. But the rules are the rules, and surely the fear and loathing factor must respond to the sheer horror of the no campaign.

Glue Labour: how you can miss out the coalition and come up in 2015!

If you were asked to write 1,572 words for a national newspaper on the role of voluntary associations in the Labour Movement, and thank the Lord neither you or I is likely to meet this fate any time soon, would you manage to mention trade unions at least once? Not if you're Maurice Glasman. I don't intend to waste much time on "Blue Labour", but I do think this point needs to be made. Also, if you or I was to write such a slab, I think we'd make a better job of it. Here are a couple of pearls.

It required new work agreements so that all was not relentlessly up for grabs in an exclusively contractual churn.

In order to do this, Labour must establish those conversations that broker a common good within which party organisations such as Progress, the Fabians, Compass and the Christian Socialist Movement and Blue Labour talk and build a common programme.

The Obscurer seems to have anointed him as the new leader of Continuity Blairism, going by the eight handsome volumes it's dedicated to the sage of Holloway Road's thoughts on agriculture...sorry, the 1,572 words across a double centre page spread with a photo of him smoking cerebrally into the middle distance against a backdrop carefully blurred in line withOKTrends's empirically derived guidelines for successful online dating. This, next to a really Pravdaesque bit of sycophancy, from which I quote:

A source close to Miliband said the Labour leader was moved by what he heard as he munched on a bacon sandwich in Chrissy's. "Ed met a porter who said the proudest day of his life was when he got his badge. His dad had been a porter and then he got his badge. We have nothing against people in call centres but I am not sure there would be the same emotion on being given a first telephone headset."

"We have nothing against people in call centres". But, it would seem, a lot of them are women and some are even black, and anyway they aren't Proper Picturesque Proles for the TV, like. There are a whole lot of them, though, and perhaps they might have some of that emotion if more of them were in...a union. Perhaps I should start Glue Labour, for people who would rather sniff glue than read Maurice Glasman's risible tosh. Suggestions for a motto are invited.

lobbyists lobbying for lobbying

Best lobby metrics (lobbylyzer? lobster, for LoBbying Social Topology ExploreR?) result yet. Today I implemented my gatekeeper vs. flakcatcher metric - it averages the edge weights of all the neighbours of a minister, and returns a ratio of the difference between this and the average weight and the difference between the minister's weighting and the average. The principle is that if you lobby a minister, and then get access to another, your lobbying effort should gain or lose impact depending on the difference between the minister's own importance and the average. In the null hypothesis, where no minister is better or worse as a lobbying target than predicted by their department and their individual rank, you'd get the difference between the minister's weighting and the average.

If some ministers are gatekeepers, though, you would see a greater boost to your efforts at influence than the null case. Similarly, if some of them are flak catchers who mainly exist to turn away lobbying efforts, and you happened on one of those, you'd get a lesser boost. This metric should be greater than unity if the minister is a gatekeeper, 1 if they are perfectly mediocre, and less than unity if they are a flak catcher.

Interestingly, the Scotland and Wales Offices score highly. The highest value recorded is for David Jones MP, Parliamentary Under-Secretary of State for Wales, with 2.75. His closest rival is David Mundell MP, from the Scotland Office, on 1.94. The highest scoring Cabinet minister is the Scottish Secretary, Michael Moore. Some of this is down to the magic of low expectations. Nobody thinks these departments are great offices of state, not even the Welsh or the Scots - real power there has long since shifted to the devolved administrations. So if you meet any other minister, you're likely to do better. But the gatekeeper metric should handle this, as it measures influence relative to the structural difference between the minister and the average weighting. Arguably, this is a valid measurement. These ministries' role really is as a gatekeeper, something like a diplomatic representation in both directions.

So, who's the all-UK champ? It turns out to be Mark Harper MP, Minister for Political and Constitutional Reform, with an impressive 1.65, contrasting with his network degree of 0.08. Hilariously, one of his lobbies turns out to be the UK Public Affairs Council, the lobbyists' trade union, which wanted to see him in July, 2010 on the pressing matter of "lobbying".

He's followed home by Education PUSS Tim Loughton MP on 1.228, Defence PUSS Lord Astor on 1.2, Education's Lord Hill on 1.1, Justice PUSS Crispin Blunt MP with 1.09, and the sinister intellectual force that is Oliver Letwin MP, Minister of State for Government Policy, on 1.05. Francis Maude, whose horrific rise to national influence has been tracked with interest, turns out to be quite the flak catcher, on 0.43 - or is it that he claims to be a figure of authority in his own right? Does the buck stop there? After all, and as expected, once you meet the prime minister you can't really go anywhere but down.

Of course, what everyone will surely want to know is who gets the wooden spoon. Step forward Andrew Stunell MP, PUSS in the Department for Communities and Local Government, with a mighty 0.21 to go with his network degree of 0.3. Stunell held a large number of meetings around the country, notably in London, Bristol, and Bradford, as "Big Society Roundtables" with a wide range of community organisations. It would appear that nobody was more shortchanged than these. Meeting Mr. Stunell reduced one's average lobbying impact by a smacking 80%. Such was the coalition's contempt for, among other organisations, Operation Black Vote, the Stephen Lawrence Trust, and basically everyone in Bradford who showed up. The list is here.

It will surprise nobody that meeting a coalition minister would increase the UK Public Affairs Council's members' influence by 65%, but cut that of council tenants, Muslims, blacks, single mothers, young people (to list just a few) by 80%. But it's worth making it hideously explicit. And here's a lesson from all this obscure science that is easy enough to operationalise: if you see Andrew Stunell coming towards you through the Strangers' Bar with a smile on his face, don't make eye contact, don't shake hands, don't offer him your business card. Run. Spill a pint. Create a diversion. Trigger the fire alarm. Do not, in any circumstances, lobby him.

Here's the really sad bit. Stunell's dance card, from TWFY.

Voted very strongly against introducing foundation hospitals.
Voted strongly against Labour's anti-terrorism laws.
Voted very strongly against the Iraq war.
Voted very strongly for an investigation into the Iraq war.
Voted moderately against allowing ministers to intervene in inquests.
Voted a mixture of for and against greater autonomy for schools.
Voted moderately against replacing Trident.
Voted very strongly for the hunting ban.
Voted moderately for more EU integration.
Voted very strongly against introducing ID cards.
Voted very strongly for laws to stop climate change.
Voted very strongly against a stricter asylum system.
Voted moderately for removing hereditary peers from the House of Lords.
Voted strongly for a wholly elected House of Lords.
Voted very strongly for equal gay rights.
Voted moderately for a transparent Parliament.

I remember the Lib Dems. Do you? I wonder if Andrew Stunell remembers Andrew Stunell.

for those non-cooperative with the system

I was wondering what "Ingeus UK" was - it's the 15th heaviest lobby in the kingdom with degree 0.17, just behind HSBC and ahead of GlaxoSmithKline, Microsoft, and LOCOG - but now we know.

Another lobbying thing

This project begins to want a name.

Routine maintenance

After this post and the outstanding response to it, I've just been working on the lobby project's underpinnings, specifically to backport some data cleaning from the analyser script into the original scraper, and to fix the one-edge-per-row version of the scraper. As a result I've had to flush the datastore and also search out some URIs that have changed. So far we've recreated 931 out of 1,721 meetings, although we're getting the dreaded "Execution status: run interrupted by a timeout". Actually, we've got 1,233 meetings back, and we've got rid of some crap. Anyone wanting the dataset can get it from the Scraperwiki API here or here for linkwise rather than meetingwise (coming soon) as either json-dict or csv. A full SQL syntax is available.

With luck, there will also be some more data quite soon. On the analysis score, notably, this and also this seem useful. The first estimates the value of a node based on its edges, which is fundamentally what I'm trying to achieve, and the second finds the cliques in the network a given node belongs to.

Regarding visualisation issues, I think one of my mistakes last time out was to visualise the data as a multi-graph - i.e. a structure with zero or more links between each node, permitting the existence of multiple links between the same pair of nodes. This invariably means a lot of links. The nature of the data - multiple meetings are absolutely central to the whole project, and lobbies meet ministers at different times and on different issues - enforces an underlying multigraph structure. But it would be possible to condense it for visualisation purposes - if we rolled up all links between the same nodes into one, we could tot up their weights and perhaps show that in the visualisation, as a thicker line for example.

lobby: update

I'm beginning to make some progress with the lobbying project. Last week I got it spitting out data; in mid-week, I optimised the process of loading the meetings from the ScraperWiki API into NetworkX. Hint: the obj_hook keyword argument in python's json.load() function is really useful!

This weekend it's producing information about lobbies, ministers, and government departments. I've got implementations nearly ready for a couple more dimensions of data - providing each actor's network degree by month, and trying to measure the extent to which ministers act as gatekeepers or flak-catchers. The first of those involves reimplementing a bit of NetworkX - you can't ask for node properties excluding certain edges by attribute, or at least you can't do so without creating a new subgraph, which seems ugly. The second, at the moment, counts the edges of a node if they have a higher weight than that of the node itself and expresses the sum of those edges' weights as a percentage of the total meetings that minister had. That doesn't take any account of time, yet.

I'm thinking of using Google App Engine to deploy it, running the data generator as a cron job and using the bulk uploader utility to slurp the results.

As a taster, the biggest single private interest lobbying Government is Barclays Bank, followed by Shell, the World Bank, the London Stock Exchange, BP, RBS, BAE, Standard Chartered, Lloyds, and Ratan Tata. This may not be that surprising. Neither is it very surprising, if somehow comforting in an old-fashioned way, that the two biggest lobbies of all are the Confederation of British Industry and the TUC, which is achieving about two-thirds the lobbying effort of the CBI and about twice that of Barclays. I was surprised to find that lobby 26 is Facebook, above Tesco, Microsoft, or UNISON. (Google is far, far down the list.) The highest placed individual trade union is the CWU at 24, between HSBC and the Electoral Commission. The littlest lobby is a nursery school in Leeds that got herded into a Big Society meeting with Nick Hurd MP.

I'm not so sure about using this model to assess the ministers, as we're using a priori weightings on them. But the decision to lobby a given minister must contain some information about the lobbyist's perceptions of their power and influence. Britain's most lobbied minister is Chris Grayling MP, Minister of State for Employment, who achieves a weighted degree of 4.2, not far off twice the prime minister. David Willetts, Vince Cable, Nick Clegg, and Francis Maude are the next four before the prime minister. They range between 2.8 and 2.6 with the PM on 2.3. Britain's least influential minister appears to be Baroness Warsi, minister without portfolio, on a score of 0.057.

BIS is the most lobbied department on 12.42, followed by the Department for Work and Pensions on 9.65, the Treasury on 7.065, the Cabinet Office on 5.62, and the DCLG on 3.825. Delight to the econophysicists (are they still around?): the distributions seem to show a nice power-law relationship! Which tells us what precisely? Well....not much except that it's a social network and they usually have them!

There were 2,073 nodes, either ministers or lobbies, in the graph at the last data upload. 2,848 interactions between them were analysed.

Does anyone have any ideas for other metrics that might be interesting?

AV: roundup and conclusion of a sort

So I asked for your help to decide what I thought about the alternative vote.

A few of you were supportive on the grounds that it was a start and it would be possible to demand further improvement later.

NomadUK said..I say vote yes. It's not great, but it greases the skids by changing the system; once changed, it's that much easier to change it again — much as the Reform Act 1832, whilst imperfect, led to far greater changes in the electoral system. If AV is rejected, it'll be touted as public approval of the current system, and it'll be a generation or more before anyone dares try again.
9:21 AM
Jonathan Hopkin said...Agree with Nomad. If you want PR, voting against this makes it less likely. Remember the sinking of devolution in the 1970s? Took 20 years to get that going again. Nick Clegg is ****ed anyway, AV isn't going to save him.
11:21 AM
Pauline said...I agree with Nomad as well. Just think of the "told you so" smirking if there's a no vote. And I can't bring myself to side with Cameron and bloody Nick Griffin.

The problem here is that every attempt to model its effects I've seen primarily benefits the Lib Dems. All other things being equal, this fulfils their primary interest in supporting electoral reform. More proportionality starts to bring other parties into the game. In a transitional AV system, the Lib Dems would be in the position of the German FDP, and they wouldn't have any interest in weakening this position. They would tend to swing against anyone who suggested STV or more, and as kingmakers, prevent it from happening. If making a move on electoral reform really did make it easier to go further, we'd somehow have to get to AV without having a single swing party with an interest in sticking at AV. Essentially, we'd need to have a Labour government with a manifesto commitment to STV, which requires that the Lib Dems get such a thrashing that even the AV bonus can't keep them relevant.

Well, I can certainly imagine the Lib Dems getting a thrashing at the next election. But if the point is to beat the coalition and elect a strong Labour government, either on its own or as a hegemonic coalition partner with a few Lib Dem survivors, and then pass STV, why risk the scenario where the Lib Dems just squeak by thanks to AV and put the Tories in again? What benefit does the detour through AV provide? Isn't it just a more complicated and slightly riskier route to the same goal?

There were those who strongly opposed AV on the grounds that losing the vote would destabilise the coalition and bring a general election closer:

Phil - April 11, 2011 at 10:56 am: Anything that makes the coalition less cohesive is good for us (and for the country), as is anything that stops the Lib Dem leadership from carrying on as if 2010 was politics as usual. Turn it round: the fact that a Yes vote would make Nick Clegg happy wouldn’t be a good enough reason to vote No, but the prospect of a Yes vote consolidating Clegg’s leadership of the Lib Dems and hence stabilising the coalition is quite good enough for me.

Chris Williams - April 11, 2011 at 11:05 am: I’m with Phil on this: vote No to split the Lib Dems and bring down the Coalition ASAP. The next lot in will find it harder to screw up the public sector.

On the other hand:

Tom said..." I'm teetering between the principle of spanking Clegg and the principle of doing anything the Murdoch papers are lying about" Organ grinder or monkey? Yes all the way, baby. 8:18 PM

Some people had technical arguments in favour of AV:

Raphael - April 10, 2011 at 10:14 pm: As far as I can tell, if voters don’t act too stupidly, the main effect of AV in Britain would be that the Tories (or UKIP, or the BNP,) wouldn’t be able anymore to win a constituency where most voters are more or less left of centre or centre-left through a split in the left-leaning vote.

Which, going by past results, might mean that it would become a lot more difficult for the Tories to win a majority in the Commons anytime soon, or even to get as close to an outright majority as they’re now again.

Phil Hunt: April 10, 2011 at 6:47 pm: Another advantage of AV — it makes it easier to get rid of unpopular MPs. I would love to see the look on Clegg’s face if AV wins and the voters of Sheffield Hallam use it to get rid of him.

Anonymous said...STV is fairer. But AV at least prevents you 'wasting' your vote by voting for a minor party. And over time minor parties can grow in strength to win an AV seat. Adam Brandt winning the seat of Melbourne for the Greens at the last Federal election being a case in point. 12:50 PM

One reader was strongly in favour of a majoritarian system, which is surprising as he's a Lib Dem. Another was worried that the Tories would win an early election, to which I can only respond that the UK Polling Report's projection based on the current state of the polls forecasts a Labour majority of 86. And at least one reader believes that voting is just part of the system, like, and I shouldn't bother.

In general, it seems to me that the problem is basically whether you consider the Lib Dems to be a credible partner for a left-wing government. If so, then all the stuff about a progressive majority and keeping the Tories out of as many seats as possible retains its force, up to a point. But only up to a point. One thing we know now that we didn't in May, 2010 is that the Lib Dems are indeed capable of enabling a radical Tory government. For the "anti-Tory AV" model to work, you have to assume that Labour-Lib Dem coalitions will drag the political spectrum far enough to the Left to balance out the inevitable periods of Tory-Lib Dem coalition. That might be true in a STV world where a Labour-led coalition would have to be concerned about its left flank, but it wouldn't be true in an AV world where, assuming mediocrity, the election would be decided by the Liberals. Of course, the Liberals might be a moderating influence on the Tories, but have we seen that much evidence of this?

And if you don't believe they can be treated as a reliable factor in the Left's calculations, well, you just have to consider them to be Tories operationally, more like the Aussies' National party than the FDP.

In fact, I'm coming around to the view that AV itself sucks. Isn't it just a way of dignifying swing-voter politics? Rather than hypertargeting five people in the bit of Stevenage with no smelly foreigners, close to the Tesco and just far enough from the motorway, isn't it just a way of redefining them as the Lib Dem base?

So what about "no, and campaign to get STV on the next Labour manifesto"?

(Someone also dropped off this link, which makes a strong argument against letting Labour become a second preference party.)

(I really am starting to talk like I'm back in again, aren't I?)

ring ring! who's there?

On the same day that NATO sort-of apologised for a fratricide incident in which a group of tanks the Libyan rebels had put into service were mistaken for Libyan government tanks, it turned out that the MOD was going to send the rebels 500 satellite phones. Well, you can see the point, but the first thing that came to mind was - what? now? why didn't this happen weeks ago? Is this whole campaign being managed by clowns? And then, of course, I remembered Dave from PR and Sarko and Liam He's a doctor, you know and Pocket Bismarck. Right.

But then, there's the Big Society. This is a deeply cool story - Libyan GSM engineers work out how to take over the network in rebel territory and get it going again. The WSJ overstates some elements - it's not so much that Gadhafi's government designed the network to be centralised in Tripoli, GSM networks are very centralised by design - but overall it's a pretty good account. They set up their own switch, home and visitor location registers, and international gateway with satellite connectivity, piped all the base station controllers in their territory into their own set-up, and obtained a copy of the original Libyana HLR with all the phone numbers. Fortunately they decided to let everyone make free calls (viva la revolucion!), or they'd have still been waiting for the billing system to be integrated six months later, whether in the minister's office or the Libyan Lubyanka.

Ironically, they got quite a bit of help from, of all telcos, Etisalat, the UAE's national operator. They lent them a lot of equipment and provided the satellite hookup and international access. This is amusing as Etisalat is famous for censoring more Web sites than the Chinese Great Firewall. For their part, the monster Chinese manufacturer Huawei refused to have anything to do with the rebels (or should that be "splittists"?)

This is good as far as it goes, but nobody in NATO CAOC-9 in Naples or the former AIRSOUTH now in Izmir or anywhere else with a NATO TLA is going to let random cell phones talk to the airpower infrastructure. Why didn't anyone send those satellite phones earlier? Ah, yes, clowns.

Of course, there's a possibility that they may have been worrying about releasing them into the wild. Here's Secret Défense confirming assorted loose MANPADs wandering about. But they're more trackable than arms, less directly dangerous, and far easier to buy anyway.

In other news, there's a really excellent piece on the Toyota Land Cruiser as an engine of war in the FAZ, for German-speakers only.

of course, Gauck tells me my file is probably in Moscow

Well, if you're the prime minister, you're not allowed to fight back against your enemies in case you win because Gus O'Donnell says so, as long as the enemies aren't the right kind of enemies like Coronation Street editors and UNISON hospital porters. All clear so far? When are the Tory apologies to O'Donnell going to show up - they thought he was biased against them....

Further, interesting subplot - Brown told Peter Mandelson his phone might have been compromised. Mandelson asked the Information Commissioner, who had nothing to say. Of course he didn't - as far as I know he had nothing to do with the case. If Mandelson had wanted to know, he needed to ask the police. Was Mandelson trying not to find out, so as not to burn his bridges with the Murdochs? Or was he just ill-informed?

Also, did the central government have any communications security at all? Did CESG or MI5 not have anything at all to say about this? Didn't any of them just change their damn password, or even change their damn number?

three lessons on risks: a movie

Daniel Davies is notorious for making more than full use of a joke once he gets hold of it. I think this is the original source (perhaps even the Urquell) of his line that Black Swan author Nassim Nicholas Taleb must be furious at what Hollywood did to his book in development.

Well, I watched the movie on a plane the other day. It was that or The Social Network - I was planning on a week in Silicon Valley surrounded by tightly wound super-ambitious geeks, so I get enough of that at work. As it happens, there are a couple of good lessons about risk in Black Swan. Perhaps Taleb shouldn't be so touchy about it after all.

1. Tail-risk is real

Just as Nassim Taleb said in the book, no matter how good your planning, you can't hedge everything and you will tend to underestimate the weirder and wilder ends of the distribution. One day, something not just bad but incalculably weird, something you never expected you didn't expect, will come rapid-roping into your back garden and piss in the pool. Of course, it's likely to happen on stage on the first night at the worst possible moment. You'll have to be ready, but you can only be ready in a general sense. Get your trigger movements right - far better to be calling an ambulance and plunging into the fray than locked in the bunker with a PR agent and a large amount of toilet paper. Act right in the crisis and much will be forgiven.

Inevitably, if you want a clue, look at the things you try to repress and deny and don't believe could ever happen. That's why you deny them.

2. That said, you've got to put up with it

All precautions must be seen in the light of the scale of the threat. Too much security is as dangerous as too little (this may be more Schneier than Taleb). Without a certain amount of optimism bias and risk tolerance, you'll never get anything done. In fact, you'll end being terrified of your shadow. (And why did you choose the word shadow, with its, ah, many meanings, Mr. Garrovell?) Your colleagues may well wish they had your job, but that's no reason to kill yourself. In fact, after a certain level of neurosis is passed, self-protection shades over into self-sabotage - delivering just what you imagine your enemies want, whether they be real or imagined.

3. Don't draw conclusions based on regional accents

Black Swan is the only movie I can think of in which New Yorkers see an outsider - a Californian - as being unimaginably evil, sophisticated, cool, and cunning. In fact, this was the plot detail that kept coming back to me. Wall Street and City investors in dozens of regional mortgage lenders that turned into financial neutron bombs imagined they were smarter than the offcomed'uns.

I cannot begin to theorise what may have caused such a catastrophic malfunction

So, a free, jetlagged afternoon by the pool in Palo Alto, after this experience. What to do? Obviously, hack on some code. I dragged out the lobby analyzer project and got it to actually spit out ministers, lobbyists, and MPs, with their weighted degrees in the network, onto the command line. The conclusions are dreadful and confirm all my preliminary work. We are being ruled by Francis Maude and David Willetts. They both have significantly higher scores than the Prime Minister, with Willetts topping the poll. Of course, this is using the idea the wrong way up, but presumably the lobbyists' choice of who to lobby contains information about their perception of ministerial importance and influence.

BritiLeaks: the best website that doesn't exist yet

Absolutely, certainly, without a doubt the most important bit in the News of the World case:

Last Friday, a high court judge ordered NoW to make available Mulcaire's notes to the growing list of people suing the paper. Justice Geoffrey Vos, who is in charge of the hacking cases, ordered "rolling disclosure" to all claimants.

Hundreds of thousands of emails will now be handed over to alleged victims.

E-mails. Thousands of them. Hundreds of thousands of them. Just once, to have such means at my disposal! To have as many planes as this!... I don't know how the claimants' briefs intend to handle this epic dump, but I can't think of any more interesting political document going. Does anyone know if any of this stuff will end up in the public domain?

It's no surprise, in the light of that, that they're serving the people, with the people's currency, although if I read this right, it doesn't necessarily stop the disclosure. And, of course, there may well be more claims now a precedent has been set. There is of course also the police operation, but I don't hold out too much hope there.

And then there's this. Yet another story showing the police in a very bad light, in the week I had to thank'em. It doesn't show the industry in a great one either, as four out of five carriers seem to have treated the message from the police as a legalistic excuse to do nothing. Actually, three out of four - there's no mention of 3UK at all. I wonder why? O2 gets the prize, having apparently decided unilaterally to inform all its subscribers who were affected. The Guardian gets it right here, making the point that:

It also means many of the victims were deprived of the chance to check the call data, which is kept by the phone companies for only 12 months, and that they had no opportunity to change their pin codes or to assess the damage done by the interception of their messages.

This. RIPA III requires telecommunications data retention for 12 months, no more and no less. It looks horribly like there was an effort to ensure Screws' comedy STELLAR WIND wasn't disclosed until the logs were purged at the end of the canonical 12 months.

Of course, this won't help them any with the people who were spied upon last year. It looks even more as if the police accepted promises from News International that they would behave in future. Will they get their act together and sue, already?

Alternative vote: a reconnaissance by fire

I've realised that I don't know where I stand on electoral reform. (With that, even my readership vanishes into the distance like Libyan rebels encountering Gadhafi's one loyal artillery observer.) So I've decided to abuse both sides in the hope that they react interestingly.

For AV: What, you want to give Nick Clegg what he wants? Crush! Like! Rat! Are you chicken? Anyway, it's not proper proportional voting - it's just a weak compromise that happens to give Cleggy Boy a life pension without even the embarrassment of going to the Lords. Alan Beattie is right. The Left, the Greens, the 'kippers and fascists and God knows what, they'll all be shut out just as much as ever. And, y'know, AV is cheating, like stealth aircraft and quantum computing and gas barbecues. Caroline Lucas didn't need AV to get elected - she had to do it the hard way. And if the Lib Dems get AV, do you really think they'll be in a hurry to concede STV? They get all the benefit of AV, and then they'll pocket it and just sit there. No is good. Look at the polls, anyway - the solution is to force a general election as soon as possible. Power, baby!

Against AV: Well, nice mates you've got there. Have you seen the fuckos and liars and thugs and gargoyles on your side? There's "Tom" "Newton"-"Dunn" of the "Sun", talking about "Baroness Warsi warns that the DANGEROUS alternative vote system will let FASCISTS into the House of Commons..." Rather than the papers, where they belong. It's the M factor - whatever Murdoch wants is evidently wrong. There's the charming No campaign, which is so rich it refused to take its public funding so it didn't have to admit to who's funding it. And just imagine Eric Pickles' face! And Redwood! The horror of it! We have a moral responsibility to vote against the Tory in all its forms.

Does that cover everything? I'm teetering between the principle of spanking Clegg and the principle of doing anything the Murdoch papers are lying about. On the merits, as far as I can see, AV is a little better than the current system, but not really enough to be worth having and certainly not enough to let the Lib Dems off the hook. And I'm in the mood for approval voting, the system where you cross off all the candidates you absolutely reject and the least hated wins. But please try to convince me.

A regrettable breach of security regulations

Remember this post from 2006, and especially this one from a year later on the next big miscarriage of justice? Well, look what just happened. It's far worse than even I thought - the police were well aware that there were serious problems with the Landslide case as early as February, 2003. Specifically, the old National Crime Squad seems to have been extremely gung-ho about the whole project while the regional police forces were much more sceptical. Later the whole thing was slung to the Child Exploitation and Online Protection Centre, one of the weird sort-of police agencies that proliferated in the late Blair years. Meanwhile, a suspect has succeeded in claiming damages. Both cases show various police forces in a very bad light indeed - the US postal inspector in the first suddenly retired to look after his "sick wife" when his evidence was challenged, while as for Hertfordshire Police:

Despite this, the officer, Detective Constable Brian Hopkins, pressed three charges of possession of indecent images of children. Mr Justice Mackay said he cut a "rather pathetic figure" in the witness box, having initially claimed he could not give evidence because of a psychiatric condition....The judge found that Mr Hopkins, who has since left policing, not only had "no honest belief in the possession charges when he caused them to be brought against [Mr Clifford]", but did so "to protect his own position".

Feel the fremdscham, baby!

Meanwhile, my bank card has been compromised. So I was in San Francisco of a Sunday, walking around the Tenderloin looking for a cash point that wasn't looking back at me with mischief in its eye. Preferably one attached to a bank. I eventually walked up as far as Van Ness and found a Wells Fargo branch. It wouldn't give me any money, nor would the Bank of America. So I ended up phoning the bank at extortionate roaming rates, standing on the forecourt with a small encampment of the homeless. Thinking that I had less US currency to my name than they did, I struggled through the IVR thickets, confirmed my salary hadn't somehow vanished, and got into a queue to report that the fraud-detection robots had zapped me. I stayed on the line until AT&T dropped the call after 12 minutes. The phone started whining; it's like a little jet fighter. You can do a lot of cool things with it, but it's best not to go too far from the refuelling tanker or you're screwed. Back to the hotel. I tried to call them on Skype, but AT&T's WLAN was too bad to hear the IVRs. I plugged in the phone, called again, explained that I didn't want to report the card stolen but rather the opposite, and sat in 23 minutes of queues. Curses...curses...24 hour fraud algorithms...not 24 hour staffing, though...why not call me?...banks...banks...banks...!

And then I got through. And the fraud investigators told me that the police had found my card in a list of cloned cards offered for sale on the black market. In the circumstances, they hadn't called me or given out any information for fear of giving away the secret, as the investigation was still going on. Oh...right. They listed some transactions, agreed to let me withdraw up to £100 a day in cash and honour direct debits, and left the Visa facility frozen. They refused to say anything about where or when the security breach might have occurred, although I think the detail about the Visa card might be significant. Call us when you get back to the UK - and by the way, here's the direct number.

The whole incident had just been annoying up to that point, but this changed the game. I was left with a whole load of surplus indignation on my hands past its use-by date. It cluttered up my room at the Phoenix like a chunk of un-Californian, clanking machinery. I suppose I could spend it on the thieves, but who were they? Rather than just harassing me and profiteering, my bank had actually done something I could only agree with. And the police had actually protected me from an actual crime, without my even noticing, with the occult efficiency Norman Lewis said had attached itself to the word "intelligence".

As far as I know, no money is missing, but I haven't audited as many as 14 months' worth of transactions through my current account yet. That's since this card was issued - they couldn't give me any other bounds on it. After all, as they said, it was impossible to say how far the list of cards had been sold on by whoever had originally collected them.

Anyway, I didn't even need to draw any more cash after the first $100. My expenses in Silicon Valley were unusually frugal - the nearest I came to spending significant amounts of money was trying to catch up with two colleagues who'd gone out looking for amusement. (I spent 20 minutes looking for a cab in Palo Alto at 10 o'clock at night and eventually gave up, having noticed that there seemed to be less traffic on the roads at that time of night than I would have expected in a Yorkshire Dales village.) I read the two Operation Ore articles and logged them for future use. As briefed, I called HSBC on my arrival back in Britain and they initiated a new card.

And it was only as I wrote this that I remembered I ought to be scared. After all, it is impossible to say how far the list of cards...

Scaling and scoping the NYT paywall

Amusingly for a comment on scalability, I couldn't post this on D^2's thread because Blogger was in a state. Anyway, it's well into the category of "comments that really ought to be posts" so here goes. So various people are wondering how the New York Times managed to spend $50m on setting up their paywall. D^2 reckons that they're overstating, for basically cynical reasons. I think it's more fundamental than that.

The complexity of the rules makes it sound like a telco billing system more than anything else - all about rating and charging lots and lots of events in close to real-time based on a hugely complicated rate-card. You'd be amazed how many software companies are sustained by this issue. It's expensive. The NYT is counting pages served to members (easy) and nonmembers (hard), differentiating between referral sources, and counting different pages differently. Further, it's got to do it quickly. Latency from the US West Coast (their worst case scenario) to nytimes.com is currently about 80 milliseconds. User-interface research suggests that people perceive a response as instant at 100ms - web surfing is a fairly latency tolerant application, but when you think that the server itself takes some time to fetch the page and the data rate in the last mile will restrict how quickly it can be served, there's a very limited budget of time for the paywall to do its stuff without annoying the hell out of everyone.

Although the numbers of transactions won't be as savage, doing real-time rating for the whole NYT website is going to be a significant scalability challenge. Alexa reckons 1.45% of global Web users hit nytimes.com, for example. As comparison, Salesforce.com is 0.4% and that's already a huge engineering challenge (because it's much more complicated behind the scenes). There are apparently 1.6bn "Internet users" - I don't know how that's defined - so that implies that the system must scale to 268 transactions/second (or about 86,400 times the daily reach of my blog!)

A lot of those will be search engines, Internet wildlife, etc, but you still have to tell them to fuck off and therefore it's part of your scale & scope calculations. That's about a tenth of HSBC's online payments processing in 2007, IIRC, or a twentieth of a typical GSM Home Location Register. (The usual rule of thumb for those is 5 kilotransactions/second.) But - and it's the original big but - you need to provision for the peak. Peak usage, not average usage, determines scale and cost. Even if your traffic distribution was weirdly well-behaved and followed a normal distribution, you'd encounter a over 95th percentile event one day in every 20. And network traffic doesn't, it's usually more, ahem, leptokurtotic. So we've got to multiply that by their peak/mean ratio.

And it's a single point of failure, so it has to be robust (or at least fail to a default-open state but not too often). I for one can't wait for the High Scalability article on it.

So it's basically similar in scalability, complexity, and availability to a decent sized MVNO's billing infrastructure, and you'd be delighted to get away with change from £20m for that.