it could be you, but not while she's around

This is fascinating, especially when you remember GTech's role in our own dear sordid raffle. Great reporting, too.

OpenTech washup, and an amended result

So it was OpenTech weekend. I wasn't presenting anything (although I'm kicking myself for not having done a talk on Tropo and Phono) but of course I was there. This year's was, I think, a bit better than last year's - the schedule filled up late on, and there were a couple of really good workshop sessions. As usual, it was also the drinking conference with a code problem (the bar was full by the end of the first session).

Things to note: everyone loves Google Refine, and I really enjoyed the Refine HOWTO session, which was also the one where the presenter asked if anyone present had ever written a screen-scraper and 60-odd hands reached for the sky. Basically, it lets you slurp up any even vaguely tabular data and identify transformations you need to clean it up - for example, identifying particular items, data formats, or duplicates - and then apply them to the whole thing automatically. You can write your own functions for it in several languages and have the application call them as part of the process. Removing cruft from data is always incredibly time consuming and annoying, so it's no wonder everyone likes the idea of a sensible way of automating it. There's been some discussion on the ScraperWiki mailing list about integrating Refine into SW in order to provide a data-scrubbing capability and I wouldn't be surprised if it goes ahead.

Tim Ireland's presentation on the political uses of search-engine optimisation was typically sharp and typically amusing - I especially liked his point that the more specific a search term, the less likely it is to lead the searcher to a big newspaper website. Also, he made the excellent point that mass audiences and target audiences are substitutes for each other, and the ultimate target audience is one person - the MP (or whoever) themselves.

The Sukey workshop was very cool - much discussion about propagating data by SMS in a peer-to-peer topology, on the basis that everyone has a bucket of inclusive SMS messages and this beats paying through the nose for Clickatell or MBlox to send out bulk alerts. They are facing a surprisingly common mobile tech issue, which is that when you go mobile, most of the efficient push-notification technologies you can use on the Internet stop being efficient. If you want to use XMPP or SIP messaging, your problem is that the users' phones have to maintain an active data connection and/or recreate one as soon after an interruption as possible. Mobile networks analogise an Internet connection to a phone call - the terminal requests a PDP (Packet Data Profile) data call from the network - and as a result, the radio in the phone stays in an active state as long as the "call" is going on, whether any data is being transferred or not.

This is the inverse of the way they handle incoming messages or phone calls - in that situation, the radio goes into a low power standby mode until the network side signals it on a special paging channel. At the moment, there's no cross-platform way to do this for incoming Internet packets, although there are some device-specific ways of getting around it at a higher level of abstraction. Hence the interest of using SMS (or indeed MMS).

Their other main problem is the integrity of their data - even without deliberate disinformation, there's plenty of scope for drivel, duplicates, cockups etc to get propagated, and a risk of a feedback loop in which the crap gets pushed out to users, they send it to other people, and it gets sucked up from Twitter or whatever back into the system. This intersects badly with their use cases - it strikes me, and I said as much, that moderation is a task that requires a QWERTY keyboard, a decent-sized monitor, and a shirt-sleeve working environment. You can't skim-read through piles of comments on a 3" mobile phone screen in the rain, nor can you edit them on a greasy touchscreen, and you certainly can't do either while looking out that you don't get hit over the head by the cops.

Fortunately, there is no shortage of armchair revolutionaries on the web who could actually contribute something by reviewing batches of updates, and once you have reasonably large buckets of good stuff and crap you can use Bayesian filtering to automate part of the process.

Francis Davey's OneClickOrgs project is coming along nicely - it automates the process of creating an organisation with legal personality and a constitution and what not, and they're looking at making it able to set up co-ops and other types of organisation.

I didn't know that OpenStreetMap is available through multiple different tile servers, so you can make use of Mapquest's CDN to serve out free mapping.

OpenCorporates is trying to make a database of all the world's companies (they're already getting on for four million), and the biggest problem they have is working out how to represent inter-company relationships, which have the annoying property that they are a directed graph but not a directed acylic graph - it's perfectly possible and indeed common for company X to own part of company Y which owns part of company X, perhaps through the intermediary of company Z.

OpenTech's precursor, Notcon, was heavier on the hardware/electronics side than OT usually is, but this year there were quite a few hardware projects. However, I missed the one that actually included a cat.

What else? LinkedGov is a bit like ScraperWiki but with civil servants and a grant from the Technology Strategy Board. Francis Maude is keen. Kumbaya is an encrypted, P2P online backup application which has the feature that you only have to store data from people you trust. (Oh yes, and apparently nobody did any of this stuff two years ago. Time to hit the big brown bullshit button.)

As always, the day after is a bit of an enthusiasm killer. I've spent part of today trying to implement monthly results for my lobby metrics project and it looks like it's much harder than I was expecting. Basically, NetworkX is fundamentally node-oriented and the dates of meetings are edge properties, so you can't just subgraph nodes with a given date. This may mean I'll have to rethink the whole implementation. Bugger.

I'm also increasingly tempted to scrape the competition's meetings database into ScraperWiki as there doesn't seem to be any way of getting at it without the HTML wrapping. Oddly, although they've got the Department of Health's horrible PDFs scraped, they haven't got the Scottish Office although it's relatively easy, so it looks like this wouldn't be a 100% solution. However, their data cleaning has been much more effective - not surprising as I haven't really been trying. This has some consequences - I've only just noticed that I've hugely underestimated Oliver Letwin's gatekeepership, which should be 1.89 rather than 1.05. Along with his network degree of 2.67 (the eight highest) this suggests that he should be a highly desirable target for any lobbying you might want to do.

a dance to the music of confirmation bias

OKTrends has an amusing post, but what I like about it is that it's consilient with the process I defined here. My idea was that songs that were rated 5 might be good, but might also just be violently weird to the reviewer. By the same logic, the same must be true of the 1s. Assuming that my tastes aren't the same as the reviewer, the information in the reviews was whether the music was either mediocre, or potentially interesting. The output is here.

The OKTrends people seem to have rediscovered the idea independently looking at dating profiles - it's better to be ugly to some and beautiful to others than it is to be boringly acceptable to everybody.

three links about false positives

Via Bruce Schneier's, an interesting paper in PNAS on false positives and looking for terrorists. Even if the assumptions of profiling are valid, and the target-group really is more likely to be terrorists, it still isn't a good policy. Because the inter-group difference in the proportion of terrorists is small relative to the absolute scarcity of terrorists in the population, profiling means that you hugely over-sample the people who match the profile. Although it magnifies the hit-rate, it also magnifies the false positive rate, and because a search carried out on someone matching the profile is one not carried out elsewhere, it increases the chance of missing someone.

In fact, if you profile, you need to balance this by searching non-profiled people more often.

The operators of Deepwater Horizon disabled a lot of alarms in order to stop false alarms waking everyone up at all hours. Shock! In some ways, though, that was better than this story about a US hospital, from comp.risks. There, a patient died when an alarm was missed. Why? Too many alarms, beeps, and general noise, and people had turned off some devices' alarms in order to get rid of them.

Unlike Transocean, they had a solution - remove the off switches, because that way, they'll damn well have to listen. At least the oil people didn't think that would work. Of course, they didn't think that if your warning system goes off so often that nobody can sleep when nothing unusual is going on, there's something wrong with the system.

fast zombies shoot!

So the England Zombies are looking more like Fast Zombies again. If I've bored you by talking up James Milner, I'd like to take this opportunity to claim my bragging rights. Here's something interesting; back at the weekend, in the depths of self-loathing, the Obscurer published a table showing the teams with various statistics, including shots on goal. It struck me that England were looking rather good on that, and that the top four looked mostly like a plausible semi-final line up. So I've put together a spreadsheet ranking the teams by shots on target/matches played.



Data source here. Having fifa.com in my browser history makes me feel dirty for some reason.

That puts England 5th in the world - quarter finals again - but ahead of all the three possible opponents in the second round, Germany (7 on target/game vs. 7.333 - Google Spreadsheets is lax about sig figs), Ghana, and Serbia, and well ahead of the Netherlands and Italy. Further, out of the top four, Spain aren't looking a cert to qualify out of their group, and they have an even worse tradition of World Cup choking than we do. This may be daft, sunshine and beer optimism; but it's daft, sunshine and beer optimism with data.

Update: Well, would you look at that.

IPPR: I agree with...

The Institute for Public Policy Research has issued a report on the correlates of BNP membership and support (pdf).

Fascinatingly, they reckon that there is very little or no correlation between BNP support and key socio-economic indicators like GVA per capita, growth, unemployment, immigration, etc. It's as if a typical BNP supporter was, well, a case of free-floating extremism. (A dedicated swallower of fascism; an accident waiting to happen.)

Oddly enough, this replicates an earlier result.

The Nottingham University Politics blog has a more nuanced response, but I'm quite impressed by the fact that two analyses based on two different metrics of BNP support - votes in the IPPR study, membership in mine - converged on the same result.

science!

Here's something interesting. I grabbed the last 6 months' worth of national opinion polls from Wellsy's and graphed the Tory lead in percentage points. On the tiny chart below, you'll observe that the mean is 10 points; the hatched area shows one standard deviation each side of the mean, and I've plotted a linear trend through it. (You can see a full-size version of it here.)



The interesting bit; there are 21 polls, out of 158, that showed a Conservative lead of more than one standard deviation greater than the mean. All of them occurred before the 29th of January. There are 24 that showed a lead more than one standard deviation less than the mean. 20 out of 24 occurred since the 19th of February. What on earth could have happened between these dates?



The posters broke in a big way around the 19th of January and the second wave hit in early February. Clifford Singer deserves a knighthood for this.

(link)

a single net of conspiracy

Well, this is hardly surprising; the FBI was in the habit of pretending to be on a terrorism case every time they wanted telecoms traffic data. Their greed for call-detail records is truly impressive. Slurp! Unsurprisingly, the lust for CDRs and the telcos' eagerness to shovel them in rapidly got the better of their communications analysis unit's capacity to crunch them.

Meanwhile, Leah Farrell wonders about the problems of investigating "edge-of-network" connections. Obviously, these are going to be the interesting ones. Let's have a toy model; if you dump the CDRs for a group of suspects, 10 men in Bradford, and pour them into a visualisation tool, the bulk of the connections on the social network graph will be between the terrorists themselves, which is only of interest for what it tells you about the group dynamics. There will be somebody who gets a lot of calls from the others, and they will probably be important; but as I say, most of the connections will be between members of the group because that's what the word "group" means. If the likelihood of any given link in the network being internal to it isn't very high, then you're not dealing with anything that could be meaningfully described as a group.

By definition, though, if you're trying to find other terrorists, they will be at the edge of this network; if they weren't, they'd either be in it already, or else they would be multiple hops away, not yet visible. So, any hope of using this data to map the concealed network further must begin at the edge of the sub-network we know about. And the principle that the ability to improve a design occurs primarily at the interfaces - this is also the prime location for screwing it up also points this way.

But there's a really huge problem here. The modelling assumptions are that a group is defined by being significantly more likely to communicate among itself than with any other subset of the phone book, that the group is small relative to the world around it, and that it is boring; everyone has roughly similar phoning behaviour, and therefore who they call is the question that matters. I think these are reasonable.

The problem is that it's exactly at the edge of the network that the numbers of possible connections start to curve upwards, and that the density of suspects in the population falls. Some more assumptions; an average node talks to x others, with calls being distributed among them on a well-behaved curve. Therefore, the set of possibilities is multiplied by x for each link you follow outwards; even if you pick the top 10% of the calling distribution, you're going to fall off the edge as the false positives pile up. After three hops and x=8, we're looking at 512 contacts from the top 10% of the calling distribution alone.

In fact, it's probably foolish to assume that suspects would be in the top 10% of the distribution; most people have mothers, jobs, and the like, and you also have to imagine that the other side would deliberately try to minimise their phoning or, more subtly, to flatten the distribution by splitting their communications over a lot of different phone numbers. Actually, one flag of suspicion might be people who were closely associated by other evidence who never called each other, but the false positive rate for that would be so high that it's only realistically going to be hindsight.

Conclusions? The whole project of big-scale database-driven social network analysis is based on the wrong assumptions, which are drawn either from military signals intelligence or from classical policing. Military traffic analysis works because it assumes that the available signals are a subset of a much bigger total, and that this total is large compared to the world. This makes sense because that's what the battlefield of electronic warfare is meant to look like - cleared of civilian activity, dominated by one side or the other's military traffic. Working from the subset of enemy traffic that gets captured, it's possible to infer quite a lot about the system it belongs to.

Police investigation works because it limits the search space and proceeds along multiple lines of enquiry; rather than pulling CDRs and assuming the three commonest numbers must be suspects, it looks for suspects based on the witness and forensic evidence of the case, and then uses other sources of data to corroborate or refute suspicion.

To summarise, traffic analysis works on the assumption that there is an army out there. We can only see part of it, but we can make inferences about the rest because we know there is an army. Police investigation works on the observation that there has been a crime, and the assumption that probably, only a small number of people are possible suspects.

So, I'm a bit underwhelmed by projects like this. One thing that social network datamining does, undoubtedly, achieve is to create handsome data visualisations. But this is dangerous; it's an opportunity to mistake beauty for truth. (And they will look great on a PowerPoint slide!)

Another, more insidious, more sinister one is to reinforce the assumptions we went into the exercise with. Traffic-analysis methodology will produce patterns; our brains love patterns. But the surge of false positives means that once you get past the first couple of hops, essentially everything you see will be a false positive result. If you've already primed your mind with the idea that there is a sinister network of subversives everywhere, techniques like this will convince you even further.

Unconsciously, this may even be the purpose of the exercise - the latent content of Evan Kohlmann. At the levels of numbers found in telco billing systems, everyone will eventually be a suspect if you just traverse enough links.

Which reminded me of Evelyn Waugh, specifically the Sword of Honour trilogy. Here's his comic counterintelligence officer, Colonel Grace-Groundling-Marchpole:

Colonel Marchpole's department was so secret that it communicated only with the War Cabinet and the Chiefs of Staff. Colonel Marchpole kept his information until it was asked for. To date that had not occurred and he rejoiced under neglect. Premature examination of his files might ruin his private, undefined Plan. Somewhere, in the ultimate curlicues of his mind, there was a Plan.

Given time, given enough confidential material, he would succeed in knitting the entire quarrelsome world into a single net of conspiracy in which there were no antagonists, only millions of men working, unknown to one another, for the same end; and there would be no more war.

Want a positive idea? One reading of this and this would be that the failure of intelligence isn't a failure to collect or analyse information about the world, or rather it is, but it is caused by a failure to collect and analyse information about ourselves.

did you know Twitter stimulates Shatner's bassoon?

Quite ridiculous microtale about the head of MI6's wife being on Facebook. But what's this, from Patrick Mercer MP?

The Conservative MP Patrick Mercer, who chairs the counter-terrorism sub-committee, said the mistake had left the Sawers family "extremely vulnerable". Referring to Miliband's suggestion that the incident was not significant, Mercer said: "If that is the case why has the site being taken down?" He also pointed out that military chiefs had warned that the Taliban get 80% of their intelligence from Twitter and Facebook.

Can he really believe this? Eighty per cent? What percentage of users of either are located in Afghanistan? I'm going to stick a target on the wall and say it's much less than 1%, so this suggests that a very few people are very insecure indeed. Perhaps we could just ask the guy to knock it off, or post him to the Falklands?

I'd be surprised if 80 per cent of their intelligence didn't come from informers, friendly civilians reporting where our patrols go, if not more. Rather like it did in Northern Ireland. And Patrick Mercer of all people ought to be well aware of the possibilities...

He's got form for Chris Morris-esque nonsense, mind you; remember his role in the Glen Jenvey/Comedy Gladio affair? Some people are, indeed, very insecure indeed about the world of today, and it remains truly remarkable just what stuff a lot of MPs will happily read out to the camera without passing it through their brains. The question remains whether Facebook is a made-up Web site.

free our data, I suppose

Following on from the last post, we're unlikely to have funding to dose every school kid in Britain with radioactive markers and fMRI-scan them a term later to see how their neurons are getting on any time soon, even if you could get that past the ethics committee and the Nuclear Dread. So unless someone comes up with a field-expedient diagnostic test, we'll need some other way of assessing the problem. Which means that this annoyed me.

So some firm decided to try analysing the primary school SAT results better. They broke down the UK into much smaller units than Local Education Authorities or even schools - neighbourhoods of 300 people on average. They then classified them into 24 groups based on demographic and socio-economic indicators, looked at the average results for each group, and arrived at an expected score for each school based on the distribution of those groups in the school's intake. They then compared the actual results to see which schools were really doing better or worse.

And they got quite a lot of criticism for not using a database of pupils that...wait for it...the government won't let them use. This is a pity. Ever since Pierre Bourdieu, we've been well aware that there is much more to class than money. With all that data, we could do a lot of interesting things; we could, for example, use principal components analysis to establish objectively defined groups and see how well schools are doing that way. We could benchmark them against the Flynn effect, and I suspect quite a lot of schools would turn out just to be tracking the gradual uplift overall. But if we can't see the data we can't do anything.

in which we get down...to the unconscious!

So somebody reviewed 1,302 songs by the same number of bands, giving each one six words only.

But how to centrifuge this toxic dump? Clearly there was no possibility of scraping the page and wget-ing the lot; Sturgeon's Law (90% of everything is shit) applies to music as it does to few other things. I thought of trying to express my tastes in a set of criteria, that I might even implement in a python script, but on reflection this seemed to be too much like work, and anyway, it didn't really fit the aim. I wanted surprises, not confirmation.

Then I had an idea; what about applying some sort of statistical method? Yer man had given each song a rating between 1 and 5; as you know, Bob, if you ask people in a survey to rate something on a scale of 1 to 5, they will go for 3 far more often than you'd expect from a normal distribution, because it's the safe choice. But presumably the ones he gave a top rating to must have something.

And there were basically two ways a song could get into the bottom rank; either it was objectively arrant shite, or else it was incompatible with the other guy's tastes. Now, I have no idea what those are and no reason to assume they are anything like mine, so in fact, being one-starred could actually be a recommendation. Similarly, being top-rated could be either evidence of quality, or else just a matter of taste. And I had no reason to imagine either case was more likely. Further, the principle of management by exception was in my mind; the top and bottom 10% must be doing something right or wrong, so they're the ones to look at.

So I decided to ignore all the 2s and 3s and most of the 4s, and then make a selection from the ones that remained, based on unreason and hunch, and at least once on the basis that they came from Leeds.

And? I'm grinning with delight at the results, a pile of 31 MP3s of which 30 are by people I've literally never heard of and at least 28 are utterly great. Here's the really interesting bit, though: I can't tell which ones were 1s and which were 5s. Well, there is at least one exception to that, but as a rule, no, it is far from obvious. And why are so many fronted by women? This isn't something I'd noticed as a taste, although - horribly - I just remembered that my father owns a vast amount of vinyl by early 1970s hippy-chick singer-songwriters. Boxes and Nick Hornbyesque boxes of 'em. That's hardly characteristic of the list I came up with, but it is scary. Perhaps it's sampling bias - or maybe the quasi-automatic process got around my unconscious prejudices?

Blogging Rugby League: while you're playing it

Manly-Warringah RLFC's successful trip to the UK in the last few weeks, which saw them beat Leeds for the World Club Challenge, was assisted by an interesting piece of technology. All the players have been wearing networked GPS data loggers during the games, so Statto gets a live feed of data on precisely where they move, how fast, and what they're doing. And just how hard they go in; there's a three axis accelerometer in there too. It's the work of their conditioner Dean Robinson.

Aussie clubs have been very good with statistics for years; in the 1990s, the Brits were still very impressed with themselves for counting tackles while the Aussies were looking at how you could measure the energy battle and coach to tire the other side out. But this impresses even me.

Weirdly, in a sense their team is blogging all the time it's playing. Discussion ensues, over here. They used to say that you can smoke while playing a game but not while playing a sport, but then, the legendary French fullback Puig-Aubert used to bum fags off the fans and he was in the French World Cup winning side of 1951. It's probably true that you can blog while playing a game, etc, but as you can see, technological change is even getting rid of that distinction.

In a surveillance society, you can be a star blogger without even noticing.

Even More BNP Data...

Various people asked what would happen if I excluded London and Northern Ireland from the BNP analysis. Here's a table showing the R-squared for each factor, first for the whole data set and then excluding these two outliers. (After all, who needs statistical analysis to know those two are weird?)

Factor	R-Squared	R-Squared Excluding NI, London
Immigration	0.0364	0.0810
Emigration	0.0330	0.0460
Migration	0.0343	0.0843
Services % GDP	0.0639	0.0009
Industry % GDP	0.0885	0.0066
Agriculture % GDP	0.0659	0.1697
Long Term Unemployment	0.2078	0.0074
Unemployment %	0.1080	0.0235
Economic Growth %, 1991-2006	0.0782	0.0692
Density Change 1991-2006 %	0.0369	0.0035
Population Change %	0.0008	0.0160

I'm still not convinced there is any rational pattern here at all. Immigration is still astonishingly weak as a predictor of BNP membership; weirdly, economic growth is even weaker, and positive! (I'm feeling so prosperous...I'm going to join the BNP!) In fact, the only factor in the second set of numbers that has an effect measurable without going into three significant figures is the proportion of GDP accounted for by agriculture. Northern Ireland is both surprisingly agricultural (2.3% of GDP - 130% of the UK average) and unsurprisingly low in BNP members (0.0024 per 100), so we wouldn't have seen this earlier on.

The Thatcher legacy - long term unemployment as a percentage of all unemployment - was the strongest correlate with R-squared = 0.2078, but when you drop London and NI, it vanishes, as does unemployment in general.

Yet More BNP Data Analysis: Does Not Compute!

OK, so I've spent some time getting more data together on the correlates of BNP membership. I've created a table which contains the following metrics: population growth (%), change in population density (%), Gross Value Added(GVA) in 1991, 2006, change in GVA, % GVA growth, unemployment, long-term unemployment as a % of total unemployment, the shares of GDP accounted for by agriculture, industry, and services, total immigration between 1994 and 2002 per capita, total emigration per capita for the same period, total migration per capita, and BNP members per 100 citizens.

And you know what? I was expecting to find a correlation with the economic variables. I had a theory that long-term, Thatcher legacy unemployment, especially, would be a strong correlate of BNP recruitment. But nothing correlates. None of those metrics have any predictive power. Have a look at this.



Immigration per head is up the Y-axis, bigotry on the X-axis, and the data points are scaled by the unemployment rate. It doesn't seem to track any of these variables at all; I urge you to visit the visualisation home page, where you can try the different data series for yourself.

This strongly suggests that some completely different force is at work; perhaps BNP membership is driven by something else entirely. It could be the distribution of social authoritarian tendencies in the population, as Robert Altemeyer theorises. Or alternatively, it could just be that a gratifyingly small percentage of people are completely fucking stupid and pig-ignorant, that this is normally distributed in the population, and it's essentially a matter of chance what pig-ignorant fucking stupidity they get up to.

It's probably worthwhile pointing out that the average concentration of BNP members is 0.0203 per 100 citizens and the standard deviation is 0.0116. So with the sole exception of Northern Ireland, 1.54 standard deviations below the mean and therefore staggering towards the edge of the 90% confidence interval, the variation between regions is entirely explicable by chance - strong backing for the wanker theory.

(For some reason, this post has started to remind me of Donald Crowhurst's campaign leaflet, which bore the headline "YOU MAY THINK YOU ARE LOGICAL - BUT DARE YOU TAKE THIS TEST?" Inside was a sort of flowchart designed to explain logically why everyone should vote Liberal.)

The Swamping Myth

You will hear all kinds of people in authority say that immigration, or population growth, is causing people to turn into racists and vote BNP, either just because (the rightwing version) or because of "pressure" on public services (the Decent Left version).Therefore, they usually say, we need a stingier immigration policy. If you're reading this, you probably think this is crap. But now, I can prove this scientifically. Thanks to the leaked BNP membership list, we can empirically measure how many people are active racists, active and committed enough that they joined a political party and paid a subscription. Using the data by county, I established a table that matched the UK regions.

Now, if immigration or population growth really is causing people to go fascist, we'd expect to find a correlation between population growth and BNP membership. Or, perhaps, we might find that places that are losing population are economically depressed and hence susceptible. A further detail might be changes in population density; becoming more urban might lead to a perception of being "swamped", or becoming more rural/exurban might lead to one of isolation. So I drew up a table of population growth from 1991 to 2006, change in density for the same period, and BNP members per 100 citizens.

Here are the results.



Population growth is on the Y axis, bigots on the X axis; the size of each dot represents the change in density. There is no correlation whatsoever. Anyone who tells you this story is talking nonsense.

don't compromise, visualise

Oh yes, gleeful leftie hacker tournament after the BNP did a 0.16 megarecord datafart. My effort contains absolutely no personally-identifying data; it's made with this guy's count by region and population data from National Statistics, to show the number of BNP activists per 100 citizens in each UK region. People kept asking for that kind of information, so I made it. Note that the g-spreadsheet guy used classifications that don't quite map to NatStats' regions, so I decided to assume that his "South Central England" was the West Midlands and "Midlands" was the East Midlands, and total Yorks & Humber and North-East to match his "North East England".


Update: Well, in the end I used his numbers by county to create a table that matches the regions. Here's a new and correct visualisation that shows Yorkshire where it should be, in the lead. Ernst Wilhelm Bohle lives!

A Truth Moment at the CRB

According to the BBC, the Home Office really, really doesn't get the basic truth that 0.01% of a really big number is quite a big number. The Torygraph reported that the Criminal Records Bureau had mistakenly told its customers between February 2007 and February 2008 that some 680 people had criminal records when in fact they had none. The Home Office's response:

The Home Office said CRB has a 99.98% accuracy rate in vetting people working with children and vulnerable adults.

Indeed. I keep saying this; 99.98% accuracy, which is the politician's way of saying a 0.02% failure rate, is only good enough if 0.02% of the total isn't a large number. It must seem silly to people outside the telecoms business that we go on about 99.999% reliability. But that is a percentage of up to hundreds of millions of calls and signalling events.

Fortunately, there are some numbers in the story. The Home Office claims that 80,000 (a round number, but we've got nothing else to go on) people were prevented from taking up posts involving "vulnerable people"; there's no way of telling whether this means only ones involving "vulnerable people", only ones where a job offer was withdrawn, or just the total CRB checks that came up positive, and there's no telling what period of time it refers to. If it was the total for 2007-2008, that means the chance of a positive CRB check being a false positive is 0.85 per cent (99.15 per cent in contractorspeak). And we *haven't* even considered the false negatives....

So where's your 0.02 per cent now? Naturally, it's possible that the 80,000 covers more than one year...but hold on. If there were many more, some such figure recurring every year, then this suggests the actual numbers are even worse. The CRB has been going since, what, 2002? 13,333 refusals a year on average. We know the 680 false positives are for just one year; which would make it a 5.1% false positive rate for 2007-08. (That's 94.9% in contractorspeak.) So, the Home Office's figures cannot possibly be right; it's impossible to have a negative number of false negatives, so we *know* that the CRB does not provide 99.98% accuracy. Surely this means the Government should be suing Capita or whoever?

Bashful Brownites

Anthony Wells is arguing about what percentage of don't knows in the polls for the Crewe & Nantwich byelection are actually Labour voters who feel embarrassed to say so - analogous to the "shy Tories" of the 1990s. I would think the number is considerable.

During my candidacy in Egham Hythe, I knocked on and got an answer from around 100 doors. In the event, there were roughly 600 Conservative votes, 300 Labour, and 200 Liberals counted. My own canvassing numbers logged 22 Liberal, 28 Tory, 1 BNP, 1 UKIP, 31 Don't Know among those who said they would vote...and 4 Labour. Yes - four. 5.19% of the total answers, as against 30% of the vote. Redoing the sums, assuming the same pattern for the non-answerers, predicted 51% Tory, 40% Liberal, 7.3% Labour (note this is a two-councillor ward, so votes for candidates must be divided by 2); the event was more like 60% Tory, 30% Liberal, 20% Labour, 10% nutters and spoilt ballots.

Conclusion: There are a *lot* of quiet Labour voters out there.

Ou est la masse de manoeuvre?

It's been said, I think by Alanbrooke, that strategy consists in the proper handling of reserves. Looking at the situation with this framework, what can we say about Iraq? It's now been a year since the announcement of "The Surge", the deployment of the US Army's strategic reserve in Iraq. And what's happened?

First, it helps to think about an army as a flow, not a stock concept. The size of an army is really the size of the force it can maintain for a given period of time; this is a function, mostly, of either the provision of soldiers as replacements or the rotation of units out of the theatre of war. Eventually, as in the second world war, even a replacement-based army has to take whole units out of the line, but let's keep it simple.

The surge was accomplished, essentially, by boosting the flow temporarily; bringing forward deployments from this year and the next. This made it possible to temporarily increase the strength by 28,500 men; but the crucial point here is that this is a borrowing from the future. As the units that were planned to rotate back to the US do so, which they have begun doing, they won't be replaced; the units that were to replace them have already been sent and will in their turn complete their tours of duty. Not only will the extra troops leave; the force in place will itself weaken. This can only be avoided if the US Army decides not to reconstitute its strategic reserve. The peak was 182,000 troops in October; we're already down by almost 10,000, or in other words a division equivalent.

This would be largely academic if committing the reserve had led to decisive results. But it has not; yes, there have been three reasonable months by 2006-2007 standards, but this is a claim that requires close examination. The press has effectively taken a holiday since the summer, and the US military PR men have become very keen to quote percentages ("Violence down 60%" - down 60% on what exactly?) but never any absolute numbers. Fortunately the Brookings Iraq Index is still going.

As far as their estimate of civilian casualties goes, the peak month was November, 2006; almost a third of the fall was between then and January. The rate of enemy action was at an all-time high as late as June, 2007, and was still running at 3,000 attacks a month in September. The five worst months for multiple-fatality bombings were all post-surge. The chief evidence for surge effectiveness is the drop in US casualties since August, 2007; that was a pretty bad month itself (84), but was also the moment of the Sadrist ceasefire. It's also noticeable that the rate of attacks on oil and gas installation went to near-zero in August as if a valve had been screwed shut; August was one of the worst months for oil production, but it has noticeably increased. However, oil products supply in Iraq is still at just over two-thirds of requirements; actually worse than during the worst period of 2006.

Electricity production is still almost one-third below target, and it only exceeded the figures for last year after the Sadrist ceasefire; it's also worth noting that the figures exhibit a strong seasonal variation, and have improved every winter since 2003 only to decline again in the summer. (The turn of the year is also usually a low point in casualties.) Further, it's worth noticing that the frequency distribution is not especially normal; 18 months out of 58 are over 1.4 standard deviations away from the mean, mostly on the bad side (the split is 5 low/12 high). In a nonnormal distribution you'd expect to find yourself that far from the mean at most one-third of the time, which is precisely what has happened.



Regression to the mean has no divisions, but it's notable that all the worst months for US casualties are associated with a Sadr crisis; his six-month ceasefire expires roughly now. The US Army has used its strategic reserves and not achieved a decision; this is historically a very dangerous strategic moment. If you examine that Brookings pdf, you'll note that it includes some order-of-battle details; currently, some 7 US brigades are disposed around Baghdad and 3 more around the southern suburbs, and another six across the north, with one reinforced brigade in Anbar. There is currently a Polish battalion group at Diwaniyah and a National Guard infantry brigade based on Kut; nothing between them, and nothing before you get to the British brigade camped outside Basra.

It's Sadr's move; it always has been. And Diyala is still the battlefield; and the guerrillas still know that we're coming. Read Phil "Intel Dump" Carter. But what the hell; snark on this issue has been outsourced to Jamie Kenny.

For extra TYR points, it seems that there was a major clash between the Iraqi Army and the Sadrists in Basra over Christmas (during which the IA discovered a cache including a *drone*). This is not going to be conducive to the renewal of the Sadrist ceasefire.

Some data points

OK, so by chance we have some real data to put into the sums in this post. The head of MI5 has just announced that we should all be very scared, because he reckons there may be 2,000 people in Britain who pose a threat to national security because of their support for terrorism.

So let's run the Terroriser. 59 million people; 2,000 terrorists. So there's a 0.0034% chance of any given citizen being a terrorist. Remember that the Terroriser will catch 99 per cent of the real terrorists - so that's all but 20 terrorists. Now, the Terrorist will also miss 98 per cent of the non-terrorists - but that means we'll get some 1,180,000 false positives. 1,980 terrorists plus 1,180,000 false positives = 1,181,980 suspects. (1,980/1,181,980)x100=0.1675155. There is a 0.167 per cent chance that any one of the suspects is a terrorist.

And there are still 20 terrorists out there; easily enough for a major terrorist attack. Now consider this hilarious report; apparently the FBI mined supermarket sales figures in the hope that sales of falafels would indicate the presence of Iranian terrorists! As well as, ah, Israelis, presumably. Note the involvement of half-arsed fearmonger Steven Emerson, and also old TYR butt Yossef Bodansky.