I need to be more damn effective and better organised.
If it helps at all, I just bought the O'Reilly books on Android development and on JavaScript, as it's getting increasingly ridiculous to not know it.
Blogging a noisy and socialistic view on politics, security, and whatever may take my fancy. "All the world now is in the Ranting humour" - Samuel Sheppard, 1647
Sunday, May 22, 2011
OpenTech washup, and an amended result
So it was OpenTech weekend. I wasn't presenting anything (although I'm kicking myself for not having done a talk on Tropo and Phono) but of course I was there. This year's was, I think, a bit better than last year's - the schedule filled up late on, and there were a couple of really good workshop sessions. As usual, it was also the drinking conference with a code problem (the bar was full by the end of the first session).
Things to note: everyone loves Google Refine, and I really enjoyed the Refine HOWTO session, which was also the one where the presenter asked if anyone present had ever written a screen-scraper and 60-odd hands reached for the sky. Basically, it lets you slurp up any even vaguely tabular data and identify transformations you need to clean it up - for example, identifying particular items, data formats, or duplicates - and then apply them to the whole thing automatically. You can write your own functions for it in several languages and have the application call them as part of the process. Removing cruft from data is always incredibly time consuming and annoying, so it's no wonder everyone likes the idea of a sensible way of automating it. There's been some discussion on the ScraperWiki mailing list about integrating Refine into SW in order to provide a data-scrubbing capability and I wouldn't be surprised if it goes ahead.
Tim Ireland's presentation on the political uses of search-engine optimisation was typically sharp and typically amusing - I especially liked his point that the more specific a search term, the less likely it is to lead the searcher to a big newspaper website. Also, he made the excellent point that mass audiences and target audiences are substitutes for each other, and the ultimate target audience is one person - the MP (or whoever) themselves.
The Sukey workshop was very cool - much discussion about propagating data by SMS in a peer-to-peer topology, on the basis that everyone has a bucket of inclusive SMS messages and this beats paying through the nose for Clickatell or MBlox to send out bulk alerts. They are facing a surprisingly common mobile tech issue, which is that when you go mobile, most of the efficient push-notification technologies you can use on the Internet stop being efficient. If you want to use XMPP or SIP messaging, your problem is that the users' phones have to maintain an active data connection and/or recreate one as soon after an interruption as possible. Mobile networks analogise an Internet connection to a phone call - the terminal requests a PDP (Packet Data Profile) data call from the network - and as a result, the radio in the phone stays in an active state as long as the "call" is going on, whether any data is being transferred or not.
This is the inverse of the way they handle incoming messages or phone calls - in that situation, the radio goes into a low power standby mode until the network side signals it on a special paging channel. At the moment, there's no cross-platform way to do this for incoming Internet packets, although there are some device-specific ways of getting around it at a higher level of abstraction. Hence the interest of using SMS (or indeed MMS).
Their other main problem is the integrity of their data - even without deliberate disinformation, there's plenty of scope for drivel, duplicates, cockups etc to get propagated, and a risk of a feedback loop in which the crap gets pushed out to users, they send it to other people, and it gets sucked up from Twitter or whatever back into the system. This intersects badly with their use cases - it strikes me, and I said as much, that moderation is a task that requires a QWERTY keyboard, a decent-sized monitor, and a shirt-sleeve working environment. You can't skim-read through piles of comments on a 3" mobile phone screen in the rain, nor can you edit them on a greasy touchscreen, and you certainly can't do either while looking out that you don't get hit over the head by the cops.
Fortunately, there is no shortage of armchair revolutionaries on the web who could actually contribute something by reviewing batches of updates, and once you have reasonably large buckets of good stuff and crap you can use Bayesian filtering to automate part of the process.
Francis Davey's OneClickOrgs project is coming along nicely - it automates the process of creating an organisation with legal personality and a constitution and what not, and they're looking at making it able to set up co-ops and other types of organisation.
I didn't know that OpenStreetMap is available through multiple different tile servers, so you can make use of Mapquest's CDN to serve out free mapping.
OpenCorporates is trying to make a database of all the world's companies (they're already getting on for four million), and the biggest problem they have is working out how to represent inter-company relationships, which have the annoying property that they are a directed graph but not a directed acylic graph - it's perfectly possible and indeed common for company X to own part of company Y which owns part of company X, perhaps through the intermediary of company Z.
OpenTech's precursor, Notcon, was heavier on the hardware/electronics side than OT usually is, but this year there were quite a few hardware projects. However, I missed the one that actually included a cat.
What else? LinkedGov is a bit like ScraperWiki but with civil servants and a grant from the Technology Strategy Board. Francis Maude is keen. Kumbaya is an encrypted, P2P online backup application which has the feature that you only have to store data from people you trust. (Oh yes, and apparently nobody did any of this stuff two years ago. Time to hit the big brown bullshit button.)
As always, the day after is a bit of an enthusiasm killer. I've spent part of today trying to implement monthly results for my lobby metrics project and it looks like it's much harder than I was expecting. Basically, NetworkX is fundamentally node-oriented and the dates of meetings are edge properties, so you can't just subgraph nodes with a given date. This may mean I'll have to rethink the whole implementation. Bugger.
I'm also increasingly tempted to scrape the competition's meetings database into ScraperWiki as there doesn't seem to be any way of getting at it without the HTML wrapping. Oddly, although they've got the Department of Health's horrible PDFs scraped, they haven't got the Scottish Office although it's relatively easy, so it looks like this wouldn't be a 100% solution. However, their data cleaning has been much more effective - not surprising as I haven't really been trying. This has some consequences - I've only just noticed that I've hugely underestimated Oliver Letwin's gatekeepership, which should be 1.89 rather than 1.05. Along with his network degree of 2.67 (the eight highest) this suggests that he should be a highly desirable target for any lobbying you might want to do.
Things to note: everyone loves Google Refine, and I really enjoyed the Refine HOWTO session, which was also the one where the presenter asked if anyone present had ever written a screen-scraper and 60-odd hands reached for the sky. Basically, it lets you slurp up any even vaguely tabular data and identify transformations you need to clean it up - for example, identifying particular items, data formats, or duplicates - and then apply them to the whole thing automatically. You can write your own functions for it in several languages and have the application call them as part of the process. Removing cruft from data is always incredibly time consuming and annoying, so it's no wonder everyone likes the idea of a sensible way of automating it. There's been some discussion on the ScraperWiki mailing list about integrating Refine into SW in order to provide a data-scrubbing capability and I wouldn't be surprised if it goes ahead.
Tim Ireland's presentation on the political uses of search-engine optimisation was typically sharp and typically amusing - I especially liked his point that the more specific a search term, the less likely it is to lead the searcher to a big newspaper website. Also, he made the excellent point that mass audiences and target audiences are substitutes for each other, and the ultimate target audience is one person - the MP (or whoever) themselves.
The Sukey workshop was very cool - much discussion about propagating data by SMS in a peer-to-peer topology, on the basis that everyone has a bucket of inclusive SMS messages and this beats paying through the nose for Clickatell or MBlox to send out bulk alerts. They are facing a surprisingly common mobile tech issue, which is that when you go mobile, most of the efficient push-notification technologies you can use on the Internet stop being efficient. If you want to use XMPP or SIP messaging, your problem is that the users' phones have to maintain an active data connection and/or recreate one as soon after an interruption as possible. Mobile networks analogise an Internet connection to a phone call - the terminal requests a PDP (Packet Data Profile) data call from the network - and as a result, the radio in the phone stays in an active state as long as the "call" is going on, whether any data is being transferred or not.
This is the inverse of the way they handle incoming messages or phone calls - in that situation, the radio goes into a low power standby mode until the network side signals it on a special paging channel. At the moment, there's no cross-platform way to do this for incoming Internet packets, although there are some device-specific ways of getting around it at a higher level of abstraction. Hence the interest of using SMS (or indeed MMS).
Their other main problem is the integrity of their data - even without deliberate disinformation, there's plenty of scope for drivel, duplicates, cockups etc to get propagated, and a risk of a feedback loop in which the crap gets pushed out to users, they send it to other people, and it gets sucked up from Twitter or whatever back into the system. This intersects badly with their use cases - it strikes me, and I said as much, that moderation is a task that requires a QWERTY keyboard, a decent-sized monitor, and a shirt-sleeve working environment. You can't skim-read through piles of comments on a 3" mobile phone screen in the rain, nor can you edit them on a greasy touchscreen, and you certainly can't do either while looking out that you don't get hit over the head by the cops.
Fortunately, there is no shortage of armchair revolutionaries on the web who could actually contribute something by reviewing batches of updates, and once you have reasonably large buckets of good stuff and crap you can use Bayesian filtering to automate part of the process.
Francis Davey's OneClickOrgs project is coming along nicely - it automates the process of creating an organisation with legal personality and a constitution and what not, and they're looking at making it able to set up co-ops and other types of organisation.
I didn't know that OpenStreetMap is available through multiple different tile servers, so you can make use of Mapquest's CDN to serve out free mapping.
OpenCorporates is trying to make a database of all the world's companies (they're already getting on for four million), and the biggest problem they have is working out how to represent inter-company relationships, which have the annoying property that they are a directed graph but not a directed acylic graph - it's perfectly possible and indeed common for company X to own part of company Y which owns part of company X, perhaps through the intermediary of company Z.
OpenTech's precursor, Notcon, was heavier on the hardware/electronics side than OT usually is, but this year there were quite a few hardware projects. However, I missed the one that actually included a cat.
What else? LinkedGov is a bit like ScraperWiki but with civil servants and a grant from the Technology Strategy Board. Francis Maude is keen. Kumbaya is an encrypted, P2P online backup application which has the feature that you only have to store data from people you trust. (Oh yes, and apparently nobody did any of this stuff two years ago. Time to hit the big brown bullshit button.)
As always, the day after is a bit of an enthusiasm killer. I've spent part of today trying to implement monthly results for my lobby metrics project and it looks like it's much harder than I was expecting. Basically, NetworkX is fundamentally node-oriented and the dates of meetings are edge properties, so you can't just subgraph nodes with a given date. This may mean I'll have to rethink the whole implementation. Bugger.
I'm also increasingly tempted to scrape the competition's meetings database into ScraperWiki as there doesn't seem to be any way of getting at it without the HTML wrapping. Oddly, although they've got the Department of Health's horrible PDFs scraped, they haven't got the Scottish Office although it's relatively easy, so it looks like this wouldn't be a 100% solution. However, their data cleaning has been much more effective - not surprising as I haven't really been trying. This has some consequences - I've only just noticed that I've hugely underestimated Oliver Letwin's gatekeepership, which should be 1.89 rather than 1.05. Along with his network degree of 2.67 (the eight highest) this suggests that he should be a highly desirable target for any lobbying you might want to do.
Labels:
networks,
programming,
protest,
Python,
statistics,
Tories
Sunday, May 15, 2011
Links
The National Rail website has borrowed the sensible URIs from traintimes.org.uk. Although it does seem a bit of a miss that there's no shortener like - say - tra.in, as those are by definition quite long strings.
The website that identifies the satellite a dish is pointing at, or tells you where to point the dish for a given satellite. Includes a mobile augmented-reality app!
The Vatican has IPv6, and they peer with major Italian ISPs.
Blackwater, after the party. Curiously the entire piece doesn't mention that the UAE military operates Mirage 2000s and such things, but it still makes it fairly clear that Erik Prince's new job in Dubai is basically providing foreign goons to beat up the subcontinental construction workers if they make any trouble.
An interview about Piggipedia.
You know there's been a revolution when people enjoy a talk by Alex Callinicos.
The website that identifies the satellite a dish is pointing at, or tells you where to point the dish for a given satellite. Includes a mobile augmented-reality app!
The Vatican has IPv6, and they peer with major Italian ISPs.
Blackwater, after the party. Curiously the entire piece doesn't mention that the UAE military operates Mirage 2000s and such things, but it still makes it fairly clear that Erik Prince's new job in Dubai is basically providing foreign goons to beat up the subcontinental construction workers if they make any trouble.
An interview about Piggipedia.
You know there's been a revolution when people enjoy a talk by Alex Callinicos.
Don't read Michael Lewis articles, do check if he's still writing
So that Michael Lewis - you know, the one who writes articles about how stupid Icelanders were. Via The Oil and the Glory, here's what he was saying in January, 2007, just as the last bulls in the US housing market's dizziest bubbles finally cracked and the big plummet began. Here it is.
This is a case in point of Daniel Davies's contention that while forecasting the future is great, forecasting the recent past is almost as good and much more likely to be successful. That is to say, it's not just enough to have the information, it's also important to draw consequences from it and react to them. This is why John Boyd held that the most important element of his Observation-Orientation-Decision-Action loop was the Orientation.
In this case, Lewis was well aware that a massive housing market bust was actually in progress, that the oil price was very high, and that a range of generally weird things had happened. But he didn't integrate this information into his broader situational awareness - he noted that nothing had exploded yet and reached for a variety of rationalisations, notably confirmation-bias, to explain why nothing would happen.
However, for his own part, he actually responded more effectively. He spent quite a lot of this article whining about people who weren't taking risks and wondering why nobody was going short if things were so bad. Well, in fact somebody was going short the US housing market. (Logically, somebody had to be short if so many people were so long - someone had to be taking the other end of those trades, as in fact he said in as many words.) And Michael Lewis later wrote a book about him, making a considerable profit.
Obviously, he was in a position to wait and see what happened, and then write the book either about the Great Bull Run of 2008 or the Great Crash. That helps. From a Boydian point of view, it is critical to all strategy that you keep your options open as far as possible and avoid being forced into reacting on the other side's time-table. It's also true, though, that the precondition of not being bankrupt in 2008 and therefore being able to take opportunities as they came was not taking risks in 2007, perhaps by doing a well-paying writing gig at Bloomberg in which you moan endlessly about people who aren't taking enough risks.
In fact, anyone looking for economic advice from Lewis in 2007 would have been well advised to ignore everything he said and instead study what he was actually doing.
But the most striking thing about the growing derivatives markets is the stability that has come with them. More than eight years ago, after Long-Term Capital Management blew up and lost a few billion dollars, the Federal Reserve had to be wheeled in to save capitalism as we know it.
Last year Amaranth Advisors blew up, lost more than LTCM, and the financial markets hardly batted an eyelash. ``The financial markets in 2007,'' some member of the global economic elite might have said but didn't, ``are astonishingly robust. They seem to be working out how to absorb and distribute risk more intelligently than any member of the global economic elite could on his own.'' Once the laughter subsided -- and someone took down his name to make sure he didn't make next year's guest list -- he might go on to point out that in spite of a great deal of political turmoil the markets have remained calm.
The Sarbanes-Oxley Act sticks a wrench in the American market for initial public offerings, and the capital-raising business simply removes itself to London and Hong Kong. Thailand installs capital controls and the markets force it to reverse its policy, virtually overnight -- again with nary a ripple. The Brazilian real is now less volatile than the Swiss franc; Botswana's debt is now more highly rated than Italy's. Oil prices double, the U.S. housing market tanks -- no matter what happens, financial markets adjust quickly and without hysteria.
This is a case in point of Daniel Davies's contention that while forecasting the future is great, forecasting the recent past is almost as good and much more likely to be successful. That is to say, it's not just enough to have the information, it's also important to draw consequences from it and react to them. This is why John Boyd held that the most important element of his Observation-Orientation-Decision-Action loop was the Orientation.
In this case, Lewis was well aware that a massive housing market bust was actually in progress, that the oil price was very high, and that a range of generally weird things had happened. But he didn't integrate this information into his broader situational awareness - he noted that nothing had exploded yet and reached for a variety of rationalisations, notably confirmation-bias, to explain why nothing would happen.
However, for his own part, he actually responded more effectively. He spent quite a lot of this article whining about people who weren't taking risks and wondering why nobody was going short if things were so bad. Well, in fact somebody was going short the US housing market. (Logically, somebody had to be short if so many people were so long - someone had to be taking the other end of those trades, as in fact he said in as many words.) And Michael Lewis later wrote a book about him, making a considerable profit.
Obviously, he was in a position to wait and see what happened, and then write the book either about the Great Bull Run of 2008 or the Great Crash. That helps. From a Boydian point of view, it is critical to all strategy that you keep your options open as far as possible and avoid being forced into reacting on the other side's time-table. It's also true, though, that the precondition of not being bankrupt in 2008 and therefore being able to take opportunities as they came was not taking risks in 2007, perhaps by doing a well-paying writing gig at Bloomberg in which you moan endlessly about people who aren't taking enough risks.
In fact, anyone looking for economic advice from Lewis in 2007 would have been well advised to ignore everything he said and instead study what he was actually doing.
no matter who the government votes for...the people still get in
This is incredibly great, and will instantly catapult you into the top 3% of the information distribution on top-level Chinese politics. One for the RSS queue, even if his views on economics are very much what you might expect.
Acid Helvetica titles do not make you a decent person
(Alt. title: Look, it's like a society but smaller!)
OK, so you may remember the case of Startupbritain.org, a well-publicised eye-catching initiative that turned out to be little better than spam paid for with a government grant to some whose-kid-are-you types. It seems that the whole pointless wankabout was kicked off when one of the WKAYers cornered poisonous old Thatcherite gargoyle Lord Young and pitched his eye out.
Ah well, the Big Society was always going to be a happy hunting ground for grantsmanship, wanktanking, and various other kinds of ligging and general availability enterpreneurship. Small business grant programs - which it essentially is - are notoriously vulnerable to fraud and general dodgy dealing.
And then it came to the MySociety listserv. So this message plunked into the trap last Monday. In it, a thing called "Sidekick Studios" offers to hire some software developers. Note - because this will be important later on - that they didn't make any bones at all that they were making an offer of employment. (A shorter version of this post is in the list archive.)
Fair enough. The first one is for a Ruby on Rails developer with 3-5 years' experience, for an 8 week consultancy gig at their headquarters in London Bridge, working alongside a mobile developer and a user-interface designer on a project for an unnamed client. Nothing to see here.
But here's the problem.
To be honest, it was the peerlessly idiotic "concepts" that caught my eye. "Wife Swap for your gran". Had Nathan Barley been contracted to write the requirements statement? Or was the whole thing a reality-TV show? Had I stumbled upon a new Chris Morris project? We could read more "hear", and some of us did.
OK...well, it would appear from the next paragraph that the team is us. Actually, it would be more accurate to say that part of the team is us - the 10 ground crew have apparently been recruited already. It just remains to find Laika and Strelka the space dogs.
Note that the software jobs are additional to this. So you're being offered the chance to implement..."Wife Swap for your gran"...over three months, and if you succeed you get to keep it. And they'll pay your salary. Or maybe not - let's skip ahead here...
Note that they have no interest in any ideas you may have:
and reserve the right to terminate you at any moment:
So even if the project is a success, in fact, there's no guarantee of getting the job, contrary to prior statements, and no guarantee of any pay even if you get it. And there is a lot of horrible macho Alan Sugar rhetoric and passive-aggressive expectations-wank.
Indeed it does, but not so much that they're willing to offer anything lame like a job. As previously noted, they're not actually promising any money or any commitment of any kind, but they do keep using the word job:
Ahem? What was that again?
Now, I mentioned that the projects are non-profit and are to stay that way. Fine. This doesn't, however, mean that Sidekick itself is non-profit. In fact it's a commercial company, registration no. 6707987, and its customer in this project is, well, us. Specifically, they've scored a grant from the UK Technology Strategy Board.
So, to summarise: You get to work your arse off trying to make some daft idea like "Wife Swap for grans" fly. For this you get a suspiciously unspecified sum of money. You may be terminated without cause at any moment. After three months, you may get a job, you may get the same job but without the money, or you may get "no tea and walk home", depending on no conditions that anyone is willing to state at the outset. You may not profit from the business, no matter how well it may do, but Sidekick's directors get theirs whatever happens. And we're all paying for this exploitative, cynical, spammy, hilariously ill-thought out shite.
Further, and just in case anyone thinks I'm biased in any way, does anyone else think The Guardian could perhaps make its pay-for-play Guardian Professional advertorial and events operation look and feel distinct from the actual newspaper's website?
Why am I talking about this?
Well, Sidekick also recommended "a Guardian article" to us - this one. Now, unless you follow Fleet Street politics closely you'd probably think that's just another Guardian URI. But the key bit is that it's in their "Social Enterprise Network", one of their "Professional Networks". Guardian Professional's front page is here. It's possible that there is some difference between Guardian Professional Networks and Guardian Professional, in which case it might be more professional to make this obvious. But I suspect that this piece is advertorial.
It is, however, informative.
Those civil servants who have been made redundant - they collected their redundancy money! The bastards.
Indeed, just like last time. Perhaps it would be better not to do the privatisation and mass sackings in the first place?
I think this might have been improved by an admission that Sidekick Studios is in fact a private sector, for profit, company. What offends me about this is that they are quite openly trying to help push public servants out of real jobs with pensions and union recognition and that stuff and into the hyper-precarious code-for-pizza status we just outlined. Come to think of it, if you can't get hipster interns living off the Bank of Mum & Dad you might as well try sacked civil servants living off the redundancy money.
But then, what else would you expect from people who want to replace the poor sods working in local authority social care with "Wife Swap for your gran" and the police with "Foursquare for criminals", all held together with sales-training day bullshit and shameless volunteer-mining?
OK, so you may remember the case of Startupbritain.org, a well-publicised eye-catching initiative that turned out to be little better than spam paid for with a government grant to some whose-kid-are-you types. It seems that the whole pointless wankabout was kicked off when one of the WKAYers cornered poisonous old Thatcherite gargoyle Lord Young and pitched his eye out.
Ah well, the Big Society was always going to be a happy hunting ground for grantsmanship, wanktanking, and various other kinds of ligging and general availability enterpreneurship. Small business grant programs - which it essentially is - are notoriously vulnerable to fraud and general dodgy dealing.
And then it came to the MySociety listserv. So this message plunked into the trap last Monday. In it, a thing called "Sidekick Studios" offers to hire some software developers. Note - because this will be important later on - that they didn't make any bones at all that they were making an offer of employment. (A shorter version of this post is in the list archive.)
"*We’re Sidekick – And We’re looking to hire 3 Developers…We're a social innovation organisation - we develop web and mobile tools to tackle social issues and public services in a different way. We have 3 different jobs – please take a read and get in touch if the roles spark interest
Fair enough. The first one is for a Ruby on Rails developer with 3-5 years' experience, for an 8 week consultancy gig at their headquarters in London Bridge, working alongside a mobile developer and a user-interface designer on a project for an unnamed client. Nothing to see here.
But here's the problem.
*2 x Creative technical leads / senior developers *
Project – “SS3”
*Key info: *
· 3 month contract – maybe more
· Social innovation projects
a) Social Care Swap “wife swap for your gran”
b) Youth Justice Game Project “4 square for criminals”
· Experienced and creative developer to build proof of concept for an
innovative social project
· Start up experience preferred
· No language preference – but solid front and back end web-dev skills
required - whatever environment you’re used to
· Work in small dynamic team
· Based at Sidekick studios, London Bridge
· Read more hear http://sidekickstudios.net/ss3/
To be honest, it was the peerlessly idiotic "concepts" that caught my eye. "Wife Swap for your gran". Had Nathan Barley been contracted to write the requirements statement? Or was the whole thing a reality-TV show? Had I stumbled upon a new Chris Morris project? We could read more "hear", and some of us did.
SS3 is an experimental technology and design incubator to create new types of public services. We've assembled a team of 10 people, with diverse skills across research, design, technology, social venturing and commerce. Over 3 months, the team will work together out of a studio in London Bridge to launch 3 social businesses, each with the potential to deliver sustainable, social impact. And rock the world.
OK...well, it would appear from the next paragraph that the team is us. Actually, it would be more accurate to say that part of the team is us - the 10 ground crew have apparently been recruited already. It just remains to find Laika and Strelka the space dogs.
We're offering 3 people the chance to become the resident entrepreneur for each venture. And have their salary paid for 3 months. And have the opportunity to become the full-time Managing Director of the venture at the end of the SS3 experience.
Note that the software jobs are additional to this. So you're being offered the chance to implement..."Wife Swap for your gran"...over three months, and if you succeed you get to keep it. And they'll pay your salary. Or maybe not - let's skip ahead here...
How much will I get paid? Good question. And we're going to duck it. Sort of. We are not saying exactly because we might pay people differently depending on their level of experience. We can say the salary will be more than enough to live off in London, pay exorbitant rent, travel to and from work, and have money left over each month. The value of SS3 is the awesome and talented people who'll help build the venture. If you can't see that, then we're not for you.
Note that they have no interest in any ideas you may have:
I have my own idea / startup / service - will you help me with that? No. The startpoint is the ideas that we've already developed with partners, providers and people in the sector and which we know have some potential.
and reserve the right to terminate you at any moment:
What happens at the end of 3 months? We don't know! The venture could be up and running, with money coming in. In which case, you'll be in the box seat to be the person to get the job on a permanent basis. That might be paid at a higher salary, at a lower salary, or no salary at all. If the venture has limited chance of getting to market, we're going to just kill it and you'll be left with a whole load of training and skills, and some happy memories we hope. If the venture has a chance of going somewhere, but we're not right for each other, we move on and find someone else. Sounds risky, huh? That's the point.
So even if the project is a success, in fact, there's no guarantee of getting the job, contrary to prior statements, and no guarantee of any pay even if you get it. And there is a lot of horrible macho Alan Sugar rhetoric and passive-aggressive expectations-wank.
But if you don't do a good job, or your venture isn't going anywhere fast, we replace you...Or we kill it...No remorse...We'll try not to come to blows and then move on...Sounds risky, huh? That's the point...Entrepreneurial..Committed. Passionate..Not an idiot...If you want exits / earnouts / options, we're not for you...No...you have to be in the office in London Bridge...Or out telling people how great this is / learning / selling...If you can't see that, then we're not for you....It's all about the team....If you're afraid of sales, we're not for you...Definitely not. Full-time. 100 MPH. Absolutely committed. As if your life depended on this. Well at least, your financial security. Because it does.
Indeed it does, but not so much that they're willing to offer anything lame like a job. As previously noted, they're not actually promising any money or any commitment of any kind, but they do keep using the word job:
If you do well, and the venture is doing well, you'll get a job. A good job.
Ahem? What was that again?
In which case, you'll be in the box seat to be the person to get the job on a permanent basis. That might be paid at a higher salary, at a lower salary, or no salary at all.
Now, I mentioned that the projects are non-profit and are to stay that way. Fine. This doesn't, however, mean that Sidekick itself is non-profit. In fact it's a commercial company, registration no. 6707987, and its customer in this project is, well, us. Specifically, they've scored a grant from the UK Technology Strategy Board.
So, to summarise: You get to work your arse off trying to make some daft idea like "Wife Swap for grans" fly. For this you get a suspiciously unspecified sum of money. You may be terminated without cause at any moment. After three months, you may get a job, you may get the same job but without the money, or you may get "no tea and walk home", depending on no conditions that anyone is willing to state at the outset. You may not profit from the business, no matter how well it may do, but Sidekick's directors get theirs whatever happens. And we're all paying for this exploitative, cynical, spammy, hilariously ill-thought out shite.
Further, and just in case anyone thinks I'm biased in any way, does anyone else think The Guardian could perhaps make its pay-for-play Guardian Professional advertorial and events operation look and feel distinct from the actual newspaper's website?
Why am I talking about this?
Well, Sidekick also recommended "a Guardian article" to us - this one. Now, unless you follow Fleet Street politics closely you'd probably think that's just another Guardian URI. But the key bit is that it's in their "Social Enterprise Network", one of their "Professional Networks". Guardian Professional's front page is here. It's possible that there is some difference between Guardian Professional Networks and Guardian Professional, in which case it might be more professional to make this obvious. But I suspect that this piece is advertorial.
It is, however, informative.
Over the past year, in the face of chaotic reorganisations and relentless manager bashing from politicians, we've seen many of them decide to take up generous redundancy packages, in some cases over a year of full pay...
Those civil servants who have been made redundant - they collected their redundancy money! The bastards.
Aside from the huge drop in productivity during the prolonged reorganisation, this mass firing and re-hiring carries huge risks for costs going forward. The same manager is thinking of moving on to work for an international research agency. If the GP consortia wants her unique skills and community relationships in twelve months time, they'll find her charged out to clients at £1000 a day....
Indeed, just like last time. Perhaps it would be better not to do the privatisation and mass sackings in the first place?
If all this sounds a bit gloomy, social enterprises could offer a way to stem the flow of this talent and make use of the experience and knowledge that the taxpayer has invested in...the fear is the shadow of large private sector companies coming in and cleaning up....At Sidekick Studios, we're going t try to grab some of this talent first. Our reasons are pretty selfish – if public services don't want these people, and their knowledge, and their skills, and their networks, then we sure do....From May 1st, we're starting SS3...
I think this might have been improved by an admission that Sidekick Studios is in fact a private sector, for profit, company. What offends me about this is that they are quite openly trying to help push public servants out of real jobs with pensions and union recognition and that stuff and into the hyper-precarious code-for-pizza status we just outlined. Come to think of it, if you can't get hipster interns living off the Bank of Mum & Dad you might as well try sacked civil servants living off the redundancy money.
But then, what else would you expect from people who want to replace the poor sods working in local authority social care with "Wife Swap for your gran" and the police with "Foursquare for criminals", all held together with sales-training day bullshit and shameless volunteer-mining?
A control room full of computers
Here's an interesting story about a successful response to the seizure of a ship, MV Full City, by pirates in the Indian Ocean. Indian, Chinese, and Turkish (NATO) ships responded, and the pirates abandoned the prize after an Indian Navy aircraft (slightly ironically, a Tupolev Bear) overflew the ship. But the really interesting thing isn't so much that there was good international cooperation, or even that there were pirates 450 miles off the Indian coast, and certainly isn't that Indian naval aviation was flying a Tu-142 (they have been for decades).
In fact, it's that the Chinese Maritime Rescue Coordination Centre was in charge of the whole thing. The China MRCC is based in Beijing (not very maritime, but it is essentially a control room full of computers). The British one is at RAF Kinloss (so presumably will be moving pretty soon) and has coordinated operations everywhere in the world. This is the first time I've heard about an oceanic rescue being coordinated from the Chinese one, though. It's a data point.
Meanwhile, they're loading up on PhDs.
In fact, it's that the Chinese Maritime Rescue Coordination Centre was in charge of the whole thing. The China MRCC is based in Beijing (not very maritime, but it is essentially a control room full of computers). The British one is at RAF Kinloss (so presumably will be moving pretty soon) and has coordinated operations everywhere in the world. This is the first time I've heard about an oceanic rescue being coordinated from the Chinese one, though. It's a data point.
Meanwhile, they're loading up on PhDs.
Errr
Following up this post, here's a really interesting piece in Dawn on the Indian-Pakistani nuclear balance and the implications of the COLD START doctrine. It's an especially good point that if India really wanted to punish Pakistan after a "Mumbai II" terrorist attack, they could do so very effectively and much less dangerously through economic sanctions, given how much fuel Pakistan imports and that most of it passes through one port.
In the light of this, it's hard to avoid the conclusion that the Indian military preparations are simply unwise - in a classic post at Arms Control Wonk, Michael Krepon discusses why Pakistan is continuing to build more nuclear weapons and concludes that the factors at work are as follows. First of all, Indian leaders' public statements are threatening - to use cold-war terminology, although their military planning is moving towards "flexible response", their declaratory policy contains a lot of "massive retaliation". The combination is toxic. Trying to make the conventional forces more usable is potentially provocative. Statements about nuclear strategy like this one, combined with faster response times, begin to look a lot like an offensive doctrine:
Secondly, although nuclear weapons cost a lot to acquire in the first place, they get much cheaper once the programme has been capitalised and the process industrialised. This was a major theme in the high cold war - the original Manhattan Project was designed to scale up to five bombs a month, achieved that ahead of schedule, and in fact scaled even further. Also, they are often considered cheap in terms of their strategic value. Nukes scare people; Pakistan will never be an industrial power like India, but now it has the production line going, it certainly can add more bombs and more target packages faster than the Indian economy can grow. Krepon makes the interesting point that the limiting factor isn't the nukes so much as the delivery systems - a country like North Korea can build a nuclear device of sorts, and Pakistan can run a bomb factory, but only a fully diversified industrial economy can make the aeroplane or the missile to carry them.
This has certain consequences for the Pakistani strategic targeting plan. In comments at ACW, someone asks whether they might be thinking of making use of man- or at least vehicle-portable weapons, the famous suitcase nukes. Another, slightly less terror-licious point about this is how the Pakistan Air Force is operating. If they have plenty of bombs but relatively few aircraft, they have to preserve the strike-force (the P-Force, perhaps, by analogy with the 1960s RAF V-Force) at all costs. This implies putting as many planes as possible on quick-reaction alert, dispersing them early in a crisis with the weapons, and keeping open the option of dispersing them in Afghanistan. (We may now begin to see why they care so much.) It also suggests that it would be very difficult to target anything in the Pakistan Air Force without threatening the nuclear assets, and that they might be keen to use tactical nuclear weapons - it's a relatively cheap substitute for a much bigger army, and (as NATO found out in the high cold war) if you have more and more atom bombs hanging about, pure bureaucratic logic tends to get them assigned to targets.
This is a special case of the principle that mayhem is easy and order is difficult, of course.
The good news, such as there is, is contained in this wikileak, a 2008 cable from the US Ambassador to India. Interestingly, he points out, there are good reasons to think that COLD START is likely to be well named. It takes longer than you think, and when you turn the key there's a lot of grinding and coughing and fuss before anything happens. So you might be tempted to go for a nice cup of tea and come back later, or perhaps have some biscuits and another cup of tea and turn to page 3, or just do something else.
Although the doctrine is explicitly designed to avoid threatening the existence of Pakistan as a state, and therefore to permit Indian military retaliation without triggering anything nuclear, it is seen as threatening both because it is intended to permit military action - to sneak under the wires of deterrence - and also because it is intended to reduce the relevance of Pakistani nuclear forces. The Indians, if the ambassador's analysis is sound, are aware of this and are actually quite unlikely to implement it. One way of looking at the complex administrative machinery and politics he outlines is as a deliberate brake on doing anything hasty. Alternatively, it may not have been created deliberately as a check on the military, but if that is the case, it is interesting that it is tolerated. A state that really did intend to carry out a partial mobilisation and a 72-hour blitz from a standing start would have made sure that the code-word would be given. To some extent, the Indians may be experiencing self-deterrence.
The cable also points out that the terrain has changed since 1971 and that some of the ground is now much more urban and more defensible, and also that there are logistical problems that have yet to be solved. Taking an interpretative view, you might say that the real purpose of COLD START is to reject the idea that the international community has any veto on Indian action and to signal non-deterrence to the Pakistanis, while not actually doing anything dangerous. However, the problem is that the signalling succeeds all too well. In fact, the point that all arguments based on "credibility" are crap strongly applies. Either they are taken at face value, in which case they are dangerous, or they are seen through, in which case they are useless.
So, the D-word. What should anyone do about it? This is traditionally the moment at which it becomes obvious why the abbreviation for the discipline of international relations is pronounced "Errr". But I think the answer is that Kashmir is still the issue. Only real concessions affect perception. Further, it would be very good news if the Indians disavowed COLD START and looked at an alternative reaction plan, perhaps concentrating on the economic side as mentioned in the Dawn link. But you try getting them to do that. Finally, and again spinning off that Dawn piece, the real role of the Pakistani nukes is to secure the special place of the military. Errr, indeed.
In the light of this, it's hard to avoid the conclusion that the Indian military preparations are simply unwise - in a classic post at Arms Control Wonk, Michael Krepon discusses why Pakistan is continuing to build more nuclear weapons and concludes that the factors at work are as follows. First of all, Indian leaders' public statements are threatening - to use cold-war terminology, although their military planning is moving towards "flexible response", their declaratory policy contains a lot of "massive retaliation". The combination is toxic. Trying to make the conventional forces more usable is potentially provocative. Statements about nuclear strategy like this one, combined with faster response times, begin to look a lot like an offensive doctrine:
The Indian Chief of Army Staff, S. Padmanabhan, sang the same tune – that if Pakistan resorted to first use, “the perpetrator of that particular outrage shall be punished so severely that their continuation thereafter in any form will be doubtful.”
Secondly, although nuclear weapons cost a lot to acquire in the first place, they get much cheaper once the programme has been capitalised and the process industrialised. This was a major theme in the high cold war - the original Manhattan Project was designed to scale up to five bombs a month, achieved that ahead of schedule, and in fact scaled even further. Also, they are often considered cheap in terms of their strategic value. Nukes scare people; Pakistan will never be an industrial power like India, but now it has the production line going, it certainly can add more bombs and more target packages faster than the Indian economy can grow. Krepon makes the interesting point that the limiting factor isn't the nukes so much as the delivery systems - a country like North Korea can build a nuclear device of sorts, and Pakistan can run a bomb factory, but only a fully diversified industrial economy can make the aeroplane or the missile to carry them.
This has certain consequences for the Pakistani strategic targeting plan. In comments at ACW, someone asks whether they might be thinking of making use of man- or at least vehicle-portable weapons, the famous suitcase nukes. Another, slightly less terror-licious point about this is how the Pakistan Air Force is operating. If they have plenty of bombs but relatively few aircraft, they have to preserve the strike-force (the P-Force, perhaps, by analogy with the 1960s RAF V-Force) at all costs. This implies putting as many planes as possible on quick-reaction alert, dispersing them early in a crisis with the weapons, and keeping open the option of dispersing them in Afghanistan. (We may now begin to see why they care so much.) It also suggests that it would be very difficult to target anything in the Pakistan Air Force without threatening the nuclear assets, and that they might be keen to use tactical nuclear weapons - it's a relatively cheap substitute for a much bigger army, and (as NATO found out in the high cold war) if you have more and more atom bombs hanging about, pure bureaucratic logic tends to get them assigned to targets.
This is a special case of the principle that mayhem is easy and order is difficult, of course.
The good news, such as there is, is contained in this wikileak, a 2008 cable from the US Ambassador to India. Interestingly, he points out, there are good reasons to think that COLD START is likely to be well named. It takes longer than you think, and when you turn the key there's a lot of grinding and coughing and fuss before anything happens. So you might be tempted to go for a nice cup of tea and come back later, or perhaps have some biscuits and another cup of tea and turn to page 3, or just do something else.
Although the doctrine is explicitly designed to avoid threatening the existence of Pakistan as a state, and therefore to permit Indian military retaliation without triggering anything nuclear, it is seen as threatening both because it is intended to permit military action - to sneak under the wires of deterrence - and also because it is intended to reduce the relevance of Pakistani nuclear forces. The Indians, if the ambassador's analysis is sound, are aware of this and are actually quite unlikely to implement it. One way of looking at the complex administrative machinery and politics he outlines is as a deliberate brake on doing anything hasty. Alternatively, it may not have been created deliberately as a check on the military, but if that is the case, it is interesting that it is tolerated. A state that really did intend to carry out a partial mobilisation and a 72-hour blitz from a standing start would have made sure that the code-word would be given. To some extent, the Indians may be experiencing self-deterrence.
The cable also points out that the terrain has changed since 1971 and that some of the ground is now much more urban and more defensible, and also that there are logistical problems that have yet to be solved. Taking an interpretative view, you might say that the real purpose of COLD START is to reject the idea that the international community has any veto on Indian action and to signal non-deterrence to the Pakistanis, while not actually doing anything dangerous. However, the problem is that the signalling succeeds all too well. In fact, the point that all arguments based on "credibility" are crap strongly applies. Either they are taken at face value, in which case they are dangerous, or they are seen through, in which case they are useless.
So, the D-word. What should anyone do about it? This is traditionally the moment at which it becomes obvious why the abbreviation for the discipline of international relations is pronounced "Errr". But I think the answer is that Kashmir is still the issue. Only real concessions affect perception. Further, it would be very good news if the Indians disavowed COLD START and looked at an alternative reaction plan, perhaps concentrating on the economic side as mentioned in the Dawn link. But you try getting them to do that. Finally, and again spinning off that Dawn piece, the real role of the Pakistani nukes is to secure the special place of the military. Errr, indeed.
Sunday, May 08, 2011
technical update
It has just come to my attention that both Dubai and Sharjah airports have redesigned their websites. Also, I've added 395 more meetings to the scraper this weekend, but for some weird reason the DFID disclosure isn't actually being treated as a csv file by the csv module. Scraping, scraping, scraping, always bloody well scraping. I even had to write a scraping script for work last week.
Also, does anyone else find the OpenTech schedule a bit thin?
Meanwhile, I think I may be about to buy a laptop. Does anyone have experience of the new, cheap-end Lenovo ThinkPads or indeed their top-end netbook-cum-tablet?
Also, does anyone else find the OpenTech schedule a bit thin?
Meanwhile, I think I may be about to buy a laptop. Does anyone have experience of the new, cheap-end Lenovo ThinkPads or indeed their top-end netbook-cum-tablet?
Callaghan-era surge
I have recently been reading David McKittrick (et al)'s Making Sense of the Troubles. An interesting point, which I wasn't aware of before, is their contention that the late 70s and Roy Mason's tenure as Northern Ireland secretary was an important turning point. In fact, you could make an interesting comparison with Iraq in 2007-2008.
Mason's policy was to forget about further top-level negotiations, after the collapse of the Sunningdale agreement, and focus on security and economic issues in the hope that progress from the bottom up would bring the conflict parties back to the negotiating table on better terms. In fact, during his tenure, there was a dramatic drop in the rate of killings that was never reversed. This is the hub of McKittrick's argument - having surged in 1971 and peaked in 1972-3, levels of violence stayed very high until 1977 and then dropped to a new, much lower average level.
The British government in this period tried various expedients. The military used more special forces, and integrated their intelligence systems and those of the police and security services. Overall, there was a deliberate effort to project the whole war as a law-enforcement problem (very much a theme of US and Iraqi government propaganda during the "surge"), and to launch as many criminal prosecutions as possible. They tried hard to recruit informers and to make use of supergrasses.
McKittrick and his co-authors argue that there was a sort of cycle-of-recruitment effect at work - terrorism caused recruitment into the loyalist terrorist organisations, whose violence caused further IRA recruitment, which led to more military intervention and more loyalist violence. They further argue that this cycle was broken or at least slowed down in the late 1970s - whether the government was succeeding in providing security or not, it at least reduced the demand for unofficial, privatised violence. The government also tried hard to recruit potential paramilitaries into its own forces, and demonstrated that it was willing to use force against loyalists as well as republicans (they argue that the failure of the 1977 loyalist strike was important). This is all quite familiar.
However, the drop in violence was an over-determined event. There was a major change in the IRA leadership and strategy, which emphasised holding out for the long term and eventually led to an increased emphasis on the ballot box. (The parallel with the Sadr movement stands out.) There was a major protest movement demanding peace, which is another way of saying that the people were unwilling to tolerate so much violence any more.
Although this turning point meant much less violence, it didn't solve anything in and of itself. And it involved quite a lot of state violence in itself - especially during interrogations. This Musings on Iraq roundup is telling - rather than gunbattles and mass bombings, the war continues with a low-level assassination campaign. In Northern Ireland, the best any security solution could ever do in the absence of a political solution was to hold the levels of violence down around the post-77 average, with a very significant cost to society as a whole.
Mason's policy was to forget about further top-level negotiations, after the collapse of the Sunningdale agreement, and focus on security and economic issues in the hope that progress from the bottom up would bring the conflict parties back to the negotiating table on better terms. In fact, during his tenure, there was a dramatic drop in the rate of killings that was never reversed. This is the hub of McKittrick's argument - having surged in 1971 and peaked in 1972-3, levels of violence stayed very high until 1977 and then dropped to a new, much lower average level.
The British government in this period tried various expedients. The military used more special forces, and integrated their intelligence systems and those of the police and security services. Overall, there was a deliberate effort to project the whole war as a law-enforcement problem (very much a theme of US and Iraqi government propaganda during the "surge"), and to launch as many criminal prosecutions as possible. They tried hard to recruit informers and to make use of supergrasses.
McKittrick and his co-authors argue that there was a sort of cycle-of-recruitment effect at work - terrorism caused recruitment into the loyalist terrorist organisations, whose violence caused further IRA recruitment, which led to more military intervention and more loyalist violence. They further argue that this cycle was broken or at least slowed down in the late 1970s - whether the government was succeeding in providing security or not, it at least reduced the demand for unofficial, privatised violence. The government also tried hard to recruit potential paramilitaries into its own forces, and demonstrated that it was willing to use force against loyalists as well as republicans (they argue that the failure of the 1977 loyalist strike was important). This is all quite familiar.
However, the drop in violence was an over-determined event. There was a major change in the IRA leadership and strategy, which emphasised holding out for the long term and eventually led to an increased emphasis on the ballot box. (The parallel with the Sadr movement stands out.) There was a major protest movement demanding peace, which is another way of saying that the people were unwilling to tolerate so much violence any more.
Although this turning point meant much less violence, it didn't solve anything in and of itself. And it involved quite a lot of state violence in itself - especially during interrogations. This Musings on Iraq roundup is telling - rather than gunbattles and mass bombings, the war continues with a low-level assassination campaign. In Northern Ireland, the best any security solution could ever do in the absence of a political solution was to hold the levels of violence down around the post-77 average, with a very significant cost to society as a whole.
links, the last refuge of really bored bloggers
This came up in my twitter feed recently. Charmingly 90s-ish, but I did like the point that it's very difficult to get decent bass out of a mobile device/laptop/whatever, and this probably has consequences for the kind of music people will make with them. Come to think of it, there's an interesting economic angle. The electronics will only keep getting cheaper, and the software can be free. But some things require a large physical lump that needs transporting and storing awkwardly. It's a little like Baumol's cost disease.
Meanwhile, I read the Grauniad interview with Adam Curtis this weekend and it didn't make me want to see his next after all. This post sums up why. It's TV thinking - if the edit is right, it doesn't matter if the logic contains more handwaving than might be ideal.
What were the most successful public policies of the last 30 years? Apparently, the minimum wage, devolution, privatisation, and the Northern Ireland peace process. I suspect they may not use my definition of success, but even on their terms it's telling that three out of four of them come from the 1997-2001 Labour government.
Finally, some music.
A lot of their other stuff seems terribly dated now but that one holds up.
Meanwhile, I read the Grauniad interview with Adam Curtis this weekend and it didn't make me want to see his next after all. This post sums up why. It's TV thinking - if the edit is right, it doesn't matter if the logic contains more handwaving than might be ideal.
What were the most successful public policies of the last 30 years? Apparently, the minimum wage, devolution, privatisation, and the Northern Ireland peace process. I suspect they may not use my definition of success, but even on their terms it's telling that three out of four of them come from the 1997-2001 Labour government.
Finally, some music.
A lot of their other stuff seems terribly dated now but that one holds up.
Economic facts
Owning the walls of a building separately may seem eccentric and ill-advised, but perhaps it will make a comeback in the light of stories like these? Deutsche Bank "Stuffed Mortgage Reviews In a Closet". They're also accused of being a slum-lord and a public menace. More slums, via Felix Salmon. And there's a slightly unnerving macroscopic take, from Hernando de Soto. I think that's the first time I've seen anyone link the housing bubble and financial crisis itself directly to the mortgage-servicing fuckups. Via.
I'm looking over the wall and they're looking at me
Alliance Géostrategique is having an interesting-sounding meetup on the theme of walls, borders, and checkpoints. Here's a little contribution.
In my home village in the Yorkshire Dales, there lived two men, who both owned a large shed they used for their business. Somehow, one of them owned the roof and two walls, and one of them owned the other. There was some arrangement about the land under it, but this bit of the story escapes me. Anyway, years pass, eventually they give up the business and sell a chunk of land next door to a property developer. But they can't agree on what to do with the shed. They fall out to the extent that only genuine Yorkshiremen can; and the shed is left to rot quietly.
Eventually, bits of it start to fall down. One day, down goes a great chunk, and the local council decides that the building is a menace to the public. So they knock it down. Surely this must resolve the issue one way or the other. Or can the wall exist in a purely legal, moral sense, without the tiresome requirement of a couple of hundredweight of rocks? It turns out that it can, at least for a while. The owner of the roof sells the plot of land, less the strip the wall actually occupied, to another property developer, who builds on it.
At this point, our man gets worried about his interest. Even if the legal position was clear, it might be difficult to assert his rights if the question had actually been physically built over. So he visited the site and put up a sort of temporary favela barrier of corrugated iron, tape, and cones along the route of the wall. Rather like the original, overnight barricade the East Germans set up before the more permanent Berlin Wall was created. On this he stuck a sign reading "For Sale: Valuable Amenity Land".
It's worth stopping here and thinking for a moment. Strangely, the replacement of the original wall probably strengthens his position from an economic point of view. If it was a sound structure of traditional Yorkshire stone, would anybody notice it? Probably not. They would treat it as a wall, not an economic interest. Nobody would think of removing it, or even that someone else owned it. It would just be a brute stone fact. The wall, however, has been replaced with a sort of deliberately absurd, theatrical monument to the guy's ego. (Actually, the original wall wasn't Yorkshire stone or any stone - it was made of breeze-blocks.)
Of course, the other parties to the dispute have not been idle. They are still unwilling to pay the greenmail required to make the whole thing go away, and the very ridiculousness of the new wall makes it clear that destroying it would solve nothing. So they came up with a plan of their own - they painted the whole thing pink, in the hope that this would embarrass the other guy into giving in.
He still hasn't.
In my home village in the Yorkshire Dales, there lived two men, who both owned a large shed they used for their business. Somehow, one of them owned the roof and two walls, and one of them owned the other. There was some arrangement about the land under it, but this bit of the story escapes me. Anyway, years pass, eventually they give up the business and sell a chunk of land next door to a property developer. But they can't agree on what to do with the shed. They fall out to the extent that only genuine Yorkshiremen can; and the shed is left to rot quietly.
Eventually, bits of it start to fall down. One day, down goes a great chunk, and the local council decides that the building is a menace to the public. So they knock it down. Surely this must resolve the issue one way or the other. Or can the wall exist in a purely legal, moral sense, without the tiresome requirement of a couple of hundredweight of rocks? It turns out that it can, at least for a while. The owner of the roof sells the plot of land, less the strip the wall actually occupied, to another property developer, who builds on it.
At this point, our man gets worried about his interest. Even if the legal position was clear, it might be difficult to assert his rights if the question had actually been physically built over. So he visited the site and put up a sort of temporary favela barrier of corrugated iron, tape, and cones along the route of the wall. Rather like the original, overnight barricade the East Germans set up before the more permanent Berlin Wall was created. On this he stuck a sign reading "For Sale: Valuable Amenity Land".
It's worth stopping here and thinking for a moment. Strangely, the replacement of the original wall probably strengthens his position from an economic point of view. If it was a sound structure of traditional Yorkshire stone, would anybody notice it? Probably not. They would treat it as a wall, not an economic interest. Nobody would think of removing it, or even that someone else owned it. It would just be a brute stone fact. The wall, however, has been replaced with a sort of deliberately absurd, theatrical monument to the guy's ego. (Actually, the original wall wasn't Yorkshire stone or any stone - it was made of breeze-blocks.)
Of course, the other parties to the dispute have not been idle. They are still unwilling to pay the greenmail required to make the whole thing go away, and the very ridiculousness of the new wall makes it clear that destroying it would solve nothing. So they came up with a plan of their own - they painted the whole thing pink, in the hope that this would embarrass the other guy into giving in.
He still hasn't.
OBL links
Doing real-time PCR in unusual conditions. Just how Obama watched the raid in real time.
Hoping for the end of the war in Afghanistan.
More expectations for a quicker end. You can't rely on the ISI any more.
The crucial information may have been that nobody ever mentioned the courier's name. Indian politicians rock the boat a bit more. Good piece and discussion at Arms Control Wonk.
Hoping for the end of the war in Afghanistan.
After weeks of debate among civilian and military leaders, the National Security Council recently endorsed key elements of the State Department’s reconciliation strategy. Starting peace talks has now become the top priority for Marc Grossman, who succeeded Richard C. Holbrooke as the U.S. government’s special representative for Afghanistan and Pakistan.
On Tuesday, Grossman met in Islamabad with Pakistan’s foreign secretary and Afghanistan’s deputy foreign minister. The three agreed to constitute a “core group for promoting and facilitating the process of reconciliation and peace in Afghanistan,” Pakistan’s Foreign Ministry said in a statement. U.S. officials expressed hope on Tuesday that Pakistan’s failure to find bin Laden — or its possible complicity in sheltering him — could lead Islamabad to adopt a softer position on Afghan reconciliation. They think that Pakistani officials, who have interfered with peace efforts in the past, have an opportunity to play a more constructive role.
“Our hope is that they are so embarrassed by this that they try to save face by trying to help their neighbor,” one U.S. official said.
More expectations for a quicker end. You can't rely on the ISI any more.
The crucial information may have been that nobody ever mentioned the courier's name. Indian politicians rock the boat a bit more. Good piece and discussion at Arms Control Wonk.
obl isi faq
There's a good debate in Jamie Kenny's comments about the political upshot of bin Laden's death in Pakistan. For what it's worth, I'm with Dan Hardie on this - it's a very important political fact that the intelligence hierarchy a) couldn't or wouldn't catch bin Laden and b) couldn't or wouldn't protect him any longer. The ISI needs to be useful to the Americans and also to the jihadis to maintain its private foreign policy and its special role in Pakistani politics.
In the short term, it's still in a position to make trouble - NATO is still using the roads to Afghanistan, after all - but this ability to cause trouble is now significantly constrained. When bin Laden turns out to have been living two miles from the Abbottabad Golf Club all these years, playing the supply-route card is very close indeed to committing to his side and burning the bridges. It may not be as dramatic as the Falklands War was for the Argentine military as a political actor. The outrage of actual Americans staging an air-mobile assault in urban Pakistan buffers this a bit. But they can't count on nationalist outrage as a source of support - they didn't, after all, prevent the raid, whether by shooting up the helicopters or by getting rid of bin Laden themselves.
The defining issue now is whether Pakistan's other institutions can assert more power faster than the Americans (and everyone else) can cut-and-run. The end of US support after 1992, after all, tended to strengthen the ISI as a force in Pakistani politics. If the ISI director Ahmed Shuja Pasha is indeed sacked, this will be quite an ambiguous move, as General Kayani brought him in from the armoured corps in order to keep an eye on the organisation. But this is only a short-term coping strategy - or in other words, tactics rather than strategy.
In the short term, it's still in a position to make trouble - NATO is still using the roads to Afghanistan, after all - but this ability to cause trouble is now significantly constrained. When bin Laden turns out to have been living two miles from the Abbottabad Golf Club all these years, playing the supply-route card is very close indeed to committing to his side and burning the bridges. It may not be as dramatic as the Falklands War was for the Argentine military as a political actor. The outrage of actual Americans staging an air-mobile assault in urban Pakistan buffers this a bit. But they can't count on nationalist outrage as a source of support - they didn't, after all, prevent the raid, whether by shooting up the helicopters or by getting rid of bin Laden themselves.
The defining issue now is whether Pakistan's other institutions can assert more power faster than the Americans (and everyone else) can cut-and-run. The end of US support after 1992, after all, tended to strengthen the ISI as a force in Pakistani politics. If the ISI director Ahmed Shuja Pasha is indeed sacked, this will be quite an ambiguous move, as General Kayani brought him in from the armoured corps in order to keep an eye on the organisation. But this is only a short-term coping strategy - or in other words, tactics rather than strategy.
A question
How did the Americans make sure their raid on Osama bin Laden wasn't misidentified as Pakistan's real enemy? This was surely a major planning constraint. It's been suggested, plausibly, that the bulk of their radar assets are positioned along the international border and the LOC, but once you get to Abbottabad you're not that far from the Line of Control. There's been a lot of interest in the helicopter that was destroyed, and specifically if it was either a hitherto unknown type or else a Blackhawk modified to be stealthy. But stealthy is a relative term, and a helicopter will never be really stealthy as its rotor blades are constantly changing aspect towards any radar source.
There's an interesting French paper here on Indian military doctrine - apparently, part of the lessons-learned exercise after the 2002 crisis and mobilisation was that the whole process took too long, and left far too many opportunities for the international community to get involved and yell "stop!". (This may not be the lesson one would hope had been learned.) As a result, they came up with a new doctrine, known as Cold Start, which foresaw a quicker response to provocation from Pakistan, using forces already posted nearer the border to carry out raids with limited territorial objectives, closely integrated with air power. The point that the objectives are limited in terms of territory is important - as I mentioned above, a lot of things in Pakistan are not far from the border. They might not be very limited in terms of importance, for example, nuclear sites or major headquarters, or perhaps key ISI or jihadi figures.
(Ah, we had one of those, didn't we?)
Obama's counter-terrorism advisor, John Brennan, was quoted as saying that the Pakistani air force scrambled its quick-reaction alert of fighters during the mission. This may of course be disinformation, or just wrong. It could imply that for a while at least, there was an elevated risk. Or perhaps the plan was designed to make it obvious that the helicopters were coming from the direction of Afghanistan, and they wanted the radars to detect them at some point during the operation...
It would be very interesting to know if the Indian government was informed at any point.
There's an interesting French paper here on Indian military doctrine - apparently, part of the lessons-learned exercise after the 2002 crisis and mobilisation was that the whole process took too long, and left far too many opportunities for the international community to get involved and yell "stop!". (This may not be the lesson one would hope had been learned.) As a result, they came up with a new doctrine, known as Cold Start, which foresaw a quicker response to provocation from Pakistan, using forces already posted nearer the border to carry out raids with limited territorial objectives, closely integrated with air power. The point that the objectives are limited in terms of territory is important - as I mentioned above, a lot of things in Pakistan are not far from the border. They might not be very limited in terms of importance, for example, nuclear sites or major headquarters, or perhaps key ISI or jihadi figures.
(Ah, we had one of those, didn't we?)
Obama's counter-terrorism advisor, John Brennan, was quoted as saying that the Pakistani air force scrambled its quick-reaction alert of fighters during the mission. This may of course be disinformation, or just wrong. It could imply that for a while at least, there was an elevated risk. Or perhaps the plan was designed to make it obvious that the helicopters were coming from the direction of Afghanistan, and they wanted the radars to detect them at some point during the operation...
It would be very interesting to know if the Indian government was informed at any point.
Monday, April 25, 2011
Even more AV
More AV comment, from a colleague:
I'm not as impressed with Arrow's theorem as a lot of people are - perhaps because I'm just more comfortable with better rather than perfect than my correspondent here is. Actually, having just been reading the wikipedia discussion, I have a couple of fundamental disagreements - notably that I don't really have a problem with the notion that there is a potential case in which one "dictatorial" voter or coalition of voters decides the election. For a start, the whole point of an election is that the candidates try to secure the support of enough voters to win. Secondly, if who turns out to be the critical voter is random with respect to the demographic makeup of the electorate and not known in advance, there's an argument that this would in fact be fair, in that it wouldn't privilege any interest-group over any other. As it would be impossible to target them for campaigning purposes, politics would have to operate as if the system was formally fair in Arrow's sense. Given the entrenched biases in the current system, this would in fact be a move towards justice.
Anyway, there's nothing in Arrow to say that we should prefer a system that is pathological when it's operating normally to one that is usually better but might have odd corner cases. It's also ironic that economic libertarians get so hung up on Arrow when the same logic about Pareto efficiency is a well-known problem for free-market economics (making any given market more free doesn't necessarily move the macroeconomy closer to full allocative efficiency).
I voted "no". Why?
1. Arrow's theorem. Getting democracy "right" is mathematically impossible.
2. Elections are not there to select the "right" government, but to confer legitimacy on that which is elected, and to enable truly awful ones to be removed. FPTP is good enough.
3. Fiddling with the electoral maths is thinking too small. If you want to reform the system, and have a referendum on it, then a much more comprehensive reallocation of power between centre and edge is needed.
I'm not as impressed with Arrow's theorem as a lot of people are - perhaps because I'm just more comfortable with better rather than perfect than my correspondent here is. Actually, having just been reading the wikipedia discussion, I have a couple of fundamental disagreements - notably that I don't really have a problem with the notion that there is a potential case in which one "dictatorial" voter or coalition of voters decides the election. For a start, the whole point of an election is that the candidates try to secure the support of enough voters to win. Secondly, if who turns out to be the critical voter is random with respect to the demographic makeup of the electorate and not known in advance, there's an argument that this would in fact be fair, in that it wouldn't privilege any interest-group over any other. As it would be impossible to target them for campaigning purposes, politics would have to operate as if the system was formally fair in Arrow's sense. Given the entrenched biases in the current system, this would in fact be a move towards justice.
Anyway, there's nothing in Arrow to say that we should prefer a system that is pathological when it's operating normally to one that is usually better but might have odd corner cases. It's also ironic that economic libertarians get so hung up on Arrow when the same logic about Pareto efficiency is a well-known problem for free-market economics (making any given market more free doesn't necessarily move the macroeconomy closer to full allocative efficiency).
Sunday, April 24, 2011
rewind to 2006
The log says AFRICOM wants to know what's up with B.727 msn 22045, ex of Irbis Air and now with "Mega" of Kazakhstan. Last heard of in Shymkent in January 2010, before that in Djibouti in 2008. I've no new information to add.
Even more AV
OK, so I'd just about reasoned myself around to voting no. But every time I get there I run into some nightmarish turdpool of mendacity from the no campaign. I may just not be able to stomach doing anything that George Osborne, John Reid, Robert Edmiston, Dan Hodges, and "Tom" "Newton"-"Dunn" agree with. I'd abstain, but that's no solution.
To summarise yet again, the arguments are as follows, net-net:
Yes: It's crap, and it doesn't afford an immediate kick at the Lib Dems, but there might be some angle to be had from it and anyway everyone hates the Lib Dems.
No: Inchoate lashout.
I'm actually quite keen on policy based on gut hatred - there have been quite a few occasions over the last ten years when allowing yourself to be guided by tribal loyalty, practiced trigger-movements, and aesthetic revulsion would have put you in the right when a lot of people got it wrong based on reason. But the rules are the rules, and surely the fear and loathing factor must respond to the sheer horror of the no campaign.
To summarise yet again, the arguments are as follows, net-net:
Yes: It's crap, and it doesn't afford an immediate kick at the Lib Dems, but there might be some angle to be had from it and anyway everyone hates the Lib Dems.
No: Inchoate lashout.
I'm actually quite keen on policy based on gut hatred - there have been quite a few occasions over the last ten years when allowing yourself to be guided by tribal loyalty, practiced trigger-movements, and aesthetic revulsion would have put you in the right when a lot of people got it wrong based on reason. But the rules are the rules, and surely the fear and loathing factor must respond to the sheer horror of the no campaign.
Glue Labour: how you can miss out the coalition and come up in 2015!
If you were asked to write 1,572 words for a national newspaper on the role of voluntary associations in the Labour Movement, and thank the Lord neither you or I is likely to meet this fate any time soon, would you manage to mention trade unions at least once? Not if you're Maurice Glasman. I don't intend to waste much time on "Blue Labour", but I do think this point needs to be made. Also, if you or I was to write such a slab, I think we'd make a better job of it. Here are a couple of pearls.
The Obscurer seems to have anointed him as the new leader of Continuity Blairism, going by the eight handsome volumes it's dedicated to the sage of Holloway Road's thoughts on agriculture...sorry, the 1,572 words across a double centre page spread with a photo of him smoking cerebrally into the middle distance against a backdrop carefully blurred in line withOKTrends's empirically derived guidelines for successful online dating. This, next to a really Pravdaesque bit of sycophancy, from which I quote:
"We have nothing against people in call centres". But, it would seem, a lot of them are women and some are even black, and anyway they aren't Proper Picturesque Proles for the TV, like. There are a whole lot of them, though, and perhaps they might have some of that emotion if more of them were in...a union. Perhaps I should start Glue Labour, for people who would rather sniff glue than read Maurice Glasman's risible tosh. Suggestions for a motto are invited.
It required new work agreements so that all was not relentlessly up for grabs in an exclusively contractual churn.
In order to do this, Labour must establish those conversations that broker a common good within which party organisations such as Progress, the Fabians, Compass and the Christian Socialist Movement and Blue Labour talk and build a common programme.
The Obscurer seems to have anointed him as the new leader of Continuity Blairism, going by the eight handsome volumes it's dedicated to the sage of Holloway Road's thoughts on agriculture...sorry, the 1,572 words across a double centre page spread with a photo of him smoking cerebrally into the middle distance against a backdrop carefully blurred in line withOKTrends's empirically derived guidelines for successful online dating. This, next to a really Pravdaesque bit of sycophancy, from which I quote:
A source close to Miliband said the Labour leader was moved by what he heard as he munched on a bacon sandwich in Chrissy's. "Ed met a porter who said the proudest day of his life was when he got his badge. His dad had been a porter and then he got his badge. We have nothing against people in call centres but I am not sure there would be the same emotion on being given a first telephone headset."
"We have nothing against people in call centres". But, it would seem, a lot of them are women and some are even black, and anyway they aren't Proper Picturesque Proles for the TV, like. There are a whole lot of them, though, and perhaps they might have some of that emotion if more of them were in...a union. Perhaps I should start Glue Labour, for people who would rather sniff glue than read Maurice Glasman's risible tosh. Suggestions for a motto are invited.
lobbyists lobbying for lobbying
Best lobby metrics (lobbylyzer? lobster, for LoBbying Social Topology ExploreR?) result yet. Today I implemented my gatekeeper vs. flakcatcher metric - it averages the edge weights of all the neighbours of a minister, and returns a ratio of the difference between this and the average weight and the difference between the minister's weighting and the average. The principle is that if you lobby a minister, and then get access to another, your lobbying effort should gain or lose impact depending on the difference between the minister's own importance and the average. In the null hypothesis, where no minister is better or worse as a lobbying target than predicted by their department and their individual rank, you'd get the difference between the minister's weighting and the average.
If some ministers are gatekeepers, though, you would see a greater boost to your efforts at influence than the null case. Similarly, if some of them are flak catchers who mainly exist to turn away lobbying efforts, and you happened on one of those, you'd get a lesser boost. This metric should be greater than unity if the minister is a gatekeeper, 1 if they are perfectly mediocre, and less than unity if they are a flak catcher.
Interestingly, the Scotland and Wales Offices score highly. The highest value recorded is for David Jones MP, Parliamentary Under-Secretary of State for Wales, with 2.75. His closest rival is David Mundell MP, from the Scotland Office, on 1.94. The highest scoring Cabinet minister is the Scottish Secretary, Michael Moore. Some of this is down to the magic of low expectations. Nobody thinks these departments are great offices of state, not even the Welsh or the Scots - real power there has long since shifted to the devolved administrations. So if you meet any other minister, you're likely to do better. But the gatekeeper metric should handle this, as it measures influence relative to the structural difference between the minister and the average weighting. Arguably, this is a valid measurement. These ministries' role really is as a gatekeeper, something like a diplomatic representation in both directions.
So, who's the all-UK champ? It turns out to be Mark Harper MP, Minister for Political and Constitutional Reform, with an impressive 1.65, contrasting with his network degree of 0.08. Hilariously, one of his lobbies turns out to be the UK Public Affairs Council, the lobbyists' trade union, which wanted to see him in July, 2010 on the pressing matter of "lobbying".
He's followed home by Education PUSS Tim Loughton MP on 1.228, Defence PUSS Lord Astor on 1.2, Education's Lord Hill on 1.1, Justice PUSS Crispin Blunt MP with 1.09, and the sinister intellectual force that is Oliver Letwin MP, Minister of State for Government Policy, on 1.05. Francis Maude, whose horrific rise to national influence has been tracked with interest, turns out to be quite the flak catcher, on 0.43 - or is it that he claims to be a figure of authority in his own right? Does the buck stop there? After all, and as expected, once you meet the prime minister you can't really go anywhere but down.
Of course, what everyone will surely want to know is who gets the wooden spoon. Step forward Andrew Stunell MP, PUSS in the Department for Communities and Local Government, with a mighty 0.21 to go with his network degree of 0.3. Stunell held a large number of meetings around the country, notably in London, Bristol, and Bradford, as "Big Society Roundtables" with a wide range of community organisations. It would appear that nobody was more shortchanged than these. Meeting Mr. Stunell reduced one's average lobbying impact by a smacking 80%. Such was the coalition's contempt for, among other organisations, Operation Black Vote, the Stephen Lawrence Trust, and basically everyone in Bradford who showed up. The list is here.
It will surprise nobody that meeting a coalition minister would increase the UK Public Affairs Council's members' influence by 65%, but cut that of council tenants, Muslims, blacks, single mothers, young people (to list just a few) by 80%. But it's worth making it hideously explicit. And here's a lesson from all this obscure science that is easy enough to operationalise: if you see Andrew Stunell coming towards you through the Strangers' Bar with a smile on his face, don't make eye contact, don't shake hands, don't offer him your business card. Run. Spill a pint. Create a diversion. Trigger the fire alarm. Do not, in any circumstances, lobby him.
Here's the really sad bit. Stunell's dance card, from TWFY.
I remember the Lib Dems. Do you? I wonder if Andrew Stunell remembers Andrew Stunell.
If some ministers are gatekeepers, though, you would see a greater boost to your efforts at influence than the null case. Similarly, if some of them are flak catchers who mainly exist to turn away lobbying efforts, and you happened on one of those, you'd get a lesser boost. This metric should be greater than unity if the minister is a gatekeeper, 1 if they are perfectly mediocre, and less than unity if they are a flak catcher.
Interestingly, the Scotland and Wales Offices score highly. The highest value recorded is for David Jones MP, Parliamentary Under-Secretary of State for Wales, with 2.75. His closest rival is David Mundell MP, from the Scotland Office, on 1.94. The highest scoring Cabinet minister is the Scottish Secretary, Michael Moore. Some of this is down to the magic of low expectations. Nobody thinks these departments are great offices of state, not even the Welsh or the Scots - real power there has long since shifted to the devolved administrations. So if you meet any other minister, you're likely to do better. But the gatekeeper metric should handle this, as it measures influence relative to the structural difference between the minister and the average weighting. Arguably, this is a valid measurement. These ministries' role really is as a gatekeeper, something like a diplomatic representation in both directions.
So, who's the all-UK champ? It turns out to be Mark Harper MP, Minister for Political and Constitutional Reform, with an impressive 1.65, contrasting with his network degree of 0.08. Hilariously, one of his lobbies turns out to be the UK Public Affairs Council, the lobbyists' trade union, which wanted to see him in July, 2010 on the pressing matter of "lobbying".
He's followed home by Education PUSS Tim Loughton MP on 1.228, Defence PUSS Lord Astor on 1.2, Education's Lord Hill on 1.1, Justice PUSS Crispin Blunt MP with 1.09, and the sinister intellectual force that is Oliver Letwin MP, Minister of State for Government Policy, on 1.05. Francis Maude, whose horrific rise to national influence has been tracked with interest, turns out to be quite the flak catcher, on 0.43 - or is it that he claims to be a figure of authority in his own right? Does the buck stop there? After all, and as expected, once you meet the prime minister you can't really go anywhere but down.
Of course, what everyone will surely want to know is who gets the wooden spoon. Step forward Andrew Stunell MP, PUSS in the Department for Communities and Local Government, with a mighty 0.21 to go with his network degree of 0.3. Stunell held a large number of meetings around the country, notably in London, Bristol, and Bradford, as "Big Society Roundtables" with a wide range of community organisations. It would appear that nobody was more shortchanged than these. Meeting Mr. Stunell reduced one's average lobbying impact by a smacking 80%. Such was the coalition's contempt for, among other organisations, Operation Black Vote, the Stephen Lawrence Trust, and basically everyone in Bradford who showed up. The list is here.
It will surprise nobody that meeting a coalition minister would increase the UK Public Affairs Council's members' influence by 65%, but cut that of council tenants, Muslims, blacks, single mothers, young people (to list just a few) by 80%. But it's worth making it hideously explicit. And here's a lesson from all this obscure science that is easy enough to operationalise: if you see Andrew Stunell coming towards you through the Strangers' Bar with a smile on his face, don't make eye contact, don't shake hands, don't offer him your business card. Run. Spill a pint. Create a diversion. Trigger the fire alarm. Do not, in any circumstances, lobby him.
Here's the really sad bit. Stunell's dance card, from TWFY.
Voted very strongly against introducing foundation hospitals.
Voted strongly against Labour's anti-terrorism laws.
Voted very strongly against the Iraq war.
Voted very strongly for an investigation into the Iraq war.
Voted moderately against allowing ministers to intervene in inquests.
Voted a mixture of for and against greater autonomy for schools.
Voted moderately against replacing Trident.
Voted very strongly for the hunting ban.
Voted moderately for more EU integration.
Voted very strongly against introducing ID cards.
Voted very strongly for laws to stop climate change.
Voted very strongly against a stricter asylum system.
Voted moderately for removing hereditary peers from the House of Lords.
Voted strongly for a wholly elected House of Lords.
Voted very strongly for equal gay rights.
Voted moderately for a transparent Parliament.
I remember the Lib Dems. Do you? I wonder if Andrew Stunell remembers Andrew Stunell.
Friday, April 22, 2011
for those non-cooperative with the system
I was wondering what "Ingeus UK" was - it's the 15th heaviest lobby in the kingdom with degree 0.17, just behind HSBC and ahead of GlaxoSmithKline, Microsoft, and LOCOG - but now we know.
Another lobbying thing
This project begins to want a name.
Routine maintenance
After this post and the outstanding response to it, I've just been working on the lobby project's underpinnings, specifically to backport some data cleaning from the analyser script into the original scraper, and to fix the one-edge-per-row version of the scraper. As a result I've had to flush the datastore and also search out some URIs that have changed. So far we've recreated 931 out of 1,721 meetings, although we're getting the dreaded "Execution status: run interrupted by a timeout". Actually, we've got 1,233 meetings back, and we've got rid of some crap. Anyone wanting the dataset can get it from the Scraperwiki API here or here for linkwise rather than meetingwise (coming soon) as either json-dict or csv. A full SQL syntax is available.
With luck, there will also be some more data quite soon. On the analysis score, notably, this and also this seem useful. The first estimates the value of a node based on its edges, which is fundamentally what I'm trying to achieve, and the second finds the cliques in the network a given node belongs to.
Regarding visualisation issues, I think one of my mistakes last time out was to visualise the data as a multi-graph - i.e. a structure with zero or more links between each node, permitting the existence of multiple links between the same pair of nodes. This invariably means a lot of links. The nature of the data - multiple meetings are absolutely central to the whole project, and lobbies meet ministers at different times and on different issues - enforces an underlying multigraph structure. But it would be possible to condense it for visualisation purposes - if we rolled up all links between the same nodes into one, we could tot up their weights and perhaps show that in the visualisation, as a thicker line for example.
With luck, there will also be some more data quite soon. On the analysis score, notably, this and also this seem useful. The first estimates the value of a node based on its edges, which is fundamentally what I'm trying to achieve, and the second finds the cliques in the network a given node belongs to.
Regarding visualisation issues, I think one of my mistakes last time out was to visualise the data as a multi-graph - i.e. a structure with zero or more links between each node, permitting the existence of multiple links between the same pair of nodes. This invariably means a lot of links. The nature of the data - multiple meetings are absolutely central to the whole project, and lobbies meet ministers at different times and on different issues - enforces an underlying multigraph structure. But it would be possible to condense it for visualisation purposes - if we rolled up all links between the same nodes into one, we could tot up their weights and perhaps show that in the visualisation, as a thicker line for example.
Sunday, April 17, 2011
lobby: update
I'm beginning to make some progress with the lobbying project. Last week I got it spitting out data; in mid-week, I optimised the process of loading the meetings from the ScraperWiki API into NetworkX. Hint: the obj_hook keyword argument in python's json.load() function is really useful!
This weekend it's producing information about lobbies, ministers, and government departments. I've got implementations nearly ready for a couple more dimensions of data - providing each actor's network degree by month, and trying to measure the extent to which ministers act as gatekeepers or flak-catchers. The first of those involves reimplementing a bit of NetworkX - you can't ask for node properties excluding certain edges by attribute, or at least you can't do so without creating a new subgraph, which seems ugly. The second, at the moment, counts the edges of a node if they have a higher weight than that of the node itself and expresses the sum of those edges' weights as a percentage of the total meetings that minister had. That doesn't take any account of time, yet.
I'm thinking of using Google App Engine to deploy it, running the data generator as a cron job and using the bulk uploader utility to slurp the results.
As a taster, the biggest single private interest lobbying Government is Barclays Bank, followed by Shell, the World Bank, the London Stock Exchange, BP, RBS, BAE, Standard Chartered, Lloyds, and Ratan Tata. This may not be that surprising. Neither is it very surprising, if somehow comforting in an old-fashioned way, that the two biggest lobbies of all are the Confederation of British Industry and the TUC, which is achieving about two-thirds the lobbying effort of the CBI and about twice that of Barclays. I was surprised to find that lobby 26 is Facebook, above Tesco, Microsoft, or UNISON. (Google is far, far down the list.) The highest placed individual trade union is the CWU at 24, between HSBC and the Electoral Commission. The littlest lobby is a nursery school in Leeds that got herded into a Big Society meeting with Nick Hurd MP.
I'm not so sure about using this model to assess the ministers, as we're using a priori weightings on them. But the decision to lobby a given minister must contain some information about the lobbyist's perceptions of their power and influence. Britain's most lobbied minister is Chris Grayling MP, Minister of State for Employment, who achieves a weighted degree of 4.2, not far off twice the prime minister. David Willetts, Vince Cable, Nick Clegg, and Francis Maude are the next four before the prime minister. They range between 2.8 and 2.6 with the PM on 2.3. Britain's least influential minister appears to be Baroness Warsi, minister without portfolio, on a score of 0.057.
BIS is the most lobbied department on 12.42, followed by the Department for Work and Pensions on 9.65, the Treasury on 7.065, the Cabinet Office on 5.62, and the DCLG on 3.825. Delight to the econophysicists (are they still around?): the distributions seem to show a nice power-law relationship! Which tells us what precisely? Well....not much except that it's a social network and they usually have them!
There were 2,073 nodes, either ministers or lobbies, in the graph at the last data upload. 2,848 interactions between them were analysed.
Does anyone have any ideas for other metrics that might be interesting?
This weekend it's producing information about lobbies, ministers, and government departments. I've got implementations nearly ready for a couple more dimensions of data - providing each actor's network degree by month, and trying to measure the extent to which ministers act as gatekeepers or flak-catchers. The first of those involves reimplementing a bit of NetworkX - you can't ask for node properties excluding certain edges by attribute, or at least you can't do so without creating a new subgraph, which seems ugly. The second, at the moment, counts the edges of a node if they have a higher weight than that of the node itself and expresses the sum of those edges' weights as a percentage of the total meetings that minister had. That doesn't take any account of time, yet.
I'm thinking of using Google App Engine to deploy it, running the data generator as a cron job and using the bulk uploader utility to slurp the results.
As a taster, the biggest single private interest lobbying Government is Barclays Bank, followed by Shell, the World Bank, the London Stock Exchange, BP, RBS, BAE, Standard Chartered, Lloyds, and Ratan Tata. This may not be that surprising. Neither is it very surprising, if somehow comforting in an old-fashioned way, that the two biggest lobbies of all are the Confederation of British Industry and the TUC, which is achieving about two-thirds the lobbying effort of the CBI and about twice that of Barclays. I was surprised to find that lobby 26 is Facebook, above Tesco, Microsoft, or UNISON. (Google is far, far down the list.) The highest placed individual trade union is the CWU at 24, between HSBC and the Electoral Commission. The littlest lobby is a nursery school in Leeds that got herded into a Big Society meeting with Nick Hurd MP.
I'm not so sure about using this model to assess the ministers, as we're using a priori weightings on them. But the decision to lobby a given minister must contain some information about the lobbyist's perceptions of their power and influence. Britain's most lobbied minister is Chris Grayling MP, Minister of State for Employment, who achieves a weighted degree of 4.2, not far off twice the prime minister. David Willetts, Vince Cable, Nick Clegg, and Francis Maude are the next four before the prime minister. They range between 2.8 and 2.6 with the PM on 2.3. Britain's least influential minister appears to be Baroness Warsi, minister without portfolio, on a score of 0.057.
BIS is the most lobbied department on 12.42, followed by the Department for Work and Pensions on 9.65, the Treasury on 7.065, the Cabinet Office on 5.62, and the DCLG on 3.825. Delight to the econophysicists (are they still around?): the distributions seem to show a nice power-law relationship! Which tells us what precisely? Well....not much except that it's a social network and they usually have them!
There were 2,073 nodes, either ministers or lobbies, in the graph at the last data upload. 2,848 interactions between them were analysed.
Does anyone have any ideas for other metrics that might be interesting?
AV: roundup and conclusion of a sort
So I asked for your help to decide what I thought about the alternative vote.
A few of you were supportive on the grounds that it was a start and it would be possible to demand further improvement later.
The problem here is that every attempt to model its effects I've seen primarily benefits the Lib Dems. All other things being equal, this fulfils their primary interest in supporting electoral reform. More proportionality starts to bring other parties into the game. In a transitional AV system, the Lib Dems would be in the position of the German FDP, and they wouldn't have any interest in weakening this position. They would tend to swing against anyone who suggested STV or more, and as kingmakers, prevent it from happening. If making a move on electoral reform really did make it easier to go further, we'd somehow have to get to AV without having a single swing party with an interest in sticking at AV. Essentially, we'd need to have a Labour government with a manifesto commitment to STV, which requires that the Lib Dems get such a thrashing that even the AV bonus can't keep them relevant.
Well, I can certainly imagine the Lib Dems getting a thrashing at the next election. But if the point is to beat the coalition and elect a strong Labour government, either on its own or as a hegemonic coalition partner with a few Lib Dem survivors, and then pass STV, why risk the scenario where the Lib Dems just squeak by thanks to AV and put the Tories in again? What benefit does the detour through AV provide? Isn't it just a more complicated and slightly riskier route to the same goal?
There were those who strongly opposed AV on the grounds that losing the vote would destabilise the coalition and bring a general election closer:
On the other hand:
Some people had technical arguments in favour of AV:
One reader was strongly in favour of a majoritarian system, which is surprising as he's a Lib Dem. Another was worried that the Tories would win an early election, to which I can only respond that the UK Polling Report's projection based on the current state of the polls forecasts a Labour majority of 86. And at least one reader believes that voting is just part of the system, like, and I shouldn't bother.
In general, it seems to me that the problem is basically whether you consider the Lib Dems to be a credible partner for a left-wing government. If so, then all the stuff about a progressive majority and keeping the Tories out of as many seats as possible retains its force, up to a point. But only up to a point. One thing we know now that we didn't in May, 2010 is that the Lib Dems are indeed capable of enabling a radical Tory government. For the "anti-Tory AV" model to work, you have to assume that Labour-Lib Dem coalitions will drag the political spectrum far enough to the Left to balance out the inevitable periods of Tory-Lib Dem coalition. That might be true in a STV world where a Labour-led coalition would have to be concerned about its left flank, but it wouldn't be true in an AV world where, assuming mediocrity, the election would be decided by the Liberals. Of course, the Liberals might be a moderating influence on the Tories, but have we seen that much evidence of this?
And if you don't believe they can be treated as a reliable factor in the Left's calculations, well, you just have to consider them to be Tories operationally, more like the Aussies' National party than the FDP.
In fact, I'm coming around to the view that AV itself sucks. Isn't it just a way of dignifying swing-voter politics? Rather than hypertargeting five people in the bit of Stevenage with no smelly foreigners, close to the Tesco and just far enough from the motorway, isn't it just a way of redefining them as the Lib Dem base?
So what about "no, and campaign to get STV on the next Labour manifesto"?
(Someone also dropped off this link, which makes a strong argument against letting Labour become a second preference party.)
(I really am starting to talk like I'm back in again, aren't I?)
A few of you were supportive on the grounds that it was a start and it would be possible to demand further improvement later.
NomadUK said..I say vote yes. It's not great, but it greases the skids by changing the system; once changed, it's that much easier to change it again — much as the Reform Act 1832, whilst imperfect, led to far greater changes in the electoral system. If AV is rejected, it'll be touted as public approval of the current system, and it'll be a generation or more before anyone dares try again.
9:21 AM
Jonathan Hopkin said...Agree with Nomad. If you want PR, voting against this makes it less likely. Remember the sinking of devolution in the 1970s? Took 20 years to get that going again. Nick Clegg is ****ed anyway, AV isn't going to save him.
11:21 AM
Pauline said...I agree with Nomad as well. Just think of the "told you so" smirking if there's a no vote. And I can't bring myself to side with Cameron and bloody Nick Griffin.
The problem here is that every attempt to model its effects I've seen primarily benefits the Lib Dems. All other things being equal, this fulfils their primary interest in supporting electoral reform. More proportionality starts to bring other parties into the game. In a transitional AV system, the Lib Dems would be in the position of the German FDP, and they wouldn't have any interest in weakening this position. They would tend to swing against anyone who suggested STV or more, and as kingmakers, prevent it from happening. If making a move on electoral reform really did make it easier to go further, we'd somehow have to get to AV without having a single swing party with an interest in sticking at AV. Essentially, we'd need to have a Labour government with a manifesto commitment to STV, which requires that the Lib Dems get such a thrashing that even the AV bonus can't keep them relevant.
Well, I can certainly imagine the Lib Dems getting a thrashing at the next election. But if the point is to beat the coalition and elect a strong Labour government, either on its own or as a hegemonic coalition partner with a few Lib Dem survivors, and then pass STV, why risk the scenario where the Lib Dems just squeak by thanks to AV and put the Tories in again? What benefit does the detour through AV provide? Isn't it just a more complicated and slightly riskier route to the same goal?
There were those who strongly opposed AV on the grounds that losing the vote would destabilise the coalition and bring a general election closer:
Phil - April 11, 2011 at 10:56 am: Anything that makes the coalition less cohesive is good for us (and for the country), as is anything that stops the Lib Dem leadership from carrying on as if 2010 was politics as usual. Turn it round: the fact that a Yes vote would make Nick Clegg happy wouldn’t be a good enough reason to vote No, but the prospect of a Yes vote consolidating Clegg’s leadership of the Lib Dems and hence stabilising the coalition is quite good enough for me.
Chris Williams - April 11, 2011 at 11:05 am: I’m with Phil on this: vote No to split the Lib Dems and bring down the Coalition ASAP. The next lot in will find it harder to screw up the public sector.
On the other hand:
Tom said..." I'm teetering between the principle of spanking Clegg and the principle of doing anything the Murdoch papers are lying about" Organ grinder or monkey? Yes all the way, baby. 8:18 PM
Some people had technical arguments in favour of AV:
Raphael - April 10, 2011 at 10:14 pm: As far as I can tell, if voters don’t act too stupidly, the main effect of AV in Britain would be that the Tories (or UKIP, or the BNP,) wouldn’t be able anymore to win a constituency where most voters are more or less left of centre or centre-left through a split in the left-leaning vote.
Which, going by past results, might mean that it would become a lot more difficult for the Tories to win a majority in the Commons anytime soon, or even to get as close to an outright majority as they’re now again.
Phil Hunt: April 10, 2011 at 6:47 pm: Another advantage of AV — it makes it easier to get rid of unpopular MPs. I would love to see the look on Clegg’s face if AV wins and the voters of Sheffield Hallam use it to get rid of him.
Anonymous said...STV is fairer. But AV at least prevents you 'wasting' your vote by voting for a minor party. And over time minor parties can grow in strength to win an AV seat. Adam Brandt winning the seat of Melbourne for the Greens at the last Federal election being a case in point. 12:50 PM
One reader was strongly in favour of a majoritarian system, which is surprising as he's a Lib Dem. Another was worried that the Tories would win an early election, to which I can only respond that the UK Polling Report's projection based on the current state of the polls forecasts a Labour majority of 86. And at least one reader believes that voting is just part of the system, like, and I shouldn't bother.
In general, it seems to me that the problem is basically whether you consider the Lib Dems to be a credible partner for a left-wing government. If so, then all the stuff about a progressive majority and keeping the Tories out of as many seats as possible retains its force, up to a point. But only up to a point. One thing we know now that we didn't in May, 2010 is that the Lib Dems are indeed capable of enabling a radical Tory government. For the "anti-Tory AV" model to work, you have to assume that Labour-Lib Dem coalitions will drag the political spectrum far enough to the Left to balance out the inevitable periods of Tory-Lib Dem coalition. That might be true in a STV world where a Labour-led coalition would have to be concerned about its left flank, but it wouldn't be true in an AV world where, assuming mediocrity, the election would be decided by the Liberals. Of course, the Liberals might be a moderating influence on the Tories, but have we seen that much evidence of this?
And if you don't believe they can be treated as a reliable factor in the Left's calculations, well, you just have to consider them to be Tories operationally, more like the Aussies' National party than the FDP.
In fact, I'm coming around to the view that AV itself sucks. Isn't it just a way of dignifying swing-voter politics? Rather than hypertargeting five people in the bit of Stevenage with no smelly foreigners, close to the Tesco and just far enough from the motorway, isn't it just a way of redefining them as the Lib Dem base?
So what about "no, and campaign to get STV on the next Labour manifesto"?
(Someone also dropped off this link, which makes a strong argument against letting Labour become a second preference party.)
(I really am starting to talk like I'm back in again, aren't I?)
ring ring! who's there?
On the same day that NATO sort-of apologised for a fratricide incident in which a group of tanks the Libyan rebels had put into service were mistaken for Libyan government tanks, it turned out that the MOD was going to send the rebels 500 satellite phones. Well, you can see the point, but the first thing that came to mind was - what? now? why didn't this happen weeks ago? Is this whole campaign being managed by clowns? And then, of course, I remembered Dave from PR and Sarko and Liam He's a doctor, you know and Pocket Bismarck. Right.
But then, there's the Big Society. This is a deeply cool story - Libyan GSM engineers work out how to take over the network in rebel territory and get it going again. The WSJ overstates some elements - it's not so much that Gadhafi's government designed the network to be centralised in Tripoli, GSM networks are very centralised by design - but overall it's a pretty good account. They set up their own switch, home and visitor location registers, and international gateway with satellite connectivity, piped all the base station controllers in their territory into their own set-up, and obtained a copy of the original Libyana HLR with all the phone numbers. Fortunately they decided to let everyone make free calls (viva la revolucion!), or they'd have still been waiting for the billing system to be integrated six months later, whether in the minister's office or the Libyan Lubyanka.
Ironically, they got quite a bit of help from, of all telcos, Etisalat, the UAE's national operator. They lent them a lot of equipment and provided the satellite hookup and international access. This is amusing as Etisalat is famous for censoring more Web sites than the Chinese Great Firewall. For their part, the monster Chinese manufacturer Huawei refused to have anything to do with the rebels (or should that be "splittists"?)
This is good as far as it goes, but nobody in NATO CAOC-9 in Naples or the former AIRSOUTH now in Izmir or anywhere else with a NATO TLA is going to let random cell phones talk to the airpower infrastructure. Why didn't anyone send those satellite phones earlier? Ah, yes, clowns.
Of course, there's a possibility that they may have been worrying about releasing them into the wild. Here's Secret Défense confirming assorted loose MANPADs wandering about. But they're more trackable than arms, less directly dangerous, and far easier to buy anyway.
In other news, there's a really excellent piece on the Toyota Land Cruiser as an engine of war in the FAZ, for German-speakers only.
But then, there's the Big Society. This is a deeply cool story - Libyan GSM engineers work out how to take over the network in rebel territory and get it going again. The WSJ overstates some elements - it's not so much that Gadhafi's government designed the network to be centralised in Tripoli, GSM networks are very centralised by design - but overall it's a pretty good account. They set up their own switch, home and visitor location registers, and international gateway with satellite connectivity, piped all the base station controllers in their territory into their own set-up, and obtained a copy of the original Libyana HLR with all the phone numbers. Fortunately they decided to let everyone make free calls (viva la revolucion!), or they'd have still been waiting for the billing system to be integrated six months later, whether in the minister's office or the Libyan Lubyanka.
Ironically, they got quite a bit of help from, of all telcos, Etisalat, the UAE's national operator. They lent them a lot of equipment and provided the satellite hookup and international access. This is amusing as Etisalat is famous for censoring more Web sites than the Chinese Great Firewall. For their part, the monster Chinese manufacturer Huawei refused to have anything to do with the rebels (or should that be "splittists"?)
This is good as far as it goes, but nobody in NATO CAOC-9 in Naples or the former AIRSOUTH now in Izmir or anywhere else with a NATO TLA is going to let random cell phones talk to the airpower infrastructure. Why didn't anyone send those satellite phones earlier? Ah, yes, clowns.
Of course, there's a possibility that they may have been worrying about releasing them into the wild. Here's Secret Défense confirming assorted loose MANPADs wandering about. But they're more trackable than arms, less directly dangerous, and far easier to buy anyway.
In other news, there's a really excellent piece on the Toyota Land Cruiser as an engine of war in the FAZ, for German-speakers only.
Monday, April 11, 2011
of course, Gauck tells me my file is probably in Moscow
Well, if you're the prime minister, you're not allowed to fight back against your enemies in case you win because Gus O'Donnell says so, as long as the enemies aren't the right kind of enemies like Coronation Street editors and UNISON hospital porters. All clear so far? When are the Tory apologies to O'Donnell going to show up - they thought he was biased against them....
Further, interesting subplot - Brown told Peter Mandelson his phone might have been compromised. Mandelson asked the Information Commissioner, who had nothing to say. Of course he didn't - as far as I know he had nothing to do with the case. If Mandelson had wanted to know, he needed to ask the police. Was Mandelson trying not to find out, so as not to burn his bridges with the Murdochs? Or was he just ill-informed?
Also, did the central government have any communications security at all? Did CESG or MI5 not have anything at all to say about this? Didn't any of them just change their damn password, or even change their damn number?
Further, interesting subplot - Brown told Peter Mandelson his phone might have been compromised. Mandelson asked the Information Commissioner, who had nothing to say. Of course he didn't - as far as I know he had nothing to do with the case. If Mandelson had wanted to know, he needed to ask the police. Was Mandelson trying not to find out, so as not to burn his bridges with the Murdochs? Or was he just ill-informed?
Also, did the central government have any communications security at all? Did CESG or MI5 not have anything at all to say about this? Didn't any of them just change their damn password, or even change their damn number?
Sunday, April 10, 2011
three lessons on risks: a movie
Daniel Davies is notorious for making more than full use of a joke once he gets hold of it. I think this is the original source (perhaps even the Urquell) of his line that Black Swan author Nassim Nicholas Taleb must be furious at what Hollywood did to his book in development.
Well, I watched the movie on a plane the other day. It was that or The Social Network - I was planning on a week in Silicon Valley surrounded by tightly wound super-ambitious geeks, so I get enough of that at work. As it happens, there are a couple of good lessons about risk in Black Swan. Perhaps Taleb shouldn't be so touchy about it after all.
1. Tail-risk is real
Just as Nassim Taleb said in the book, no matter how good your planning, you can't hedge everything and you will tend to underestimate the weirder and wilder ends of the distribution. One day, something not just bad but incalculably weird, something you never expected you didn't expect, will come rapid-roping into your back garden and piss in the pool. Of course, it's likely to happen on stage on the first night at the worst possible moment. You'll have to be ready, but you can only be ready in a general sense. Get your trigger movements right - far better to be calling an ambulance and plunging into the fray than locked in the bunker with a PR agent and a large amount of toilet paper. Act right in the crisis and much will be forgiven.
Inevitably, if you want a clue, look at the things you try to repress and deny and don't believe could ever happen. That's why you deny them.
2. That said, you've got to put up with it
All precautions must be seen in the light of the scale of the threat. Too much security is as dangerous as too little (this may be more Schneier than Taleb). Without a certain amount of optimism bias and risk tolerance, you'll never get anything done. In fact, you'll end being terrified of your shadow. (And why did you choose the word shadow, with its, ah, many meanings, Mr. Garrovell?) Your colleagues may well wish they had your job, but that's no reason to kill yourself. In fact, after a certain level of neurosis is passed, self-protection shades over into self-sabotage - delivering just what you imagine your enemies want, whether they be real or imagined.
3. Don't draw conclusions based on regional accents
Black Swan is the only movie I can think of in which New Yorkers see an outsider - a Californian - as being unimaginably evil, sophisticated, cool, and cunning. In fact, this was the plot detail that kept coming back to me. Wall Street and City investors in dozens of regional mortgage lenders that turned into financial neutron bombs imagined they were smarter than the offcomed'uns.
Well, I watched the movie on a plane the other day. It was that or The Social Network - I was planning on a week in Silicon Valley surrounded by tightly wound super-ambitious geeks, so I get enough of that at work. As it happens, there are a couple of good lessons about risk in Black Swan. Perhaps Taleb shouldn't be so touchy about it after all.
1. Tail-risk is real
Just as Nassim Taleb said in the book, no matter how good your planning, you can't hedge everything and you will tend to underestimate the weirder and wilder ends of the distribution. One day, something not just bad but incalculably weird, something you never expected you didn't expect, will come rapid-roping into your back garden and piss in the pool. Of course, it's likely to happen on stage on the first night at the worst possible moment. You'll have to be ready, but you can only be ready in a general sense. Get your trigger movements right - far better to be calling an ambulance and plunging into the fray than locked in the bunker with a PR agent and a large amount of toilet paper. Act right in the crisis and much will be forgiven.
Inevitably, if you want a clue, look at the things you try to repress and deny and don't believe could ever happen. That's why you deny them.
2. That said, you've got to put up with it
All precautions must be seen in the light of the scale of the threat. Too much security is as dangerous as too little (this may be more Schneier than Taleb). Without a certain amount of optimism bias and risk tolerance, you'll never get anything done. In fact, you'll end being terrified of your shadow. (And why did you choose the word shadow, with its, ah, many meanings, Mr. Garrovell?) Your colleagues may well wish they had your job, but that's no reason to kill yourself. In fact, after a certain level of neurosis is passed, self-protection shades over into self-sabotage - delivering just what you imagine your enemies want, whether they be real or imagined.
3. Don't draw conclusions based on regional accents
Black Swan is the only movie I can think of in which New Yorkers see an outsider - a Californian - as being unimaginably evil, sophisticated, cool, and cunning. In fact, this was the plot detail that kept coming back to me. Wall Street and City investors in dozens of regional mortgage lenders that turned into financial neutron bombs imagined they were smarter than the offcomed'uns.
I cannot begin to theorise what may have caused such a catastrophic malfunction
So, a free, jetlagged afternoon by the pool in Palo Alto, after this experience. What to do? Obviously, hack on some code. I dragged out the lobby analyzer project and got it to actually spit out ministers, lobbyists, and MPs, with their weighted degrees in the network, onto the command line. The conclusions are dreadful and confirm all my preliminary work. We are being ruled by Francis Maude and David Willetts. They both have significantly higher scores than the Prime Minister, with Willetts topping the poll. Of course, this is using the idea the wrong way up, but presumably the lobbyists' choice of who to lobby contains information about their perception of ministerial importance and influence.
BritiLeaks: the best website that doesn't exist yet
Absolutely, certainly, without a doubt the most important bit in the News of the World case:
E-mails. Thousands of them. Hundreds of thousands of them. Just once, to have such means at my disposal! To have as many planes as this!... I don't know how the claimants' briefs intend to handle this epic dump, but I can't think of any more interesting political document going. Does anyone know if any of this stuff will end up in the public domain?
It's no surprise, in the light of that, that they're serving the people, with the people's currency, although if I read this right, it doesn't necessarily stop the disclosure. And, of course, there may well be more claims now a precedent has been set. There is of course also the police operation, but I don't hold out too much hope there.
And then there's this. Yet another story showing the police in a very bad light, in the week I had to thank'em. It doesn't show the industry in a great one either, as four out of five carriers seem to have treated the message from the police as a legalistic excuse to do nothing. Actually, three out of four - there's no mention of 3UK at all. I wonder why? O2 gets the prize, having apparently decided unilaterally to inform all its subscribers who were affected. The Guardian gets it right here, making the point that:
This. RIPA III requires telecommunications data retention for 12 months, no more and no less. It looks horribly like there was an effort to ensure Screws' comedy STELLAR WIND wasn't disclosed until the logs were purged at the end of the canonical 12 months.
Of course, this won't help them any with the people who were spied upon last year. It looks even more as if the police accepted promises from News International that they would behave in future. Will they get their act together and sue, already?
Last Friday, a high court judge ordered NoW to make available Mulcaire's notes to the growing list of people suing the paper. Justice Geoffrey Vos, who is in charge of the hacking cases, ordered "rolling disclosure" to all claimants.
Hundreds of thousands of emails will now be handed over to alleged victims.
E-mails. Thousands of them. Hundreds of thousands of them. Just once, to have such means at my disposal! To have as many planes as this!... I don't know how the claimants' briefs intend to handle this epic dump, but I can't think of any more interesting political document going. Does anyone know if any of this stuff will end up in the public domain?
It's no surprise, in the light of that, that they're serving the people, with the people's currency, although if I read this right, it doesn't necessarily stop the disclosure. And, of course, there may well be more claims now a precedent has been set. There is of course also the police operation, but I don't hold out too much hope there.
And then there's this. Yet another story showing the police in a very bad light, in the week I had to thank'em. It doesn't show the industry in a great one either, as four out of five carriers seem to have treated the message from the police as a legalistic excuse to do nothing. Actually, three out of four - there's no mention of 3UK at all. I wonder why? O2 gets the prize, having apparently decided unilaterally to inform all its subscribers who were affected. The Guardian gets it right here, making the point that:
It also means many of the victims were deprived of the chance to check the call data, which is kept by the phone companies for only 12 months, and that they had no opportunity to change their pin codes or to assess the damage done by the interception of their messages.
This. RIPA III requires telecommunications data retention for 12 months, no more and no less. It looks horribly like there was an effort to ensure Screws' comedy STELLAR WIND wasn't disclosed until the logs were purged at the end of the canonical 12 months.
Of course, this won't help them any with the people who were spied upon last year. It looks even more as if the police accepted promises from News International that they would behave in future. Will they get their act together and sue, already?
Labels:
Met,
networks,
press,
special relationships,
surveillance
Alternative vote: a reconnaissance by fire
I've realised that I don't know where I stand on electoral reform. (With that, even my readership vanishes into the distance like Libyan rebels encountering Gadhafi's one loyal artillery observer.) So I've decided to abuse both sides in the hope that they react interestingly.
For AV: What, you want to give Nick Clegg what he wants? Crush! Like! Rat! Are you chicken? Anyway, it's not proper proportional voting - it's just a weak compromise that happens to give Cleggy Boy a life pension without even the embarrassment of going to the Lords. Alan Beattie is right. The Left, the Greens, the 'kippers and fascists and God knows what, they'll all be shut out just as much as ever. And, y'know, AV is cheating, like stealth aircraft and quantum computing and gas barbecues. Caroline Lucas didn't need AV to get elected - she had to do it the hard way. And if the Lib Dems get AV, do you really think they'll be in a hurry to concede STV? They get all the benefit of AV, and then they'll pocket it and just sit there. No is good. Look at the polls, anyway - the solution is to force a general election as soon as possible. Power, baby!
Against AV: Well, nice mates you've got there. Have you seen the fuckos and liars and thugs and gargoyles on your side? There's "Tom" "Newton"-"Dunn" of the "Sun", talking about "Baroness Warsi warns that the DANGEROUS alternative vote system will let FASCISTS into the House of Commons..." Rather than the papers, where they belong. It's the M factor - whatever Murdoch wants is evidently wrong. There's the charming No campaign, which is so rich it refused to take its public funding so it didn't have to admit to who's funding it. And just imagine Eric Pickles' face! And Redwood! The horror of it! We have a moral responsibility to vote against the Tory in all its forms.
Does that cover everything? I'm teetering between the principle of spanking Clegg and the principle of doing anything the Murdoch papers are lying about. On the merits, as far as I can see, AV is a little better than the current system, but not really enough to be worth having and certainly not enough to let the Lib Dems off the hook. And I'm in the mood for approval voting, the system where you cross off all the candidates you absolutely reject and the least hated wins. But please try to convince me.
For AV: What, you want to give Nick Clegg what he wants? Crush! Like! Rat! Are you chicken? Anyway, it's not proper proportional voting - it's just a weak compromise that happens to give Cleggy Boy a life pension without even the embarrassment of going to the Lords. Alan Beattie is right. The Left, the Greens, the 'kippers and fascists and God knows what, they'll all be shut out just as much as ever. And, y'know, AV is cheating, like stealth aircraft and quantum computing and gas barbecues. Caroline Lucas didn't need AV to get elected - she had to do it the hard way. And if the Lib Dems get AV, do you really think they'll be in a hurry to concede STV? They get all the benefit of AV, and then they'll pocket it and just sit there. No is good. Look at the polls, anyway - the solution is to force a general election as soon as possible. Power, baby!
Against AV: Well, nice mates you've got there. Have you seen the fuckos and liars and thugs and gargoyles on your side? There's "Tom" "Newton"-"Dunn" of the "Sun", talking about "Baroness Warsi warns that the DANGEROUS alternative vote system will let FASCISTS into the House of Commons..." Rather than the papers, where they belong. It's the M factor - whatever Murdoch wants is evidently wrong. There's the charming No campaign, which is so rich it refused to take its public funding so it didn't have to admit to who's funding it. And just imagine Eric Pickles' face! And Redwood! The horror of it! We have a moral responsibility to vote against the Tory in all its forms.
Does that cover everything? I'm teetering between the principle of spanking Clegg and the principle of doing anything the Murdoch papers are lying about. On the merits, as far as I can see, AV is a little better than the current system, but not really enough to be worth having and certainly not enough to let the Lib Dems off the hook. And I'm in the mood for approval voting, the system where you cross off all the candidates you absolutely reject and the least hated wins. But please try to convince me.
A regrettable breach of security regulations
Remember this post from 2006, and especially this one from a year later on the next big miscarriage of justice? Well, look what just happened. It's far worse than even I thought - the police were well aware that there were serious problems with the Landslide case as early as February, 2003. Specifically, the old National Crime Squad seems to have been extremely gung-ho about the whole project while the regional police forces were much more sceptical. Later the whole thing was slung to the Child Exploitation and Online Protection Centre, one of the weird sort-of police agencies that proliferated in the late Blair years. Meanwhile, a suspect has succeeded in claiming damages. Both cases show various police forces in a very bad light indeed - the US postal inspector in the first suddenly retired to look after his "sick wife" when his evidence was challenged, while as for Hertfordshire Police:
Feel the fremdscham, baby!
Meanwhile, my bank card has been compromised. So I was in San Francisco of a Sunday, walking around the Tenderloin looking for a cash point that wasn't looking back at me with mischief in its eye. Preferably one attached to a bank. I eventually walked up as far as Van Ness and found a Wells Fargo branch. It wouldn't give me any money, nor would the Bank of America. So I ended up phoning the bank at extortionate roaming rates, standing on the forecourt with a small encampment of the homeless. Thinking that I had less US currency to my name than they did, I struggled through the IVR thickets, confirmed my salary hadn't somehow vanished, and got into a queue to report that the fraud-detection robots had zapped me. I stayed on the line until AT&T dropped the call after 12 minutes. The phone started whining; it's like a little jet fighter. You can do a lot of cool things with it, but it's best not to go too far from the refuelling tanker or you're screwed. Back to the hotel. I tried to call them on Skype, but AT&T's WLAN was too bad to hear the IVRs. I plugged in the phone, called again, explained that I didn't want to report the card stolen but rather the opposite, and sat in 23 minutes of queues. Curses...curses...24 hour fraud algorithms...not 24 hour staffing, though...why not call me?...banks...banks...banks...!
And then I got through. And the fraud investigators told me that the police had found my card in a list of cloned cards offered for sale on the black market. In the circumstances, they hadn't called me or given out any information for fear of giving away the secret, as the investigation was still going on. Oh...right. They listed some transactions, agreed to let me withdraw up to £100 a day in cash and honour direct debits, and left the Visa facility frozen. They refused to say anything about where or when the security breach might have occurred, although I think the detail about the Visa card might be significant. Call us when you get back to the UK - and by the way, here's the direct number.
The whole incident had just been annoying up to that point, but this changed the game. I was left with a whole load of surplus indignation on my hands past its use-by date. It cluttered up my room at the Phoenix like a chunk of un-Californian, clanking machinery. I suppose I could spend it on the thieves, but who were they? Rather than just harassing me and profiteering, my bank had actually done something I could only agree with. And the police had actually protected me from an actual crime, without my even noticing, with the occult efficiency Norman Lewis said had attached itself to the word "intelligence".
As far as I know, no money is missing, but I haven't audited as many as 14 months' worth of transactions through my current account yet. That's since this card was issued - they couldn't give me any other bounds on it. After all, as they said, it was impossible to say how far the list of cards had been sold on by whoever had originally collected them.
Anyway, I didn't even need to draw any more cash after the first $100. My expenses in Silicon Valley were unusually frugal - the nearest I came to spending significant amounts of money was trying to catch up with two colleagues who'd gone out looking for amusement. (I spent 20 minutes looking for a cab in Palo Alto at 10 o'clock at night and eventually gave up, having noticed that there seemed to be less traffic on the roads at that time of night than I would have expected in a Yorkshire Dales village.) I read the two Operation Ore articles and logged them for future use. As briefed, I called HSBC on my arrival back in Britain and they initiated a new card.
And it was only as I wrote this that I remembered I ought to be scared. After all, it is impossible to say how far the list of cards...
Despite this, the officer, Detective Constable Brian Hopkins, pressed three charges of possession of indecent images of children. Mr Justice Mackay said he cut a "rather pathetic figure" in the witness box, having initially claimed he could not give evidence because of a psychiatric condition....The judge found that Mr Hopkins, who has since left policing, not only had "no honest belief in the possession charges when he caused them to be brought against [Mr Clifford]", but did so "to protect his own position".
Feel the fremdscham, baby!
Meanwhile, my bank card has been compromised. So I was in San Francisco of a Sunday, walking around the Tenderloin looking for a cash point that wasn't looking back at me with mischief in its eye. Preferably one attached to a bank. I eventually walked up as far as Van Ness and found a Wells Fargo branch. It wouldn't give me any money, nor would the Bank of America. So I ended up phoning the bank at extortionate roaming rates, standing on the forecourt with a small encampment of the homeless. Thinking that I had less US currency to my name than they did, I struggled through the IVR thickets, confirmed my salary hadn't somehow vanished, and got into a queue to report that the fraud-detection robots had zapped me. I stayed on the line until AT&T dropped the call after 12 minutes. The phone started whining; it's like a little jet fighter. You can do a lot of cool things with it, but it's best not to go too far from the refuelling tanker or you're screwed. Back to the hotel. I tried to call them on Skype, but AT&T's WLAN was too bad to hear the IVRs. I plugged in the phone, called again, explained that I didn't want to report the card stolen but rather the opposite, and sat in 23 minutes of queues. Curses...curses...24 hour fraud algorithms...not 24 hour staffing, though...why not call me?...banks...banks...banks...!
And then I got through. And the fraud investigators told me that the police had found my card in a list of cloned cards offered for sale on the black market. In the circumstances, they hadn't called me or given out any information for fear of giving away the secret, as the investigation was still going on. Oh...right. They listed some transactions, agreed to let me withdraw up to £100 a day in cash and honour direct debits, and left the Visa facility frozen. They refused to say anything about where or when the security breach might have occurred, although I think the detail about the Visa card might be significant. Call us when you get back to the UK - and by the way, here's the direct number.
The whole incident had just been annoying up to that point, but this changed the game. I was left with a whole load of surplus indignation on my hands past its use-by date. It cluttered up my room at the Phoenix like a chunk of un-Californian, clanking machinery. I suppose I could spend it on the thieves, but who were they? Rather than just harassing me and profiteering, my bank had actually done something I could only agree with. And the police had actually protected me from an actual crime, without my even noticing, with the occult efficiency Norman Lewis said had attached itself to the word "intelligence".
As far as I know, no money is missing, but I haven't audited as many as 14 months' worth of transactions through my current account yet. That's since this card was issued - they couldn't give me any other bounds on it. After all, as they said, it was impossible to say how far the list of cards had been sold on by whoever had originally collected them.
Anyway, I didn't even need to draw any more cash after the first $100. My expenses in Silicon Valley were unusually frugal - the nearest I came to spending significant amounts of money was trying to catch up with two colleagues who'd gone out looking for amusement. (I spent 20 minutes looking for a cab in Palo Alto at 10 o'clock at night and eventually gave up, having noticed that there seemed to be less traffic on the roads at that time of night than I would have expected in a Yorkshire Dales village.) I read the two Operation Ore articles and logged them for future use. As briefed, I called HSBC on my arrival back in Britain and they initiated a new card.
And it was only as I wrote this that I remembered I ought to be scared. After all, it is impossible to say how far the list of cards...
Friday, April 01, 2011
Scaling and scoping the NYT paywall
Amusingly for a comment on scalability, I couldn't post this on D^2's thread because Blogger was in a state. Anyway, it's well into the category of "comments that really ought to be posts" so here goes. So various people are wondering how the New York Times managed to spend $50m on setting up their paywall. D^2 reckons that they're overstating, for basically cynical reasons. I think it's more fundamental than that.
The complexity of the rules makes it sound like a telco billing system more than anything else - all about rating and charging lots and lots of events in close to real-time based on a hugely complicated rate-card. You'd be amazed how many software companies are sustained by this issue. It's expensive. The NYT is counting pages served to members (easy) and nonmembers (hard), differentiating between referral sources, and counting different pages differently. Further, it's got to do it quickly. Latency from the US West Coast (their worst case scenario) to nytimes.com is currently about 80 milliseconds. User-interface research suggests that people perceive a response as instant at 100ms - web surfing is a fairly latency tolerant application, but when you think that the server itself takes some time to fetch the page and the data rate in the last mile will restrict how quickly it can be served, there's a very limited budget of time for the paywall to do its stuff without annoying the hell out of everyone.
Although the numbers of transactions won't be as savage, doing real-time rating for the whole NYT website is going to be a significant scalability challenge. Alexa reckons 1.45% of global Web users hit nytimes.com, for example. As comparison, Salesforce.com is 0.4% and that's already a huge engineering challenge (because it's much more complicated behind the scenes). There are apparently 1.6bn "Internet users" - I don't know how that's defined - so that implies that the system must scale to 268 transactions/second (or about 86,400 times the daily reach of my blog!)
A lot of those will be search engines, Internet wildlife, etc, but you still have to tell them to fuck off and therefore it's part of your scale & scope calculations. That's about a tenth of HSBC's online payments processing in 2007, IIRC, or a twentieth of a typical GSM Home Location Register. (The usual rule of thumb for those is 5 kilotransactions/second.) But - and it's the original big but - you need to provision for the peak. Peak usage, not average usage, determines scale and cost. Even if your traffic distribution was weirdly well-behaved and followed a normal distribution, you'd encounter a over 95th percentile event one day in every 20. And network traffic doesn't, it's usually more, ahem, leptokurtotic. So we've got to multiply that by their peak/mean ratio.
And it's a single point of failure, so it has to be robust (or at least fail to a default-open state but not too often). I for one can't wait for the High Scalability article on it.
So it's basically similar in scalability, complexity, and availability to a decent sized MVNO's billing infrastructure, and you'd be delighted to get away with change from £20m for that.
The complexity of the rules makes it sound like a telco billing system more than anything else - all about rating and charging lots and lots of events in close to real-time based on a hugely complicated rate-card. You'd be amazed how many software companies are sustained by this issue. It's expensive. The NYT is counting pages served to members (easy) and nonmembers (hard), differentiating between referral sources, and counting different pages differently. Further, it's got to do it quickly. Latency from the US West Coast (their worst case scenario) to nytimes.com is currently about 80 milliseconds. User-interface research suggests that people perceive a response as instant at 100ms - web surfing is a fairly latency tolerant application, but when you think that the server itself takes some time to fetch the page and the data rate in the last mile will restrict how quickly it can be served, there's a very limited budget of time for the paywall to do its stuff without annoying the hell out of everyone.
Although the numbers of transactions won't be as savage, doing real-time rating for the whole NYT website is going to be a significant scalability challenge. Alexa reckons 1.45% of global Web users hit nytimes.com, for example. As comparison, Salesforce.com is 0.4% and that's already a huge engineering challenge (because it's much more complicated behind the scenes). There are apparently 1.6bn "Internet users" - I don't know how that's defined - so that implies that the system must scale to 268 transactions/second (or about 86,400 times the daily reach of my blog!)
A lot of those will be search engines, Internet wildlife, etc, but you still have to tell them to fuck off and therefore it's part of your scale & scope calculations. That's about a tenth of HSBC's online payments processing in 2007, IIRC, or a twentieth of a typical GSM Home Location Register. (The usual rule of thumb for those is 5 kilotransactions/second.) But - and it's the original big but - you need to provision for the peak. Peak usage, not average usage, determines scale and cost. Even if your traffic distribution was weirdly well-behaved and followed a normal distribution, you'd encounter a over 95th percentile event one day in every 20. And network traffic doesn't, it's usually more, ahem, leptokurtotic. So we've got to multiply that by their peak/mean ratio.
And it's a single point of failure, so it has to be robust (or at least fail to a default-open state but not too often). I for one can't wait for the High Scalability article on it.
So it's basically similar in scalability, complexity, and availability to a decent sized MVNO's billing infrastructure, and you'd be delighted to get away with change from £20m for that.
Subscribe to:
Posts (Atom)