Friday, April 01, 2011

Scaling and scoping the NYT paywall

Amusingly for a comment on scalability, I couldn't post this on D^2's thread because Blogger was in a state. Anyway, it's well into the category of "comments that really ought to be posts" so here goes. So various people are wondering how the New York Times managed to spend $50m on setting up their paywall. D^2 reckons that they're overstating, for basically cynical reasons. I think it's more fundamental than that.

The complexity of the rules makes it sound like a telco billing system more than anything else - all about rating and charging lots and lots of events in close to real-time based on a hugely complicated rate-card. You'd be amazed how many software companies are sustained by this issue. It's expensive. The NYT is counting pages served to members (easy) and nonmembers (hard), differentiating between referral sources, and counting different pages differently. Further, it's got to do it quickly. Latency from the US West Coast (their worst case scenario) to nytimes.com is currently about 80 milliseconds. User-interface research suggests that people perceive a response as instant at 100ms - web surfing is a fairly latency tolerant application, but when you think that the server itself takes some time to fetch the page and the data rate in the last mile will restrict how quickly it can be served, there's a very limited budget of time for the paywall to do its stuff without annoying the hell out of everyone.

Although the numbers of transactions won't be as savage, doing real-time rating for the whole NYT website is going to be a significant scalability challenge. Alexa reckons 1.45% of global Web users hit nytimes.com, for example. As comparison, Salesforce.com is 0.4% and that's already a huge engineering challenge (because it's much more complicated behind the scenes). There are apparently 1.6bn "Internet users" - I don't know how that's defined - so that implies that the system must scale to 268 transactions/second (or about 86,400 times the daily reach of my blog!)

A lot of those will be search engines, Internet wildlife, etc, but you still have to tell them to fuck off and therefore it's part of your scale & scope calculations. That's about a tenth of HSBC's online payments processing in 2007, IIRC, or a twentieth of a typical GSM Home Location Register. (The usual rule of thumb for those is 5 kilotransactions/second.) But - and it's the original big but - you need to provision for the peak. Peak usage, not average usage, determines scale and cost. Even if your traffic distribution was weirdly well-behaved and followed a normal distribution, you'd encounter a over 95th percentile event one day in every 20. And network traffic doesn't, it's usually more, ahem, leptokurtotic. So we've got to multiply that by their peak/mean ratio.

And it's a single point of failure, so it has to be robust (or at least fail to a default-open state but not too often). I for one can't wait for the High Scalability article on it.

So it's basically similar in scalability, complexity, and availability to a decent sized MVNO's billing infrastructure, and you'd be delighted to get away with change from £20m for that.

Thursday, March 31, 2011

Non-Thursday music link, on Thursday

We've not done one of these for a while, and I meant to get around to a Loleatta Holloway tribute link at some point. Here goes - an absolute killer of a remix that dropped out of the YouTube playlist by chance.



Meanwhile, depress yourself!

The auction of machine tools and fittings, including a 10m antique boardroom table and 20 matching chairs is now being advertised; it takes place at the end of the month.


Ah, where did I leave that post? This stands up pretty well too.

I've been rating my way through the SXSW torrent, which is far less fun than it sounds. Sturgeon's law applies - in fact it's worse than 90% of everything being shit, it's more like 90% of everything is inoffensive and a significant chunk of the other 10% is offensive. But then it got worse, thanks to this tweet. The best is the enemy of the good, but they are natural allies against the mediocre.

Monday, March 28, 2011

a little rugby league blogging

A bit of Rugby League blogging. It's been a weird start to the season. Not so long ago - but after more than a few games - London/sorry/Harlequins RL were top of the league. (Regarding the name, I'm not the only one. I saw a Fulham RL shirt at this weekend's game.) And the top three included Huddersfield and Castleford. Meanwhile, the big four were in the bottom half of the table. It's early season madness, of course.

Things are a bit less weird now - Wigan have started to play more like the defending champions, Warrington have imposed themselves as the best side so far. Their game with St Helens was probably the best so far. But the top three are currently the Wire, Cas, and Fartown, with Saints and Wigan behind. Leeds and Bradford are further off.

Yesterday's match at the Stoop ended up being a surprising classic. Hull FC were good - they damn well should be, given some of the faces in the side, even if nobody's actually related to Willie Mason. London pulled back an early advantage, and eventually hit a blast of form in the second half. At this point they looked good value for anything. Quick. Tough. Neat. It was a great afternoon in west London, a misty bath of sunshine swimming with Airbus A380s on easterly operations, and even the Hull FC people were reasonably pleasant. (I've been hard on them in the past and they occasionally show up in comments to whine about it, but on this occasion, they were almost as nice as Hull Kingston Rovers fans.)

London chose the moment to do another of their epic chokes. Heading for the finish, Hull began to sense a crack or two. This was the moment for (who else) Sean Long, still with his Saints-colours gumshield after all these years, to crank the pressure up. Starting at 30-16 down, they hauled the game in. At 30-26, London looked like setting up a dropgoal attempt. It didn't happen - not much else did - and a couple of tackles later they gave away a daft penalty. Hull scored, pulled ahead, and despite the inevitable short kick-off, London couldn't pull off a two-minute raid. It didn't matter, in fact - they kept scoring. In the end Hull got from 30-16 down to 40-30 up in less than a point a minute. You can see why Saints want Long back. Actually it wasn't so much him but Sam Obst, their stand-off, who really did the deed.

Meanwhile, Keighley lost to Whitehaven 24-10.

Saturday, March 26, 2011

march

I'll be out tomorrow. Obviously. Setting off from Archway tube at 1015. Anyway, updates will be at the twitter feed.

Sunday, March 20, 2011

now he's stopped, he's much better in our sex life and in our general life

At my local gym on Friday, I was accused of abusing steroids. Just like that - I'd finished my weights session and moved on to the abs, when some random guy approached me and said "Here - are you on gear?" No. "I mean, are you on gear?" No, I'm not on steroids. [Cancel the maniac, please] "I mean, you're lifting heavy for your physique, man."

All the while, I couldn't stop thinking of the Geoff Wode scene in Withnail & I. "Look at him! Look at Geoff Wode!" Recreational steroid users and people who know most of the W&I script are probably two groups with a minimal intersection set.

aspirational torturer

It says something about the modern thinkers that one of the Egyptian spooks the Piggipedia team identified turns out to be working in the "Security and Loss Prevention" department of a major hypermarket chain. Having lifted photos with names on them from their HQ, they started searching PofacedBook - sorry, LinkedIn - for them, and discovered many more stories like this. That particular spy had been in charge of infiltrating NGOs before shifting over to the retail sector. Sadly, they started deleting profiles pretty quickly.

the only thing worse than being exploited...

Is if the exploiters miss you out, said Joan Robinson of capitalism.

A twofer of Owen Hatherley on Manchester. Thoughts: it's surely a slightly odd idea that London is rich because of the housing market, rather than the other way around, although I can certainly imagine an unusually dense Blairite town-hall politician getting that impression. Bu then, I wouldn't class the GMC pols as being that dense. And this:

In a way it’s hard to resent them and again this is the major flaw in my stuff about Manchester. The thing is I don’t remember it when it was fucked.


Well, this is the turd in the punch bowl. Is there an even vaguely credible alternative route from about 1983 forwards that goes anywhere else, thinking of the basically hostile central government for most of that period and the various path dependencies? Owen is working on the assumption that without the redevelopment era, we'd have found our way back to the high welfare state in the end, rather than - essentially - Thornton Road in Bradford. It's a sort of sick, el cheapo parody of Tony Wilson urbanism, with converted mills that end up being rented to not one but two serial killers in ten years, and positively Sicilian half-built failed projects like the motorway to nowhere, the Interchange, Abbey National, the Millenium Faith experience, the Alsop master plan, and the rubble zone.

Actually, making a list of 'em, the periodicity between failures seems to be declining over time, the rate picking up, and one of them includes Will Alsop, so perhaps he has a point. But I still think this rant against decay-porn in a US context could be imported.

declare independence. don't let them do that to you!

Alliance Géostrategique is having a month on the theme of independence.

Some thoughts: First of all, we're constantly exhorted to act as individuals but also to be aware of interdependence. "Interdependence" seems to be the neoliberal mirror image of "solidarity" - rather than being a force that makes for positive liberty, it seems to be a negative one. You never here of anything good coming of it - rather, it's always about financial crises, viruses spreading on the airlines, cascade failures in supply chains and Internet routing, terrorists and organised criminal networks. We are asked to fork out in the name of interdependence, while individuals profit.

Of course, this is where the similarity with "solidarity" turns up. The supposed virtues of "solidarity" or "community" are often, even usually, indistinguishable from stifling conservatism, Nosey Parker, and the sort of family ethic where everyone knows the worst about everyone else but some people are never held responsible for it. To be really cynical, you know it's solidarity (or interdependence) when someone's rattling their tin under your nose.

Getting back to the point, in the world of "interdependence", why would anyone want independence? If even major powers are constrained by rules, what's the point? Between the 1980s and the great financial crisis, there was a fashion for a sort of soft nationalism, especially in Europe, in which it was argued that small states were worth having precisely because so many of the big questions of peace and war and fundamental economics had been reserved by institutions like the EU, NATO, the Bretton Woods structures, the WTO, and the less formal systems of the international community. Although there was not much point in having a Scottish Army, by the same token, it didn't matter. Therefore things like "Europe of the regions" and friends were a valid proposition.

One of the most dangerous toys left to a small state (or autonomous province) was its financial system. If you couldn't have a Ruritanian foreign policy, you could decide to be a freewheeling sin city of a financial centre, which would give your ruling elite the sort of self-importance the dance of diplomacy did in the Edwardian era. And, in the years when the financial sector itself was exploding in size, it meant real money. Importantly, the same slice in absolute terms means a lot more in relative terms to a small state. So, everyone and their dog wanted to be their regional money centre. In much the same way as the Edwardian small powers insisted on having a battleship or two of their own, they all insisted on having a bank of sorts and building up whatever local financial institutions were available into investment banks. This could be on the grand scale (RBS, WestLB) or on a much smaller one, like some of the Spanish cajas or the Hypo Alpe-Adria in Jörg Haider's fief. (As Winston Churchill said about the proliferation of battleships, it is sport to them, it is death to us.)

It's better that people should play with banks rather than battleships, of course - J.K. Galbraith said that one of the great things about a crash was that although it was a fine opportunity to observe all that was worst in human nature, nothing more important was being lost than money. (He may have been wrong - Chris Dillow observes that financial crises seem to destroy wealth more effectively and for longer than natural disasters.)

But I think the hilarious stories of Icesave, RBS, Hypo Alpe-Adria, that caja that was managed by the Church, and so on do tell us something about the value of formal independence, even limited formal independence like that of Scotland or Catalonia, in the era of "interdependence".

Essentially, I think, the one product that any degree of legal independence lets you produce is impunity. The legal status of independence is important here - without it, you're limited to hawking the bonds of the Serbian Republic of Northern Krajina to unusually dim marks, but with it, you can be of service to the world's plutocrats. Some features of independence that always sell include:


  1. Financial regulation

  2. Tax

  3. Shipping and aviation registries

  4. Corporate registries

  5. Criminal and civil jurisdiction

  6. An Internet top-level domain

  7. A direct dialling prefix

  8. Diplomatic privileges

  9. Passports



Criminal and civil jurisdiction, as impunity services go, have lost some value over the years as extradition treaties proliferate, legal norms are internationalised, and contracts come with arbitration clauses. Further, ever since the US Marshals hauled off Noriega, it's been at least conceivable that an extradition request may be delivered by 1,000lb air courier, in a vertical fashion and without warning. But facilitating tax evasion, the concealment of ownership, and the registration of ships and aircraft without taking responsibility for them are all highly valuable services.

Similarly, being able to register a horde of spammy websites is a good business to be in, especially if your own laws provide for genuinely bullet proof hosting. Unfortunately, many small island states are vulnerable to vigilante action by the Internet operations community as they only have direct access to one transit provider. Lending your direct dial prefix to anyone who wants to originate a mass of sales or propaganda phone calls is also a good business, especially if it involves callbacks and their attendant termination fees.

Finally, selling diplomatic privileges is the individualised version of the state's impunity. Holding an Angolan diplomatic passport kept Pierre Falcone out of jail for 10 years.

Arguably, turning the libertarian view of the state on its head, it is precisely the minimal state that is the closest to the status of one of John Robb's "gangs of black globalisation". Fascinatingly, some valuable criminal functions persist even in the absence of the canonical monopoly of force. It is probably no accident that the neoliberal era has coexisted with an unprecedented proliferation of ostensibly independent states.

We've not had a Thursday music link this week....



Because if anywhere made full use of the fraudulent possibilities of sovereign status, of course...

Admin: econoposts

Two new posts on Stable & Principled: why George Osborne is a strange kind of Keynesian, and a surprisingly radical response to the crisis from the LSE.

Prior reading: here.

News

The Zimbabwe Daily News is back after eight years of repression.

Wednesday, March 16, 2011

There is no crisis

OK, so as well as the big demo, currently standing at 578 coaches and counting, there's going to be a day of action on the NHS on the 1st of April - April Fool's day, surely someone can make something creative out of that. So I thought I'd open another flank. Flipchart Fairy Tales has already covered the issue as well as anyone will, but the point is quite simply that they're all lying to you about public pensions.

They're lying to you

That's a chart of the forecast cost of the entire public sector pensions system as a percentage of GDP, with a sensitivity analysis showing what happens to it if you change various parameters like life expectancy, productivity growth, and such. The coloured area each side of the line on the chart (the central projection, showing what the forecasters think is the most likely scenario) shows the range of possible outcomes if those assumptions are set to different values, to take into account the uncertainties involved.

But the most interesting feature of the chart is this: the peak cost is this year, 2011. This isn't subject to very much uncertainty at all - we know very well how much public pensions will pay out this year. So, why on earth is anyone talking about cutting public sector pensions, when their cost is going to fall every year from here on in as far as, ah, the financial year 2059/2060, by which time we'll either be hastily revising the budget to deal with the National Union of Robots or else scratching it on part of a ruined conference centre in Harrogate with a sharp stone as Chinese occupation troops armed with spears look on.

And it's not some unusually lefty wanktank that's come up with this through highly advanced ex ano analysis. This chart was dug out of horrible Blairite gargoyle John Hutton's urgent report on why public sector pensions have to be slashed yesterday, and it was prepared by the Government Actuaries' Department.

So, I'd like to find out exactly how many and which MPs have seen it, know what it means, and what they think as a result.

Here are some questions, with the answer in (brackets):

1) What do you think public sector pensions cost as a percentage of GDP, right now? (answer: 1.9%)
2) Is this rising or falling? (falling)
3) What do you think they will cost in 2020? (1.8%)
4) What do you think they will cost in 2050, in the Treasury's worst-case scenario? (1.5%)
5) In the light of this, do you support cuts to public sector pensions? (left as an exercise to your representatives)

And tell me about it. (Apologies are of course due to Josh Marshall of Talking Points Memo fame.)

Sunday, March 13, 2011

You are experiencing the flexibility in the rule in that they haven't already deleted you

Here's Yahoo!, heroically defending the image rights of individual Egyptian torturers. It looks like it's getting time to slurp all my stuff out of Y!, but beyond that, what a thread. The number of people queuing up to explain how compassionate and reasonable the boss is being. Ugh. Frankly, I think you might as well post the damn photos if you stormed the secret police headquarters.

So you may as well support the Piggipedia and write to them at this e-mail address: case982056@support.flickr.com

PS, after all the cock about Facebook revolutions, the US State Department's taking its sweet time to call off Yahoo!'s intellectual-property Brian Coats. You might think they were embarrassed about potential disclosures.

787: the plane Milton Friedman built, and about as well as you'd expect

This LA Times story about the Boeing 787 Dreamliner (so called because it's still a dream - let's get the last drop from that joke before it goes into service) and the role of outsourcing is fascinating. It is partly built on a paper by a senior Boeing engineer which makes among other things, this point:

Among the least profitable jobs in aircraft manufacturing, he pointed out, is final assembly — the job Boeing proposed to retain. But its subcontractors would benefit from free technical assistance from Boeing if they ran into problems, and would hang on to the highly profitable business of producing spare parts over the decades-long life of the aircraft. Their work would be almost risk-free, Hart-Smith observed, because if they ran into really insuperable problems they would simply be bought out by Boeing.


Even in its own financial terms, the whole thing didn't make sense, because the job of welding together the subassemblies and hooking up the wires doesn't account for much of the profit involved. Further, the supposedly high-margin intellectual-property element of the business - the research, development, and design of the plane - is only a profit centre after it's been built. Until they're done, it requires enormous amounts of investment to get right. The outsourcers were expecting the lowest-margin element of the company, assembly, to carry the costs of developing new products. Whether they were funded with equity or with debt, this implies that the systems integrator model, for aircraft at least, fundamentally restricts innovation.

This is one of the points I'd like to bring out here. Hart-Smith's paper - you can read it here - is much stronger on this than the LA Times was willing to be. It's a fascinating document in other ways, too. For a start, the depth of outsourcing Boeing tried to achieve with the 787 is incompatible with many of the best practices used in other industries. Because the technical interfaces invariably become organisational and economic ones, it's hard to guarantee that modules from company X will fit with the ones from Y, and if they don't, the adjustment mechanism is a lawsuit at the financial level, but at the technical level, it's rework. The dodgy superblock has to be re-worked to get it right, and this tends to land up with the manufacturer. Not only does this defeat the point of outsourcing in the first place, it obviates the huge importance of avoiding expensive rework.

Further, when anything goes wrong, the cost migrates remorselessly to the centre. The whole idea of systems integration and outsourcing is that the original manufacturer is just a collection of contracts, the only location where all the contracts overlap. Theoretically, as near to everything as possible has been defined contractually and outsourced, except for a final slice of the job that belongs to the original manufacturer. This represents, by definition, all the stuff that couldn't be identified clearly enough to write a contract for it, or that was thought too risky/too profitable (depends on which end you look at it) for anyone to take the contract on. If this was finance, rather than industry, it would be the equity tranche. One of the main reasons why you can't contract for something, of course, is that you don't know it's going to happen. So the integrator essentially ends up holding all the uncertainty, in so far as they can't push it off onto the customer or the taxpayer.

This also reminded me a little of Red Plenty - one of the problems is precisely that it's impossible to ensure that all the participants' constraints are mutually compatible. There are serious Pareto issues. There may be something like an economic law that implies that, given that there are some irreducible uncertainties in each contractual relationship, which can be likened to unallocated costs, they flow downhill towards the party with the least clearly defined role. You could call it Harrowell's U-Bend. (Of course, in the macroeconomy, the party with the least well defined role is government - who you gonna call?)

Anyway, Hart-Smith's piece deserves a place in the canon of what could be termed Sarcastic Economics.

I suspect that the problems he identifies have wider consequences in the economy. Given that it's always easier to produce more or less of a given good than it is to produce something different, the degree to which it's possible to reallocate capital has a big impact on how quickly it's possible to recover from a negative shock, and how bad the transition process is. I would go so far as to argue that it's most difficult to react to an economic shock by changing products, it's next most difficult to react by producing more (you could be at a local maximum and need to invest more capital, for example), and it's easiest to react by producing less, and that therefore there's a structural bias towards deflationary adjustment.

Hart-Smith's critique holds that the whole project of retaining product development, R&D, and commercial functions like sales in the company core, and contracting everything else out actually weakens precisely those functions. Rather than being able to develop new products quickly by calling on outside resources, the outside resources suck up the available capital needed to develop new products. And the U-bend effect drags the costs of inevitable friction towards them. Does this actually reduce the economy's ability to reallocate capital at the macrolevel? Does it strengthen the deflationary forces in capitalism?

Interestingly, there's also a presentation from Airbus knocking about which gives their views on the Dreamliner fiasco. Tellingly, they seem to think that it was Boeing's wish to deskill its workforce as far as possible that underlies a lot of it. Which is ironic, coming from an enormous aerospace company. There's also a fascinating diagram showing that no major assembly in the 787 touches one made by the same company or even the same Boeing division - exactly what current theories of the firm would predict, but then, if it worked we wouldn't be reading this.

Assembly work was found to be completed incorrectly only after assemblies reached the FAL. Root causes are: Oversight not adequate for the high level of outsourcing in assembly and integration, Qualification of low-wage, trained-on-the-job workers that had no previous aerospace experience


I wonder what the accident rate was like.

A question to the reader: 1) How would you apply this framework to the cost overruns on UK defence projects? 2) Does any of this remind you of rail privatisation?

status: it's complicated

This post brings several things to mind. Apparently, eastern Libya was a hugely overrepresented area among the international jihadis who went to Iraq and there exploded. Clearly, this means that you can't assume that they're fighting for democracy, whiskey, sexy.

However, it's also very likely that this represented a deliberate policy on the part of the Libyan government to channel its dissidents into particular ideologies that its new friends also perceived as the enemy, and then to ship them out of the country and hope they would explode somewhere else. Making jihadis - repressing all other forms of dissidence, while not trying too hard to stop them recruiting or leaving the country - had the side benefit that it validated their claim to be a bastion of stability assailed by Islamic extremism. They could produce the extremists, after all. And it further allowed them to avoid burning all their bridges with the other side. If it became expedient to make friends with the terrorists again, they could produce the bloody shirts - the martyrdom videos - and demonstrate that they had been useful.

Of course, Gadhafi didn't have to be an evil genius to come up with this plan - he was essentially copying Saudi Arabia's homework, and depending on how you look at the relationship between the Egyptian regime and the Brothers, perhaps sneaking a look at the neighbours' as well. Giddens may have thought they were going to be a new Norway, but the real plan was more like Saudi 2.0, probably right down to the hereditary government.

Another lesson from this is that they're probably not going to give up easily.

If these are the compromises, the power better be good

If you need anything to read this week, don't stick around here. Nick Davies for, as they say, the win. Follow the links - it gets better. There's much more detail here. This is a truly amazing story of crypto-politics, journalism, police corruption, and general depravity. Do you prefer the police/Murdoch spy who went to jail for planting cocaine on a woman involved in a divorce case before being accused of killing his business partner with an axe, or the bent copper who investigated the axe case and then took the victim's job, who was later convicted of being an Internet paedophile? Take your pick - as they may not have said to the victim...

And wouldn't you love to have been a fly on the wall when David Cameron and Nick Clegg were briefed about the whole rotten pile when they decided to hire Andy Coulson, who, it turns out, gave coke guy his job back after he got out of jail? The sad thing is that I can imagine Clegg lapping it up. If this is what Vince means when he talks about the compromises of power, I must be in line for real power...

But it's wider than that. There's something appallingly telling about the culture of the whole thing. Morgan's body was found with two packets of crisps in his hands. Crisps. The case eventually fell apart because enormous piles of police documents kept either disappearing, as happened in the original 1987 investigation after kiddy fiddler guy got his hands on them, or else re-appearing at inconvenient moments, very much like the contents of Glenn Mulcaire's desk. The Daily Mirror hired some of the same people to investigate the finances of the Bank of England's court of governors. Sleazy left-wing tabloid journalism is clearly not the same thing as the right-wing kind, although you wonder if anyone read the story that resulted. Which, I suppose, amounts to the same thing.

Someone should make a movie out of this, but they probably won't.

Specified technical incident

So OpenSUSE11.4 was out this week. As the Jedi said here:

gah! suse is never totally easy


Indeed. I thought I'd do an online upgrade, so I scheduled this to happen when I was in the office and therefore had a fast Internet link available. I applied all the remaining 11.3 updates, configured the three additional repos, did a "zypper ref" and then a "zypper dup", paged through the Flash player licence, and watched it report 500 odd MB of packages to grab. Much churning later, it started to miss packages, which I installed manually. Eventually, it finished, and I ran "zypper verify" to check it out. This reported that vim-data was missing, so I installed it, and went for a reboot.

Oh dear, the new distro apparently didn't know what an ext4 filesystem was. And although I could still start 11.3 from the boot menu, KDE wasn't working. So, back at home, I downloaded the ISO image (2 hours 20 odd minutes at home), burned a disc, and prepared for a clean install, which failed with a message about running out of processes in this runlevel. You guessed it, dodgy install media. Wiped and downloaded again. I check the MD5 hash. It's a miss. I start the download again and go out. I come back to find the laptop has rebooted and has got to the failure point in 11.4. How? What? I restart in Windows and discover that 678 of 695MB has been fetched before something happened. It dawns on me that Microsoft has force-rebooted the bastard through Windows Update although I set it to do nothing of the sort. I'm getting seriously pissed off now. I download it again, from a different mirror (ox.ac.uk rather than Kent Uni mirrorservice.org). More hours. I check the MD5 hash. What do you know, it's wrong. And it's the same hash as last time. As an experiment, I burn it anyway, boot it, and run the media check utility.

Which fails at 63%, block 226192, in exactly the same location as the first time around. Riight, it looks like Novell has pushed a crappy image out to all the damn mirrors. Well, I can still get a Linux shell in 11.3, so I run it up, hook an ethernet cable to the linksys box, run dhclient, and repeat the command-line distro upgrade. Although Zypper still thinks all the dependencies are in place, when I tell it to "zypper dup", it still manages to find 258 package changes left to do from the original upgrade. It takes an age, but eventually, completes, and it's shutdown -r now time. And everything now works, right down to hibernated browser tabs.

Except for Python packages, of course. Pythonistas tend to dote on easy_install, but I'm still annoyed that I have to update this stuff out of sync with my linux environment, especially as it lives in my root partition. Would it be so hard to put everything in PyPi into an RPM repository and never worry about it ever again? This is actually an important lesson about the mobile app stores, and the original app store itself, Firefox extensions. Freedom goes with structure.

Lessons from this: once an upgrade shows any signs of weirdness, abort it and start again. And don't expect online upgrade to work first time - this happened to me with a past OpenSUSE upgrade, come to think of it, but I clearly learned nothing.

Thursday, March 10, 2011

non-Thursday music link, on Thursday due to exceptional circumstances

John Band links to very likely the worst recording in the history of music, ever. Imagine that George Michael, many years after anyone cared, decided to do an impossibly portentous version of New Order True Faith '94, through a vocoder (many years after everyone got sick of those), and worst of all, to not play it for laughs. It's incredibly hideous, and it's not even remotely funny.

For relief, here's a special non-Thursday remix link post, appearing exceptionally on a Thursday due to the unprecedented circumstances.

Here's the Grim Up North remix of the original. Here's the Futuremix Discotech remix, also from the original EP:



Here's another, possibly better than the last one although this may be controversial. Here's yet another. And here's Interface's cover version.

Monday, March 07, 2011

get the look

Do I notice a common style between Nasr City Security Crisis Management Centre and Erich Mielke's office? Lots of blonde panelling, display walls of spook stars.

Of course, Egypt's secret police had both Soviet and East German advisors in the Nasser years, but you wouldn't have expected that to include an interior designer.

Sunday, March 06, 2011

Inverse BOGOF

Supermarkets are getting smaller. I wonder if the productivity numbers that famously put the 90s down to big box retail will ever be revised? Of course, they didn't really - the UK imposed planning restrictions on big out-of-town developments and didn't notice any trouble, and the real point was the logistics rather than the shops themselves. Still, interesting.

Self-binding note: lobby metrics

Things to get out of the data in this scraper of mine: for each lobby, the monthly meeting counts, degrees in the weighted multigraph, impact factor (i.e. graph degree/meetings to give an idea of productivity), most met ministers, most met departments, topics. For each ministry, meeting counts, most met lobbies, most discussed topics. For each PR agency (Who's Lobbying had or has a list of clients for some of them), the same metrics as for lobbies. Summary dashboard: top lobbies, top lobbyists, top topics, graph visualisation, top 10 rising and falling lobbies by impact.

Things I'd like to have but aren't sure how to implement: a metric of gatekeeper-ness for ministers, for example, how often a lobby met a more powerful minister after meeting this one, and its inverse, a metric of how many low-value meetings a minister had. I've already done some scripting for this, and NetworkX will happily produce most of the numbers, although the search for an ideal charting solution goes on. Generating the graph and subgraphs is computationally expensive, so I'm thinking of doing this when the data gets loaded up and storing the results, rather than doing the sums at runtime.

Where's that Django tutorial? Unfortunately it's 7.05 pm on Sunday and it's looking unlikely I'll do it this weekend...

Looking back on the Guardian's MannFAIL

Remember this post? Climate Progress has some answers as to how the Grauniad lost it so badly about the UEA e-mail hack.

And it's one jet airliner, for ten prisoners...

One thing that is perhaps being overlooked by people discussing whether or not it would be wise to impose a no-fly zone over Libya is exactly what such a zone would set out to prevent. When it was first suggested, it was inspired by the general horror that the Libyan government was having crowds of civilians strafed by its Sukhoi 22 close-support aircraft. However, especially since several Libyan Air Force crews defected to Malta and to the revolution, air activity has turned out to be much less significant in what is beginning to look like a classical West- or Central-African civil war, based around Toyota pickups and 23mm Russian anti-aircraft guns and mercenaries paid with the money from exporting some mineral or other. You know the one.

It's fairly well known that Libya sponsored several of the key warlords of 90s West Africa - Foday Sankoh, Charles Taylor, and several others originally met up in Libyan-funded training camps. Interestingly, not only did one of the versions of Jetline International base itself in Tripoli and trade aircraft back and forth with two of Viktor Bout's companies, but Gaddafi's government maintains an impressive airlift capacity. As well as the two flag-carrier airlines, Libyan Arab and Afriqiyah, whose names track the changing priorities of foreign policy, the Air Force operates a semi-commercial cargo wing, Libyan Arab Air Cargo, with a fleet of Ilyushin 76 and even two enormous Antonov-124s, some of very few such aircraft owned outside the former Soviet Union.

I've put together a Google spreadsheet of transport-type aircraft with Libyan operators, sorted so that currently active aircraft are at the top, and generated URIs to look them up on Aerotransport.org, for subscribers, and on JetPhotos.net, in the two right hand columns.



There are a total of 180 airframes, of which 118 are active. It's probably worth noting that there was a report that top managers at Afriqiyah had resigned rather than take part in Gaddafi's war effort, and constant rumours of mercenaries being lifted into airfields in the southern deserts.

The upshot of this is that logistics, rather than tactical air power, might be the most important factor in Gaddafi's efforts to defeat the Libyan revolution/win the Libyan civil war. Rather than engaging in combat, the aim might instead be blockade, as a complement to the international financial sanctions already in place. (A ship has recently been stopped in British waters carrying large quantities of freshly printed Libyan currency.)

On the other hand, it also adds complexity and risk to the whole issue. There are still plenty of people who want to leave Libya, and British government-chartered airliners are ferrying some of them from Tunisia to Egypt. It would be a bad business, to say the least, to shoot down an Il-76 full of refugees. It could be better to try to cut off the supply chain at source by grounding Libyan aircraft elsewhere in the world, although this requires the cooperation of those states who are still willing to let them recruit on their territory. Further, imposing a blockade also implies a responsibility for the survival of the civilian population. Sending aid to eastern Libya has already been suggested, of course.

For a little extra, the Russian Demography blog, venturing well out of its usual beat, notes that the Libyan Government's Dassault Falcon 900EX business jet, 5A-DCN, took a trip to Minsk recently. Its ICAO identifier, useful with virtual-radar sites, is 018019. There are various things the regime might find useful in Belarus - mercenaries, again, small arms (although they don't appear to be short of them), and perhaps least disturbingly, impunity. (Hat tip.)

Thursday, March 03, 2011

fake tales of San Francisco echo through the air

It is now reasonably certain that I'll be in Palo Alto from Saturday evening to Thursday evening. I'm going to be carless but close to a CalTrain station, if that helps anyone. The explanation is of course here.

Update: I will be here from 6pm tonight with some bloggers.

Sunday, February 27, 2011

Tab-clearing link fart

Mike Konczal and the snark:
We are perpetually in a spiritual crisis that can only be fixed by a woman in the kitchen. It’s a huge win for the second half of the 20th century that this argument has been moved from leading political figures addressing graduating classes to the shadier alleyways of the Wall Street Journal opinion page.


Robert Fisk fisks Gaddafi.

Brilliant, awe-inspiring Borderland Beat piece on the rise of the Sinaloa drug cartel, based on anthropology, history, economics, and Mexican/US counterinsurgency strategies as far back as the early 70s.

Where do these wanktanks keep coming from, education edition?

The lobbyist for brothels.

Python Module of the Week as a book.

Mulcaire must answer.

Oyster cards - they work perfectly simply inside the sensible public-service world of Transport for London, but everything changes when they cross the line into the twisted libertarian fantasy of the privatised rail network. Funny that.

Trying to fix the mortgage fiasco.

Dissolving a corpse in sulphuric acid - it's harder than you think.

Who's Lobbying now knows which lobbies use which lobbyists, although there's still no API.

A case study

This Realclimate thread provides some interesting insight into how Nigel Lawson and sports scientist Benny Peiser's wanktank, the Global Warming Policy Foundation, the dodgy journal Energy & Environment, and the little gang of rightwing nutters at the University of Buckingham are being progressively integrated into a single meta-wanktank. They're all sitting on each other's boards - presumably, someone at Bucks writes a paper, their mates at E&E Googleberg it through the review committee, and then Peiser and his dog push it out to the frothing churnalists over the wires.

Enduring stare: Donal Blaney edition

Here's something interesting, from a post at Owen Jones's. Has Donal Blaney's "Young Britons' Foundation", the Tory wanktank that was too wanky for the Tories, been reactivated?

I will not distance myself from the right wing spin machine which attempts to bring down NUS’ credibility, but tackle it head on. Both of those occasions have taken NUS’ communications outrageously out of context and have been blatant attempts to weaken NUS. I also think these kinds of attacks will appear in the same kinds of newspapers consistently over the coming months in the run up to a report which is being published soon by the right wing “Young Britons Foundation” which will call NUS and Students’ Unions ‘dangerous organisations’ and attempt (again) to discredit and shut us down.


Over at the YBF website, it looks increasingly like they're being absorbed into the whole ex-Decent industry. Speakers from the Henry Jackson Society, projects on "Students' Rights" and "radicalisation". I suspect part of the reason for this is that the Department for Communities & Local Government budget is relatively big compared to your average wanktank and therefore wanktanks are irresistibly drawn towards it, by a sort of financial gravity. I presume the planned "report" is going to be an effort to accuse anti-cuts protestors of secretly being jihadis, perhaps with a side effort to get on the DCLG counter-radicalisation gravy train.

Also, here's the rundown for their next event:

Confirmed speakers:

Andrew Stephenson MP, Vice-Chairman of the Conservative Party

Steve Baker MP,Wycombe

Jonathan Isaby, ConservativeHome

Robert Halfon MP, Harlow

Shane Greer, Executive Editor, Total Politics

Eamonn Butler, Adam Smith Institute

Iain Dale, Broadcaster & Publisher

Dr Alan Mendoza, Executive Director, Henry Jackson Society

Raheem Kassam, Henry Jackson Society

Douglas Murray, Center for Social Cohesion

Simon Richards, The Freedom Association

Michael Fallon MP, Sevenoaks

Matthew Sinclair, Director TaxPayers‘ Alliance

Sajid Javid MP, Bromsgrove

Daniel Hamilton, Big Brother Watch

Lord Flight

Nigel Evans MP, Deputy Speaker

Rt Hon Peter Lilley MP, Hitchin & Harpenden

James Delingpole, Polemicist

Chris Kelly MP, Dudley South

Tim Evans, Director, Libertarian Alliance

Andrew Rosindell MP, Romford

Douglas Carswell MP, Clacton

Rt Hon Theresa Villiers MP, Minister for Transport

Mark Wallace, Senior Fellow TaxPayers’ Alliance

Conor Burns MP, Bournemouth West


Now there's a party even I wouldn't try to get into. What a bunch. For an organisation that was publicly disavowed as barking extremists during the last election campaign by the current prime minister, it's not a bad turn-out. There's a fair amount of filler, but even so. It also looks like I was right about "Big Brother Watch", the anti-ID card campaign that suddenly appeared five minutes before the repeal bill got Royal assent to claim the credit for six years of NO2ID - it's yet another Tory wanktank.

I was also amused by this:
Raheem has served on the National Executive for Conservative Future and began his political life running the blog ‘KeepRightOnline’, which swiftly became one of the best read political blogs in the country (now closed).


Yeah. One of the best read blogs in the country. Now closed. (Alexa can't find any scoreboard data on it at all.) It's apparently become a twitter feed.
Hey fat chicks; colouring and straightening your hair doesn't make you attractive. It makes you a fat chick with straight, coloured hair.


That's "quintessentially British commentary", apparently. It actually sounds more like a poor effort at pretending to be one of the crappier US trolls - Jeff Goldstein or Adam Yoshida - but then, crappy fake American trolling is Blaney's MO down to the ground. I wouldn't be at all surprised if they try to pull some sort of cod James O'Keefe trick in the run up to the 26th March demo.

a patron, Sir?

Sensible piece about US State Department funding for mobile anonymity projects, and some interesting stuff. The crack about looking with disfavour on the drowning man and then encumbering him with help once he reaches ground is relevant.

The real prize (as alluded to here) would be a mesh network application that works either instead of the PLMN or alongside it. The only way to avoid leaving traces in the enormous billing/rating/charging infrastructure of your average cellular network is not to use it. According to Comptel, the Finnish OSS/BSS software house, operators spend about €32bn a year on software, of which €11.5bn is in the revenue management segment, another €5.5bn in business analytics, and another €4bn in CRM - €21bn worth of data-mangling kit that could theoretically be repurposed. It's probably better to just leave a GPRS datacall than a phone call to the person you want to speak to in there, though.

On the other hand, there's an API for the US Army.

So what are the other side using?

OK, so I've been getting referrals from The Social Media Library, which describes itself as follows:
Social Media Library contains the first ever regionally-focused directory of influential blogs read in the UK. Social Media Library’s proprietary BlogScore™ ranks each blog in terms of its influence and its visibility in search engines - providing you with all the necessary information to determine which bloggers to engage with online. Full contact details for each blog are also provided.

Tens of millions of blogs exist online, with only a relatively small percentage holding any influence. Social Media Library allows you to cut through the noise and quickly find the online influencers that carry most sway. Social Media Library’s research team uses a proprietary ‘social media spider’ to scan the Web for the most influential blogs, across all major topics of interest. Each blog is manually checked before being added to the directory. Only the most influential and well-read blogs are included.


BlogScore! Not only is it a trademark, it's InterCapped as well - must be good. And then I started seeing referrers from CisionPoint, which seems to be some sort of cloud-based PR application. A DMS - Drivel-Management System - perhaps. Here's their blurb:
CisionPoint brings together - in one integrated customised dashboard - the on-demand tools you need to create, execute and evaluate superior campaigns from start to finish, building on the world’s largest media databases, Mediadisk and PR Planner.

Log in to plan your campaign, connect with the media directly, monitor news coverage and analyze campaign results


More here. I can just see Saif Gaddafi peering at his troll-penetration Gantts. Looking at the full URI: http://cisionpoint.com/Login.aspx?sessiontimedout=true&ReturnUrl=%2fNewsItemDetail.aspx%3fid%3d1678111250%26RowId%3d1678111250%26WorkspaceId%3d194354550

I would think that whoever's using it is trying to monitor either who's commented on some news item, or else who's commented on a blog post that's been scraped into their database. We seem to have a unique identifier for the news item - which is also used in a database somewhere as a row primary key - and another for the project or user (the WorkspaceId field). A quick check shows that this particular system is being used by InterPublic Group in New York City. Cision is UK based and I presume the data centre is too, but the browser referral puts it on an AS174(Cogent) block whose referral whois identifies it as a /29 route assigned to them.

The Observer's comment page has been overrun by zombies

...And I feel fine. These two columns appeared on the same page of today's Obscurer. I believe this is probably the most incoherent single page to ever appear in a British national newspaper.

For a start, there's Nick Cohen with a piece that accuses essentially everybody of being obsessed with Israel, because of the Protocols of the Elders of Zion, and then goes on to say that Israel's supporters are as bad for the same reasons. Further, it's because of this obsession that the media failed to report on the Tunisian revolution. And Peter Mandelson is bad. The impression I got was that if zombies wrote opinion editorial, this is what it would be like. Staggering, blundering...are they actually conscious? Sapient? Surely not. But they clearly have a restricted survival kit of dim perceptions and knee-jerk responses. They stagger about, crave brains, flock together with other zombies.

And that's what we get here. This isn't a document of the Decent movement - it's just a salad of disconnected stylistic tropes. Teh media. Protocols. One of either Auden or Orwell, from the sticky pages in the quotations dictionary. New Labour is evil (but not Tony Blair). What it reminded me of was a US right-wing comments thread around Obama's inauguration - a sense of absolute disorientation and futility, relieved by going through the reflex drill movements. And, well, the revolutions have surely rendered the Decent project as irrelevant as they have Al-Qa'ida. Cohen's hackery is best described by the fact he wrote a piece for a British newspaper about Libya that doesn't mention either Tony Blair or that there is oil there.

On the same page, we have a truly odd piece of work from Catherine Bennett that purports to say...something...about age restrictions, but seems to have been accidentally mixed up with another decrying the sexualisation of the young, yadda yadda, reality TV, the screen generation, you know, that stuff. As far as I can make out she's against age restrictions but only because our culture is so repellent that the kids are past saving, or perhaps she's putting it on. She also imagines that "tots" learn to swear from listening to Today on BBC Radio 4, which if true might explain a lot, and that your children might be watching porn on "iPlayer". Really? The BBC's Internet streaming service?

Also, there's a lot of tiresome in-joke stuff about various important people in the theatrical world. I'm not posh enough to know whether most of it is meant as a joke or not, and if so, what's meant to be funny about it. And there are paragraphs like this:

If the director-general of the BBC could be made to grovel for a baby-swap plotline borrowed from the Old Testament, there would be no difficulty in embarrassing a publicly subsidised production such as Frankenstein which, as well as naked adults, also features a stylised rape that may be slightly more graphic than the Forsyte grapple watched by 8 million people, although notably less disturbing and self-indulgent than the sexual imagery and violence that, in the absence of any comprehensible dialogue, captivated so many of the family audiences of Tim Supple's celebrated Midsummer Night's Dream.


You may find it helps to read this aloud. Pro tip: when you get to a comma, take a breath. That's some 93 words, a whole generous editorial page par, in one enormous run-on sentence. It is terrible stuff. It is absolutely no fun to read. It took me three runs through to work out what on earth the point was meant to be, and I'm still a little fuzzy. Does she mean that it would be easy to embarrass the makers of Frankenstein because of etc etc, or something about the BBC, or something about someone's production of A Midsummer Night's Dream?

I honestly don't know how I'd start trying to edit this into sense. I'd be tempted to spike the whole thing and suggest that Bennett comes back when she's decided what the column is going to be about. Or perhaps hand the half-page over to the cartoonists.

Actually, this idea is surprisingly tempting. Why not put the cartoonists in charge? Why not draw the entire opinion section? It would be reliably more informative than Martin Kettle or Andrew Rawnsley, more offensive and in a better way than Cohen, and better written than Bennett.

Fun with nonsense and hash functions

Churnalism is a brilliant idea - no surprise that it was originally one of Chris Lightfoot's. Basically, it allows you to determine how much of a given newspaper article was copied from which press release. There's a nice graphic visualisation, and a diff, so you can see precisely what was altered and what taken over in its entirety. It's right up there with Piggipedia and SukeyDating as a brilliant piece of geek activism.

However, here's something amusing. There's a basic API here; I chucked the text of the GSMA final press release from this year's MWC at it, and I was quite surprised at the results. The first article it extracted from Journalisted was none other than this piece in the Guardian from...February 2008. One consequence of churnalism is that your newspaper is likely to get repetitive.

As far as I can see, if there's anything missing here it's that the comparison is mostly the wrong way - having a newspaper article and wanting to know what vacuous NIB-fodder got regurgitated into it is a much more common use-case than having a press release and wanting to know which newspaper articles it got into. Actually, the latter use-case is far more likely if you're a PR and you're trying to measure how the talking-points are spreading. But once it has more press releases on file, it'll work better in that sense. And that's just a question of hoovering Businesswire, PRNewsWire etc up.

Tuesday, February 22, 2011

the CIA decided they wanted Roland dead, so that son of a bitch Van Owen cut off Roland's head

Sultan al-Qassemi kicks in a data point to the ArseDex. Apparently Libyan agents are distributing flyers in Guinea and Nigeria calling for mercenaries to fight for $2,000 a day. Yesterday, loyalist thugs cost $500 a day in Libya. Even with the huge supply of potential thugs in sub-Saharan Africa's demobilised militias being available, the ArseDex has gone non-linear - it's risen by a factor of four in 24 hours. Arseholes now command a premium of four hundred times the average wage. Surely Gadhafi must be doomed now.

The data's pretty sparse, but here's a spreadsheet. The edit link is here.

Sunday, February 20, 2011

from the MWC gossip column

You can consider MWC as being a giant augmented-reality computer game. You run frantically in and out of different constructed environments, very varied but actually deeply conventionalised, trying to score points. You get points for collecting gossip, useful information, sales leads, and shiny gadgets; you lose them for queuing, being publicly humiliated, or getting stuck in a vacuous keynote session you have to listen to because the guy burbling away is too important to walk out on. If you collect too many shinies in a day you may be robbed. You have various resources to manage - you can't go too long without checking into a WLAN hotspot, and at some point you must eat and sleep or at least drink more coffee. Progress is measured by winning a higher-status badge for next time.

The ruling emotion is, as always, that horrible sensation that it's all happening somewhere else. Something really fascinating is being said, in the next conference session or the other party. If you were somewhere else, you'd meet the bloke who can sign that interconnect agreement or uncork this or that barrel of money. So-and-so was at the Dilbertco stand and they were giving away Dilbertphones! Of course, this emotion is a lie. Lester Bangs nailed it in a piece about living in New York and being tormented by the feeling that everyone else must at that moment be doing something more interesting and cooler than he was. Shouldn't he be out there, getting on with it? But really, he came to understand, everyone else he knew was feeling exactly the same thing.

I grew up in the Yorkshire Dales, so for years it was literally true for me that everything was indeed happening somewhere else, and I've never been able to resist it. Which was probably why I went to the awards after-party - I was sort-of invited, which can be the same thing as not being invited at all or even better than being invited, and it's technique that makes the difference. Having changed out of my suit and donned something with horizontal stripes - literally every time you saw a horizontal stripe at MWC it was wrapped around a software developer, it was quite uncanny - I appeared comfortably after kick-off and climbed up the hill to the Palau Nacional by a back route, smiled politely at the outer layer of goons, asked one of them directions I didn't need, passed through the doors and made for the rope.

I wasn't on the list, which I knew, but I was able to muddy the waters sufficiently that they went off to talk to some authority-figure within. I considered a dart up the unguarded stairs but felt confident enough to leave it. And I was in - the usual scene, x hundred immobile suits, a depressed-looking DJ, and about three people dancing, a group of American short film-makers IIRC. Eventually we managed to get the gig going to the extent that others broke off the herd, including an expert dancer wearing (of all things, it's not usually correlated) a GSMA gold speaker pass* who was eventually hauled to the ground by a drunken, boorish Colombian (I think - the identification is hearsay). The official photographer broke out of his Douglas Adams alert crouch to document the mess - God knows why, but it reminded me of the exhibition in The Kindness of Women ("How do you feel?")

After he finally took the hint, I found a pair of sunglasses on the floor and handed them to his victim - for want of anyone else to hand them to, but she immediately put them on, before asking if they were mine. She apparently thought this had been some sort of dramatic gesture, but actually I just didn't want anything smashed underfoot. I think they belonged to the Colombian. Not long after that, as they say, I made an excuse and left.

*not quite as exclusive as all that, I've had one in the past

i haz bin in yr AR standardz, facilitatin yr interop. kthx!

So I had the opportunity to take part in an Augmented Reality standardisation meeting on the fringe of this year's 3GSM Mobile World Congress. First of all, it was the year the heavens opened (someone on twitter said it was as if the show had turned into Glastonbury) and I got drenched and my shoes went bad, and my cab didn't take me to the Telefonica R&D building in Via Augusta but instead to the main switching centre, this amazingly domineering building...

2011-02-17 13.07.09 Telcos - they live in places like this, they know where your dog goes to school, but can they tell you if it's really your bank on the line?

So I got soaked again, and eventually arrived, and spent the first session listening to my shoes rotting. I acted as scribe for the session on AR browser implementations, markup language vs. JSON, native application vs. browser plugin and the like. I hope I contributed something of value. I have a Flickr set of the annotated flip charts here; I've been asked to help prepare the final report. Which just goes to show the enduring truth that if you want to influence something, wait until the very end and sum up with a balanced account. Supposedly this used to be the way to pass the Diplomatic Service exams - buy a pipe, puff on it occasionally during the team exercise, then "sum up with a balanced account". But you're not allowed to smoke these days.

2011-02-17 19.16.26

A Little Bit of Egyptian Internet Twaddle

Pulling together various resources, I'm beginning to get a picture of what happened with the cut-off and restoration of the Internet in Egypt. First up, at least in some senses, it may be valid to say that the Internet played a role - Arbor Networks observed that traffic to and from Egyptian networks (and between them, in so far as any of them are customers of Arbor's) had spiked dramatically, almost vertically, in the two hours before the cut-off and that the whole week up to the 28th of January had been one of unusually heavy traffic.

When the cut-off went into effect, at 5.20pm local time on the 27th, it was implemented by forcing all the networks that peer at the Telecom Egypt-controlled Internet exchange to drop their BGP peering sessions with the exception of AS20928, Noor Data Networks. Famously, this is the operator that serves the Central Bank and its payments settlement system. Essentially immediately, 2,576 networks announced by 26 Autonomous Systems became unreachable. The surviving 26 ASNs including, as well as the Central Bank, the Alexandria Library, and the national research & education network, which if it is at all like most NRENs has a lot of its own infrastructure.

On the 31st of January, there was a further wave of cut-offs which removed another 14 ASNs and 134 networks. The list of the last survivors is here - notably, someone had clearly realised that not cutting off the students, of all people, was a missed opportunity, as the NREN isn't in there. However, one of the mobile operators (UAE incumbent Etisalat's national opco) stayed online although they had been ordered to cut off the mobile service itself. Perhaps they provide service to the government's mobile devices?

Interestingly, however, according to posts to NANOG, several of the .eg root DNS servers remained online (not surprisingly, as at least one is outside Egypt). Even more interestingly, even after the BGP sessions with the IX were pulled down, the lower layer equipment stayed active - Egyptian ISPs noticed that there was still link light on the fibre optic lines between their locations, and theoretically it would have been possible to cobble together static routing between their systems.

Similarly, the internal voice network remained operational and so did the international SS7 gateways that link it to other phone systems. As a result, some people found that they could still reach their ISP, whether by dial-up over the voice circuit or even sometimes on DSL. The question, though, was whether there were any routes beyond the ISP's nearest point of presence. Several foreign ISPs offered free dial-up connectivity over international phone service (notably this French one).

And, it seems, Egyptian ISPs also tried to re-establish internal connectivity after the cut-off, when they noticed that the fibres were still lit up. However, the problem was more subtle than just pointing static routes at each other. Communicating with people outside Egypt wasn't, after all, the primary need, and anyway, it required passing through the government-controlled exchange.

But the problem with Facebook, Twitter, Gmail or what have you is that unless they have data centres in your country, they're international traffic. Depending on their internal architecture, even if they do, they might be dependent on international routes. An Egyptian engineer who posted to NANOG during the revolution made the interesting point that, although Egyptian ISPs are relatively well-interconnected among themselves, not that much traffic flows over the interdomain links as so much stuff goes out to the global Internet. It's analogous to the old problem that the topological centre of the African Internet was 36 Tooley Street, London SE1 (the LINX headquarters), or 111 8th Avenue, New York, depending on whose version of the story you like better, although less pernicious as the infrastructure is there to solve it.

Sometimes this is useful - it's harder to censor stuff hosted in another jurisdiction. But it's also a problematic dependency. Back in the Egyptian NOC the New York Times was hosted on, they were struggling to find copies of key software packages to distribute, for example clients for Internet Relay Chat messaging, and also critical data files such as cached DNS zones, lists of domain names and their corresponding addresses. Some ISP engineers are now working on preparing emergency packages of software and data for use in an extreme emergency - for example, regular dumps of the root and local DNS zones, similar snapshots of the local routing table, not to mention PGP signing keys and contacts for as many other engineers as possible.

After all this, what were the government's aims? The initial cut-off was probably motivated by a combination of wanting to black out sources of independent information and hoping that it would hinder the protestors' organising. Some of its particular details - for example, leaving 20928 up and not trying to shut down interdomain links within Egypt - may have been an effort to keep some "normal service" going, as well as not preventing VIPs from transferring their money out of the country. It's also possible that cutting off link light between all Egyptian ISPs without physically grubbing up the fibres was harder than it looked.

So then, why did they bring it back on the Tuesday of camels and thugs? One interpretation is that they were hoping people would go home and update their Facebook statuses, which would have been incredibly patronising. But the Egyptian elite patronised the hell out of the public every time it went on TV, so it can't be ruled out. Another one is that they hoped to project an impression of returning normality, which didn't really fit with thugs on horseback swinging knives, but then their response wasn't characterised by coherence.

Another still is that they hoped it would help to get the government's propaganda out there. This argument - Gamal Mubarak flipping through his copy of The Net Delusion in a curtained backroom of the palace - has the advantage that when the Internet and the mobile networks were reactivated, there was a rash of reports of loyalist trolls, and one of the first things that happened was that the government forced the mobile operators to send out threatening bulk SMS messages - spam as a weapon. But this was surely incredibly optimistic.

In fact, what did happen was that people started doing precisely what they had only been doing to a limited extent the week before. Twitter feeds from Egypt filled up with what the NANOG crew would term operational content - requests for more medical supplies, reports of a lost child, calls for more protestors to mass at a specific gate into Tahrir Square. This was the real thing - a tactical radio network for the mob - and ironically it was mostly running over SMS and going out to servers elsewhere in the world. And, of course, its major carrier was the much reviled Vodafone Egypt, unwilling deliverer of Central Security's spam blitz.

Market forces live: ArseDex

Thanks to reader Koranteng for this data point. You may recall this post about the market for thugs in Egyptian politics. Specifically, when the government needed arseholes to attack the protestors, it had to pay four times per capita GDP to get them.

In Libya this week, it is said that the government is using mercenaries recruited from its various allies' wars in sub-Saharan Africa as arseholes, and that it's paying $500 a day for their services. Libyan per capita GDP is $14,884 at purchasing-power parity, so the price of privatised violence is running at a premium of over one hundred times typical earnings. Clearly, either the regime has so much less real legitimacy, or the degree of brutality required and risk involved is that much higher. In fact, those options are both consistent, as a regime with less legitimacy would need to use more force and it does seem to be doing just that.

I made the point last time out that it's typical for mercenaries to be very highly paid relative to the countries in which they operate. This is clearly an important point here. It's also true that Gadhafi's Libya has often got other people to fight its battles for it - they exported Palestinians into a variety of different wars in the 1970s and 80s, notably sending PLO volunteers to prop up Idi Amin (you bet they didn't sign on for that). Later, in the 1990s, they trained and equipped fighters in the various West African civil wars (notably Charles Taylor - there's an arsehole for you). Now they're doing the opposite.

Of course, being an oil state, they can probably afford to keep hiring the arseholes.

However, here's something interesting from the Egyptian elections last year, from Reuters.
Rates for hiring a thug start at 800 Egyptian pounds ($140) and can reach 40,000 pounds depending on the assignment, according to a study printed by the independent Wafd newspaper.The study, by criminologist Refaat Abdel Hamid, said thugs hired to attack large groups or candidates cost 25,000 pounds a day. Those hired to resist the authorities cost 6,000.

"The price of thugs includes compensation for custody and hospitalisation," the study said. "Former and current ministers and the NDP party get special prices and discounts. Prices are hiked for businessmen and first-time candidates."


That suggests that in October 2010, your entry-level goon came in at about twice the rate Mubarak was paying at the height of the revolution. Interestingly, if you were looking for goons who would be willing to assault a crowd of rivals - the same mission the camel riders had - you'd have had to pay much, much more. Thirty times more, or perhaps there's a zero missing somewhere, in which case it would imply an even bigger price drop. Part of the difference might be explained by the NDP claiming mates' rates as a large customer of long standing, and one who could offer valuable side payments in the event of success.

But it's hard to think of any explanation why the NDP would have been paying less for thugs at the height of the revolution, when they would presumably have been in demand, and the party itself would have been desperate. Also, assuming the selling party could read the writing on the wall, they would surely have been likely to insist on payment in cash on the nail, rather than promises of future side-deals that would likely never be fulfilled. Perhaps the supply of potential thugs increased, but how? Was violence just a more salient possibility?

Or perhaps there was a radical shift in the supply curve between October and January. If the usual sources of goons were for some reason unavailable, and the recruiters were fishing in other ponds, it might be quite possible that wages would be dramatically lower and that the thugs would be much less effective. Of course, another way of saying that there was a radical shift in the supply curve for state violence is to say that there was a revolution.

Sunday, February 13, 2011

Admin: MWC

In other, tangential, mobile industry news, I was amused by this:
Apparently, if you drive around Alaska with a spectrum analyzer on the front seat of your car, nobody stops you. This is awesome.


It's awesome, and it's also true of huge nuclear reactor complexes near Bratislava, where I once did just that back in 2005. We were there to cover the switch-on of T-Mobile's new mobile-broadband network, which was using an OFDM radio technology developed by a British company called Flarion, from somewhere in Oxfordshire I think, that later got bought out by Qualcomm. And so we all went out for a drive test in a gang of vans with tinted windows and James Bond-heavy looking drivers, with laptops and funny-looking antennas and spectrum analyzers and nuclear bomb-looking coverage maps. And nobody batted an eyelid. (I think it's now safe to say that I borrowed my partner's laptop, the property of Royal Holloway, University of London, which organisation never knew that it had been used for drive-testing experimental mobile networks in Slovakia...)

It was a bit different, it turned out, in Washington DC, where they had installed a radio network for the police force. There, if you drove around with a Rohde & Schwarz analyzer (about ten grand to you, sir) on your front seat, eventually the black Chevy Suburban behind you would be joined by four more converging out of the side streets and you'd end up in a Patriot Act dungeon somewhere for eighteen hours until the head of emergency services comms found out where they were keeping you and succeeded in vouching for your non-terrorist status.

Thursday, February 10, 2011

music

Non-Thursday music link. But on Thursday! Richard X vs. New Order "Jetstream".

Tuesday, February 08, 2011

lawyer

Meanwhile, a scoop from Robert Fisk. This is amusing; if some sort of art-terrorist group had wanted to mock US policy, could they have done better than appointing as special envoy a man whose father was the CIA's head of planning and who is actually Mubarak's lawyer? The State Department should probably review personnel policy. It keeps them out of trouble, after all.

Cash rules everything around me (but perhaps less than you might think)

Daniel Davies's post about arseholes, and more formally about the importance of the reactionary mob as an institution, has been a well deserved hit. Here's something interesting, though. Fairly serious rumours reckoned that the arseholes were being paid as much as $68 a day. In theory, if an arsehole was on duty 340 days a year, they'd make $23,120 a year (presumably cash in hand, too). Egypt's per capita GDP for 2010 was $6,200.

To put it another way, when the state needed thugs, it had to pay four times the per capita average income. Of course, it's possible that these numbers are seriously in error. But the principle isn't obviously false - mercenaries are usually paid a much higher spread over the typical income of the country where they operate, an implicit recognition of the fact the people want nothing to do with them or those who hire them.

In more advanced markets for thuggery, though, it's typical to hire someone for a specific act of violence, at rates considerably lower than per capita GDP. What does this tell us?

detail

So what happened in Tunisia? It's probably worth pointing out that they've signed a gaggle of UN human rights conventions, dissolved the old ruling party, and are having a strike wave. Having done the broad strokes of the revolution, they're now working on the detail.

Monday, February 07, 2011

From the noisy phase to the quiet phase

Is it meaningful to say that the Egyptian revolution is calming down, or petering out? I ask because a common flaw of the reporting on it has been to treat the basic dynamics of mobilisation as if they were signs of huge political shifts behind the curtain. It's obviously true that both revolutionaries and reactionaries need to sleep and eat. When the revolutionaries want to, they have no great difficulty in putting over a million people on the streets in Cairo and probably a bit more again elsewhere in Egypt. These are peak efforts. Idiot management-speakers like to talk about maintaining peak performance, but they are idiots: the word peak implies a supreme effort that cannot be maintained continuously. People have to eat and sleep, they have families, they have jobs, although many millions of Egyptians have been taking part in the revolution silently by essentially going on strike. Even revolutionaries have to maintain their barricades, update their blogs, and hold meetings to decide what to do next.

The result of this is that there's been a sort of media cycle - one day the papers are full of pictures from the latest day of rage, the next it's all about people grandly speculating on what happens next, and the regime's spokesmen explaining how they intend to preserve the substance of the regime. Perhaps they talk about that on the other days, but nobody is listening. Or perhaps they believe it, when they wake up and hear that there are only tens of thousands of rebels in Tahrir Square rather than hundreds of thousands. Then, the next callout of the demonstrators resets the clock again.

Today, we seem to be in one of the ebb-tide phases. So it's a good moment for a bit of speculating. What is important, in these terms, is that the government doesn't seem to be regaining much ground in between waves of protest. Instead, there seems to be a ratchet in operation - each wave extracts a new concession. Mubarak sacked his government. And appointed a vice president. Then he promised not to stand again. Then talks were opened with the opposition. Then the military accepted to talk directly with the opposition, independently. Then the NDP hierarchy was purged. Then Suleiman renounced becoming president himself. And the regime's own peak effort - Wednesday's thug raid - was dramatic and violent at the time, but with hindsight was nowhere near enough in terms of numbers to change anything. Arguably, it wrecked the government's remaining legitimacy and only demonstrated its lack of mass support.

The fear is that this is no ratchet, but a sort of retreat into the Russian hinterland, a trap. On the other hand, it's a common pattern in the end of dictatorship, a sort of political Cheyne-Stokes breathing. You may think you are saving the structural realities of power and giving away the forms, but how will those realities stand up without the Emergency Law and the special constitutional amendments and the practice of having political prisoners and the ban on opposition parties and the censorship of the press? After all, there must be a reason, rooted in the structural realities of power, why you wanted them in the first place. If owning hotels was enough to sustain a tyranny, there'd be no need for Central Security or private thugs on camels or sententious TV broadcasts or bulk SMS messages with faked originating numbers.

Revolutions come with years, like New Order remixes used to. Prague '89. Paris '68. Probably the most relevant ones now are the Polish ones - Solidarity feat. Jaruzelski '81 and '89. The first one was a lot like what everyone fears for Egypt and also quite a lot like the official preferences of our governments. There was violence, but not as much as there could have been, and a safe military dictator won. He, in turn, turned to a religious and conservative pseudo-opposition to give his rule some foundation. The second was more optimistic but less spectacular. In 1989, the end of communism in Poland involved far more negotiating than it did street-fighting, and it involved putting up with Jaruzelski sticking around for the rest of his term as a sop to the powers that be, or rather the powers that were.

Egypt is already some way beyond 1981 - there is something like a round table, and the officially designated military strongman is getting very close to the exit, having disclaimed supreme power for himself. Probably the communists of 1989 thought they were cunningly playing for time. Suleiman has a far more ruthless reputation, though; the big issue is whether he can be trusted or better, constrained from trying to either crush the opposition between here and whenever the election date is set or else to start a civil war like the Algerian generals of 1991.

One argument has been that there would be a fake revolution, leaving the security state in charge, as Jamie Kenny put it. I think this is now out of date. Similarly, although they are now talking to the Muslim Brotherhood, I think my own prediction is also out of date. We're past the point where a few Brothers in the government would convince anyone. In fact, Jamie and I saw our predictions first validated and then rendered irrelevant within a week.

Looking ahead, it's worth remembering that 1989 took time to deliver. After the original moment of success, there was a long and uncertain haul of getting rid of specific individual bastards, changing laws, moving editors around the State TV and inspectors around the police force. I think we're now into this phase. Some people seem to agree, from very different points on the spectrum. Changing the union confederation and the university professors' club is very much to the point, whether you're thinking 1989 and maintaining enough forward momentum to protect the revolution or 1917 and the second wave.

Take it easy ya Ahmad. Every revolution in history always has this carnival-like side. The insurrection will come later. #Jan25


I think I'd rather have that man on my side.

Thursday, February 03, 2011

tick-tock

This WSJ piece is a crucial document on Egypt:
At 4 p.m., the battles appeared to tip decisively in the protesters' favor. An order came down from Mr. Mubarak to the Minister of Interior, Habib al-Adly, to use live ammunition to put down the protests, according to a person familiar with the situation.

Mr. al-Adly passed on the order to his top lieutenant, Gen. Ahmed Ramzy—but Mr. Ramzy refused, according to this person.

"It was a poor assessment of what [orders] his generals would take from him," this person said.

When Mr. Mubarak saw that Mr. Adly wouldn't get the job done, he gave the order for the army to deploy, this person said. Mr. Adly was furious, according to the person. Mr. Adly then gave a sweeping order to pull all police from the streets, from lowly traffic monitors, to prison guards, to the vast armies of truncheon-wielding riot police that had been a ubiquitous presence around Egypt for decades.

Wednesday, February 02, 2011

da brother's gonna work it out

Our former prime minister supported Hosni Mubarak this morning in the name of stability.

Blair said that meant there should not be a rush to elections in Egypt.

"I don't think there's a majority for the Muslim Brotherhood in Egypt. On the other hand, what you've got to watch is that they are extremely well-organised and well-funded whereas those people who are out on the street at the moment, many of them will be extremely well-intentioned people but they're not organised in political parties yet. So one of the issues in the transition is to give time for those political parties to get themselves properly organised," he said.


How's that working out for you, Tony? Here's a taste of the "no elections, leave Mubarak and Suleiman in power" plan in action:

7:41pm Al Jazeera web producer at Tahrir Square says that a crowd of about 500 pro-Mubarak people started throwing rocks over tanks near Qasr al-Nil bridge.


Yes, today has been so chaotic and unstable that not only are Mubarak's loyalist paramilitaries fighting the revolutionaries, they're also sometimes fighting the Egyptian Army as well, or at least doing something that could be mistaken for that. Meanwhile, the revolutionaries now no longer trust the army, as they did when it seemed to be passively enabling their protests over the weekend and yesterday. Thank God we didn't do anything rash and picked the stable option. Meanwhile, people who really are organised - like the spooks - are attacking the people he wants to give more time to get organised, before they can get organised.

And of course, people do get organised. There's a new trade union movement. Tahrir Square has grown a press bureau, a documentation team recording details of Central Security's crimes, support networks of all kinds. The police withdrawal on Friday may have been an enormous blunder precisely because it shifted the revolution from just mass demonstrations to block-level organising. If people survive tonight, I wouldn't bet against a major political organisation (call it the Civic Forum) emerging from the local committees. Look out for this this bloke.

Actually, it's possible that today's news explains the military's anomalous position over the last few days - it may be that they feared that fighting for the government, against the people, would simply destroy the social contract and bring about something indistinguishable from civil war. In the light of their special political role since the Free Officers' Movement. As Fake Hosni Mubarak says:
Whoever said "A captain must go down with his ship" is wrong. The ship must go down with its captain


The genuinely depressing thing about Blair's remarks is the degree to which he's internalised the regime's public arguments, and the degree to which he may still have real influence.

First, the real influence. It seems certain that the Americans only see this whole issue as part of that horrible grab-bag of cynicism known as the Middle East. Blair's role as envoy means he's still plugged into this. He has certainly had extensive contact with Suleiman and friends. Given the importance of all this, he can probably contact the State Department as easily as David Cameron can.

Moving beyond him, Mubarak has traditionally defended his role by claiming that he is a defensive bulwark against extremist Islam, incarnated for these purposes by the Muslim Brotherhood. It has been in his interests to talk up both this movement's popularity and its extremism, and also to talk down the role of all his other enemies. The Brothers and the state have what could be called a special relationship - the Brothers guarantee the special, pseudo-cold war status of the regime, and the regime guarantees their role as a monopoly of opposition. Being the one true opposition has obvious attractions to their leaders; it's also necessary that they be portrayed as extreme enough to warrant the deliveries of M1A1 tanks and prisoners in unmarked business jets. The Guardian's Jack Shenker, who has been doing a heroic job reporting the revolution, has an excellent article on this issue here. You would think that if they could shut down one opposition, they could also shut down one they supposedly consider much more of a threat...

Blair, of course, pretends to be utterly unaware of the clammy snog between the regime and its opposition franchise. But he can't be blamed for not realising that, if anyone's going to bring religious or anti-semitic violence to the streets, it might be the other side. After all, when he spoke, he probably didn't know that Mubarak's goon squad are in the habit of screaming "Jews!" as they assault journalists.

This does, by the way, make you - or me, at least - wonder exactly which organisation was able to put a significant mob on the streets at the drop of a hat, when the NDP had spectacularly failed to mobilise any sign of mass support for days on end. Military dictatorships with religious stylings are far from unknown - that was, after all, the fix Jaruzelski tried to impose on Poland after 1981, mixing more Catholicism and nationalism in with his communism but keeping the security state more in place than ever. And I can well imagine someone - someone, quintessentially, like Tony Blair - hailing cooperation between an Islamist movement and Central Security as being just the kind of faith-based initiative that contributes, helpfully, to shared norms for the new reality. As usual with Blair, it's the secular left that is his real enemy.

kostenloser Counter