Sunday, March 28, 2010

oiling the steel to sharpen the blade to shave the yak

Progress update on fixing the Vfeed.

Dubai Airport has done something awful to their Web site; where once flights were organised in table rows with class names like "data-row2", now, exactly half the flights are like that, they've been split between separate arrival, departure, and cargo-only pages, they only show the latest dozen or so movements each, and the rows that aren't "data-row2" don't have any class attributes but random HTML colours.

And the airline names have disappeared, replaced by their logos as GIFs. Unhelpful, but then, why should they want to help me?

Anyway, I've solved the parsing issue with following horrible hack.
output = [[td.string or td.img["src"] for td in tr.findAll(True) if td.string or td.img] for tr in soup.findAll('tr', bgcolor=lambda(value): value == 'White' or value == '#F7F7DE')]

As it happened, I later realised I didn't need to bother grabbing the logo filenames in order to extract airline identifiers from them, so the td.img["src"] bit can be dropped.

But it looks like I'm going to need to do the lookup from ICAO or IATA identifiers to airline names, which is necessary to avoid having to remake the whitelist and the database and the stats script, myself. Fortunately, there's a list on wikipedia. The good news is that I've come up with a way of differentiating the ICAO and the IATA names in the flight numbers. ICAOs are always three alphabetical characters; IATAs are two alphanumeric characters, which aren't necessarily globally unique. In a flight number, they can be followed by a number of variable length.

But if the third character in the flight number is a digit, the first two must be an IATA identifier; if a string, it must be an ICAO identifier.

memetic post you can safely ignore

Oh yes, blog theme tune.



Auto-suggestion psychology. Elimination policy. The military-industrial pollution of democracy. It's what we stand for. Oh yes, and bifurcation. In order to honour the essentially dualistic nature of the blog, here's another version - how David Holmes saw it.

manufactured controversy

A question, inspired by this ruckus in Jamie Kenny's comments. Is the notion of a manufactured controversy analytically useful?

I can see that the ideas of fake consensus, or fear-uncertainty-and-doubt, are useful. But manufactured controversy presumes that someone is manufacturing the controversy. Presumably they are doing this to make a point of some sort, unless they are simply trolling. They want their side of the manufactured controversy to win. Isn't that, in fact, controversy? The climate-change deniers are full of crap and funded by the coal industry, but they exist.

Arguably, manufactured controversy is a bit of an unspeak concept; if there is no real controversy, therefore there is no real opposition, and my views embody the broad consensus. In some ways, it's a delegitimiser; in others, a tranquiliser. I don't need to worry about the opposition - it's all froth.

science!

Here's something interesting. I grabbed the last 6 months' worth of national opinion polls from Wellsy's and graphed the Tory lead in percentage points. On the tiny chart below, you'll observe that the mean is 10 points; the hatched area shows one standard deviation each side of the mean, and I've plotted a linear trend through it. (You can see a full-size version of it here.)

6 months of Tory leads

The interesting bit; there are 21 polls, out of 158, that showed a Conservative lead of more than one standard deviation greater than the mean. All of them occurred before the 29th of January. There are 24 that showed a lead more than one standard deviation less than the mean. 20 out of 24 occurred since the 19th of February. What on earth could have happened between these dates?

I can't do that, Dave

The posters broke in a big way around the 19th of January and the second wave hit in early February. Clifford Singer deserves a knighthood for this.

Geeky poster is geeky (link)

Friday, March 26, 2010

FAIL

Yes, the Viktorfeed is indeed down. Dubai Airport has redesigned its Web site, and the parser is now not parsing, and the error handling has failed to handle an error where the target page loads but contains unexpected HTML. I've stopped the cron job to save filling up the system logs. There will now be a short break.

Meanwhile, here is some music, or more likely a high-pitched buzzing sound

Note the faintly mischievous expression.

Monday, March 22, 2010

neither Hoon nor Gove, and probably not Opik either

Has there been anyone in British politics quite as depraved as Geoff Hoon? Can my readers help?

Probably the worst defence secretary ever. (You have to realise that he was probably one of the Prime Minister's two designated deputies for nuclear retaliation - Byers, not so much, and at least he got us the railways back.) Iraqi mothers will thank me. Editors should apologise. And on and on it goes. He signed the letter in an effort to sack Brown but didn't send it because he wanted a job in Europe. Now, the bungling backstabbing fucko and neocon extras girl is yours for £3000 per 86,400 seconds. It's almost worth taking a Senlis Council view and deciding that it's worth paying over the money, collecting his lobbying services in a sealed warehouse patrolled by the army, and destroying them just to get them off the streets.

But. But. Come to think of it, the Blaney precedent governs this case - whatever is worthless cannot be the object of corruption.

Meanwhile, Michael Gove reckons that accepting money from a non-taxpaying non-voter who made the money by squeezing the wages of cleaners and charging Belizeans for phone calls at monopoly rates, before joining the legislature under false pretences, and letting him define your party's strategy, without any of its members being so much as asked, is OK. Accepting money from the taxable income of millions of citizens who donated it by their free choice and elected representatives to decide what to do with it is beyond the pale. However, if some sort of national network of political activists was to offer Michael Gove donations and organisational support, that's OK so long as they start with a C.

Of course, Gove personally didn't need any organisation or money, because he got given a one-party seat and was allowed to keep writing national newspaper editorials in his own cause during his "campaign". The only conclusion is that Hoon and Gove are equally repellent; you have to make allowances for the opportunities Gove has missed out on, as part of a generation of Tories deprived of power.

Meanwhile, I called my local MP, Jeremy Corbyn, the most left-wing man in the Commons and 472nd out of 646 for responsiveness about the Digital Economy Bill. Radio silence, of course. I left a comment on Lynne Featherstone's blog, and you know what? She actually responded to it, saying that it going into the washup was good because all parties had to agree to pass it. True. If anyone remembers to object.

Sunday, March 21, 2010

and finally...

A thought: Raj Patel can honestly say that the lurkers really do support him in e-mail. Something quite hilarious about the notion of a blogger getting deified. On the Internet, no-one knows you're God.

Sunday SDR, Chapter 6 - Key Strategic Questions

And we're there! Chapter 6 - Key Questions for the future SDR - attempts to sum up the Green Paper and set some deliverable goals for the full SDR process.

There are six key questions, which also get their own comments threads here; again, one of the salient features is how little there is about the relationship with the US and also that none of the comments seem to find this at all surprising. Not so long ago, suggesting any cooperation with Europe except for the strictest possible interpretation of NATO would reliably get you an avalanche of Tories accusing you of undermining the special relationship. Now, not so much.

It is likely that the American commitment to the NATO Alliance will wane in the next 10 – 20 years...The prospect here, indeed possibly the only prospect, is of closer ties with our European partners through development of the Common Security and Defence Policy....It is obvious that the continual paring down of national capability will end in a Euroforce. Whether this is perchance or by design is a moot point. I think the time has come to stop resisting this and start positively embracing it


The six strategic questions are as follows:
* Given that domestic security cannot be separated from international security, where should we set the balance between focusing on our territory and region and engaging threats at a distance?
* What approach should we take if we employ the Armed Forces to address threats at distance?
* What contribution should the Armed Forces make in ensuring security and contributing to resilience within the UK?
* How could we more effectively employ the Armed Forces in support of wider efforts to prevent conflict and strengthen international stability?
* Do our current international defence and security relationships require rebalancing in the longer term?
* Should we further integrate our forces with those of key allies and partners?


There's also this one:

* To what extent and in what areas should we continue to refocus our current efforts on Afghanistan?


The rest is basically a summary, but it's interesting that a couple of specific policies make it through to the final cut:
Options for enhancing our cyber capabilities and structures to ensure we can defend, and take steps, against adversaries when necessary; and where we might increase our contribution to allied space capabilities or invest in our own national capabilities.


More for reference than anything else, here are the official military planning assumptions, and a list of operations since the 1998 SDR.

Sunday SDR, Chapter 5: People, Equipment, and Structures

So we've had the grand tour d'horizon; we've had the self criticism; we've had the very rapid skip over the nuclear issue; we've had a careful balance of general-purpose capability and counterinsurgent language. Now for some hardcore bureaucracy. It's Chapter 5 of the SDR Green Paper - People, Equipment, and Structures.

This kicks off with the MOD's personnel problems. As in essentially any organisation of the last 15 years or so, there's an invocation of having to learn new skills many times in your career, etc, etc. There's going to be a "whole force concept" review of how the MOD manages its people. There are warm words about looking after our veterans being a moral value. And then there's this:

The provision of accommodation, for example, is a potential disincentive to home ownership and may not represent the best investment we can make in helping families and personnel deal with the demands of Service life.


I would have thought the disincentive to home ownership would be the wages, and the, well, demands of Service life. (How many mortgage lenders are cool with the idea that the signatory may get shot at any moment?) Seriously. What the fuck? Apparently they're looking at "alternative models for accommodation", which might be good if it involved killing off the Annington Homes money pit, but it doesn't sound like it.

On equipment, the general theme of a renewed interest in industrial policy is there, although the section is very general indeed, in fact vague. Tellingly, the issue of operational sovereignty - which has flared up all over again with regard to the F-35 - is raised:
We will have to revalidate our overall approach to:
* Operational Sovereignty. Our Armed Forces rely on assured overseas sources for some important equipment and support but there are cases where specific industrial capability must be located in the UK for operational reasons

There's also a nod to arms exporters, presumably to pass the document through the bits of the MOD involved with DESO and friends.

On organisational issues, the chapter contains a bit more meat; it appears a major re-apprisal of the MOD's structure and business processes is coming, although the drafters warn that the costs of constant reorganisation have been a very serious problem.
Change must be considered carefully in the light of the risks associated with reorganisation highlighted in the Haddon- Cave Report. The future Review will offer an opportunity to re-examine the model and to determine whether and how we might be able to improve on it.

Haddon-Cave is the report on the Nimrod XV230 crash in 2006, which demonstrated that the Nimrod fleet was essentially unairworthy in its entirety and that the engineering and management systems intended to guarantee the safety and effectiveness of the MOD's aircraft. A major issue it identified was the impact of constant organisational change - something of a theme throughout the public sector in the Blair era.

The chapter finishes with a ritual call for greater efficiency. There's also this worrying statement, in the light of the bizarre property-booster bit:
the scope for further rationalisation of the defence estate;

Not again...

In short, if Chapter 3 was impressive, Chapter 5 is poor - with the exception of the reference to Haddon-Cave, it's mostly either made up of truisms or else simply too vague to mean anything at all. And what on earth is this stuff about property? Notably, the comments home in on it at once; it's also noticeable that by Chapter 5, the trolls have landed.

Sunday SDR, Chapter 4: Partnership

Next slide, please. At last, we're there - Chapter 4 of the SDR Green Paper tackles the classic question of alignment with the EU, NATO, and the special relationships. And it's a highly post-American document.

Our current relationships are mutually reinforcing. NATO remains the cornerstone of our security. However, as Europeans, we must take greater responsibility for our security together. Stronger European defence co-operation offers many opportunities, not least in the wider role defence should play in resolving conflict and building peace. The UK will greatly improve its influence if we and our European partners speak and act in concert. A robust EU role in crisis management will strengthen NATO. Playing a leading role at the heart of Europe will strengthen our relationship with the US.


This is the strongest pro-European official statement on defence for quite some time, I think. You'll observe that any contradiction between the EU and the special relationship is denied, but it's also true that there's not very much about the US here at all. In that sense, this is a radical statement.

The Review will need to determine where there is scope to increase the effectiveness of those relationships in delivering our security or to rebalance our investments across the organisations. In particular:
* how we can strengthen European nations’ contribution to global security, including through more effectively aligning resources and priorities;
* how we can further improve cooperation between NATO and the EU;
* how we increase equitable burdensharing within NATO and the EU, particularly with respect to operational deployments;
* whether there is scope for increased role specialisation or capability-pooling within NATO and the EU in order to create a more coherent and capable output;


These are the cliché questions, of course - why won't Germany let their helicopters fly at night, does Europe really need quite so many conscripts, does Austria having a dozen Eurofighter really contribute to anything much. There is truth to them, although perhaps less than there would be if the US Marines didn't impose their own national caveats on the US Army. It's in their nature that they will only be settled by long and imperfect negotiation, and if the UK wants them settled, it will probably have to signal that it's serious about European cooperation.

* whether we should increase our investment in UN peacekeeping, and in particular our contribution of forces to UN operations;
* where we could offer further assistance in strengthening the strategy and planning functions for UN operations at headquarters level; how we continue to streamline and improve the cost-effectiveness of each organisation; and


This suggests a possible use for the exportable surplus of generals identified in comments here.

* how we most effectively generate influence within coalitions and with our key partners


I would argue that an ally whose support is not totally certain has far more influence than one that will go-along-to-get-along with anything...and I suspect that so would the SDR drafters.

Beyond Europe and North America, the Review should consider the merits of formalising our long-standing bilateral relationships and where new and expanded partnerships could bring mutual advantage and reinforce global and regional security. For example, regional security organisations such as the Association of Southeast Asian Nations and the African Union are already playing an important role in ensuring international stability and there is scope to further improve links between these organisations and the EU and NATO. In the recent economic crisis, the G20 emerged as critical to coordinating the response of the international community. Some argue that we must similarly expand the international security architecture to better include emerging powers.


I'm not sure if there's much in this, but it's encouraging that the drafting process isn't focusing just on Europe and the Atlantic.

The "partnership" theme is also used to discuss working with civilian organisations, and the problems of building the reconstruction element of a counter-insurgency strategy. Although the word isn't used, there's quite a bit of the language - if Chapter 3 had a Gian Gentile-like concern for general-purpose capability and adaptability, Chapter 4 at least sounds more like Abu Muqawama.
We have made major strides forward with what is called the Comprehensive Approach – a unified approach to defence, diplomacy and development. There has been progressive improvement, driven particularly by our experiences in the Balkans, Iraq and Afghanistan. In Afghanistan, since early 2008 we have doubled the number of deployed civilian experts and we now have an integrated structure, headed jointly by a UK senior civilian representative and the UK Commander Task Force Helmand, and focused on the rapid delivery of stabilisation effect in an insecure environment, alongside military operations.

The Stabilisation Unit – jointly owned and staffed by DFID, FCO and MOD – has improved the UK’s ability to plan, deploy and direct activities in fragile and failing states, including countries emerging from conflict. In particular, it has established a new Civilian Stabilisation Group with over 800 deployable external experts and over 200 civil servants with the right skills and experience to help countries recover from conflict....

Only local people will determine whether, in the long-term, a country or region will establish self-sustaining stability. They have a right to be consulted on the path that they will take towards that stability. Ultimately, they will lead and own this path. Their knowledge and understanding will also enhance the prospects of our success.

Sunday SDR, Chapter 3: Adaptability and Influence

Well, it's going to be more like "All SDR, All the Time", as I've just noticed that MOD is planning to close the consultation in two weeks. Here goes.

Well, Chapter 3 is in my mind the most impressive bit of the SDR Green Paper so far. It basically sets out the notion that, although the military have as usual succeeded in adapting to the conditions in the field and generally cracking on, the broader defence establishment - the MOD policy-making process, the defence procurement system, the intelligence services, the defence industries - have not done anywhere near as well in coping with constantly changing priorities.

The point is made that although the MOD succeeded, eventually, in turning around a fearsome number of Urgent Operational Requirements very quickly to support the Army in Afghanistan, the very need to issue so many UORs demonstrates that the main equipment programme was dysfunctional. Further, the defence establishment is put on notice that it will have to save money in order to fix the core equipment programme.

As far as answers go, the chapter suggests that there is a need to institutionalise the practice of having regular defence reviews, rather than holding them as and when the Treasury insists, and that this should be set down in an Act of Parliament. Further, they want to alter the strategic planning process - after Iraq, who could possibly object? - in order to "increase the ability of Ministers to direct change", but also to "increase the authority of the Chief of Joint Operations".

There's obviously some tension between these goals - one increases the power of ministers, one the power of the officer corps and specifically the operational command structure rather than the Defence Staff. There's a fine political balance here; if the intention is to boost both Northwood and the ministers, the corollary is less power for the Defence Staff and the civil servants.

Of course, the key to the politics here is procurement, because that's where the money is and because technology eventually becomes policy. Chapter 3 suggests the following changes:
* generating more adaptable forces. Many of our forces are already operating outside their primary roles. We need to strengthen this trend towards taking on multiple roles;
* prioritising our investment in capabilities with wide utility, which are likely to be effective in a range of scenarios and against a range of threats. These would include, for example, support helicopters;

To put it another way, especially if there's not much money around, the MOD can't afford to indulge in hyper-specialised gear. Instead, equipment has to be general-purpose, in order to fit in with a strategy of trying to stay agile in the environment "characterised by uncertainty" laid out in Chapter 1. The helicopter example is nicely uncontroversial, but it probably won't have gone unnoticed that it also fits the carriers.

* creating greater flexibility between Regular and Reserve Forces to ensure access to a wider range of skills and a larger personnel pool;


The MOD has been trying to do this ever since the last SDR decided that the Territorial Army was too big and needed cutting; after Kosovo, when an unexpectedly large mobilisation was needed, they changed course, but despite using the reserves heavily in Iraq and Afghanistan, they continued cutting the headline numbers until very recently. The reserves were also given a homeland security/anti-terrorism role, being asked to provide a battalion-sized Civil Contingency Reaction Force for each regional brigade. Experience of actual civil contingencies, like the 2007 floods, led this to be abandoned as the civil authorities found they didn't have any need for a CCRF but did badly need almost every other specialisation.

* developing a greater understanding of the appropriate balance between technological edge and larger numbers of platforms;


This was a Hoon-to-Reid era trope - although the services wouldn't get as many (tanks/ships/aircraft/whatever), high technology would make up the difference and therefore the costs of Iraq could be absorbed.

* relying on being able to reconstitute military capabilities, to enable us to access a full range of balanced capabilities with appropriate warning time without having to maintain those capabilities at all times.


To put it another way, if it's possible to re-create certain specialities quickly, we don't need to have them permanently on hand. This requires a different view of the industrial base - does this suggest that we need to pay more attention to keeping the industries involved in the UK?

Further, the procurement system is asked to:

# increasing our use of mature technologies when setting requirements. This would reduce the risk that research and development could lead to delays and cost increases in the programme;

# increasing our use of spiral or modular development, in which we build a capability to meet our current requirements but with the capacity to upgrade that capability by adding functions or technologies as they become mature or new threats emerge.


The Adaptability section checks out by admitting that the MOD struggled to understand what was happening through the 2000s, that it lacked understanding of the countries it operated in, and that it failed to make use of expertise available in other government departments, in academia, and in other institutions. They propose more openness and suggest "empowering the Concepts and Doctrine Centre", their in-house thinktank. Unfortunately, this lacks credibility - at the same time as this statement was issued, the MOD is in the process of shutting down its Research and Assessments Branch, whose job this is, as the MOD's favourite blog points out.

As far as influence goes, the chapter suggests that the military should be doing more advising, defence diplomacy, and the like, and also tackles the nuclear question very briefly. Essentially, it simply says that we need to make decisions now about Trident so as not to commit to disarmament by accident - this is a far more controversial statement than the SDR makes it sound, and it gives the strong impression that simply nobody wants to discuss it. It's a let-down at the end of what is otherwise a very sensible document.

Sunday, March 14, 2010

Premature evaluation - The Spirit Level

Richard Wilkinson and Kate Pickett's The Spirit Level is a vigorous polemic for social democracy, something we're probably in need of as the neo-liberals recover from the 2008 experience.

Unlike most such, this one is based on data - specifically, a whole battery of socioeconomic indicators that turn out to be strongly correlated with income inequality. In fact, the paperback comes with a handy table of the R-squareds and p-values of all the indicators used, which range across life expectancy, imprisonment per capita, patents issued per capita and much else. Everywhere, it seems, more egalitarian societies tend to do better.

This observation is rather more impressive than quite a bit of the book - there's too much back-of-a-fag-packet neuroscience of the sort that actual neuroscientists run a mile to avoid about mirror neurons and such, as well as a fair bit of 1970s-ish romanticisation of the supposedly ideal status of hunter-gatherer societies. Steven Pinker's work on the history of violence hasn't landed here; in places it's almost nostalgically sweet.

The data, however, speaks for itself. It's true that quite a few of the charts derive a lot of their correlation from a few outliers, but the outliers invariably point to the same results - specifically the United States, which reliably turns out to have truly awful results for many, many tests - and also very high inequality. Similarly, there are a whole string of statistics that are driven by a group of post-Soviet states that turn out to be dramatically unhappy, conflicted, violent, unhealthy, etc for their level of income; of course, these societies underwent a historic explosion of inequality.

Many of the results have been checked by carrying out the same analyses with the 51 US states, which gives rise to the same conclusion and another crop of interesting outliers. The states of the Deep South are reliably terrible. They are highly unequal, and they get the effects - but they are far off to the top right of the trendline. In a sense, their marginal productivity in terms of inequality is unusually high - for every extra point on the Gini coefficient, they manage to produce a sharply higher degree of suffering than the national average.

On the other hand, there's the importance of being urban. The more metropolitan the state, the less it suffers from the impact of inequality - New York has the social problems of the average, despite being very unequal. And there's the Alaskan question.

The Alaskan question? Many people on the left are keen on the idea of a citizens' basic income, and oddly enough, there is one territory with one in this study. Alaska, famously, distributes its oil revenues equally among the citizenry, and is therefore the most equal society in the United States. However, it also succeeds in being reliably among the worst on every other measure you can think of. Clearly, the statecraft of Sarah Palin must have some impact, but it's equally clear that it can't be the whole explanation.

Unless there is some huge missing factor that invalidates the whole data set, we have to consider that this particular basic income experiment has failed to deliver the benefits of equality. Alaska is, of course, a very special and atypical place - but it's not that different to, say, Norway, another sparsely populated, mountainous, northern territory bordering on Russia whose economy is heavily influenced by oil and gas, forestry, fishing, and metals and whose government decided to take a radical approach to the oil revenues, and where a lot of people own guns. And Norway is both very egalitarian and reliably in the very top of all the metrics in The Spirit Level.

Perhaps the answer is precisely that the Alaskan basic income is free money? Despite all the stuff about mirror neurons, etc, etc, it seems that the trade secret of equality is - equality. It takes a long time for Wilkinson and Pickett to get to this, but the difference between handing out oil windfalls and real egalitarianism is that only one of them is founded on a different balance of power between classes. A lasting reduction of income inequality must be founded in a lasting reduction in the inequality of political power - otherwise it may not last, and it may not even have much effect.

Another interesting point is that changes in relative economic success among nations seem to have little effect on human happiness or security. Obviously, a total crash will do it. But once a certain threshold level of per-capita GDP is passed, Wilkinson and Pickett argue, pushing into the G8 doesn't change much. They therefore argue that economic growth is useless. However, they then note that a whole range of their metrics, like life expectancy, do seem to go up a percentage point or two a year in the rich nations anyway. Which sounds a lot like growth.

It might be more accurate to say that growth relative to other industrialised states is not particularly important within the normal range of variation, although in absolute terms it is. However, the chart in question is quite heavily driven by the US outlier - which suggests that the costs of enough inequality will essentially swallow all your economic growth.

Eventually, the upshot of TSL is that the world, and especially China, needs trade unions.

cooking with John von Neumann

A thought; it's surprising how much you can learn about computer science from cooking, and cooking from computer science.

The von Neumann architecture - when you're cooking, there is a central processing unit, which is the top of the stove, there is mass storage, which is the fridge and the cupboard, there is a user interface, which is your attention span, there is RAM, which is the work space, there is an output device, the table, and there's also a network interface - the cook's relationship with those around him or her. At any given time, any one of these elements can be the system's rate-limiting factor - but it is a timeless, placeless truth that there is always one that is the system's bottleneck.

More RAM is always welcome - Whether it's fridge I/O, stovetop processing cycles, the interface with the cook, the queue of jobs waiting to be written to the table, or congestion in the social network, it's always the free space in RAM that acts as a buffer for the whole system. If you've got enough RAM, you can cope with most problems without anything dire happening, by either queueing things up or else pre-fetching them from the cupboard ahead of time.

But if you go below a certain threshold level, the system tends to become increasingly unstable and you risk a crash and possibly dinner loss.

Throwing hardware at the problem works...until it doesn't - You can only go so far in clearing space around the kitchen - if your demand for space goes too high, you need a bigger kitchen. Therefore, we need to pay close attention to scaling.

Amdahl's Law and the trade-offs of parallelisation - Doing things in parallel allows us to achieve extremely high performance, but it does so at the expense of simplicity. You can see this most clearly in classic British cooking - many different high-grade ingredients all require different forms of cooking and cook at different rates, but must all arrive at the same time on the plate. Of course, as Amdahl's law states, when you parallelise a process, it's the elements you can't parallelise that are the limiting factor. You can't cook the filling of the pie and the pastry in parallel.

Distributed processing is great...until it isn't - Similarly, distributing tasks among independent nodes allows us to scale up easily and to achieve greater reliability. However, these goals are often in conflict. The more cooks you have in the kitchen, the harder it is to maintain consistency between them, and the more critical it is that you get the networking element of the problem right. Strange emergent properties of the system may surprise you, and it seems to be a law that the consumption of drink scales O(log n) with the number of cooks.

Test-driven development - only fools rely on a priori design to guarantee the quality of their sauce. It's absolutely necessary to build in tests at every step of the cooking process, both to maintain quality, and to stay agile in the face of unforeseen problems and user demands.

The only way to avoid coding bugs is to avoid coding - Ever since the days of Escoffier, cooks have known the importance of using well-known and well-tried recipes as modular building blocks. Escoffier started off with two basic sauces on which he built the entire enterprise of French haute cuisine. So use the standard libraries, and don't try to invent a new way of making white sauce - just type from sauces import roux

Love the Unix core utilities - Look around your kitchen. What utensils do you actually pick up and use every time you cook? Obviously, you need to invest in the things you actually use, rather than expensive shiny gadgets you don't fully understand. And you need to master the technique of using them. Get a big sharp knife.

Shea's Law - Shea's law states that "The ability to improve a design occurs primarily at the interfaces. This is also the prime location for screwing it up." This is always true of cooking. If you can't get back from the fridge around the inevitable spectators in time to complete a frying loop, or two flavours fail to get together, or you catastrophically fall out with someone in the kitchen, bad things happen.

Loop constructs are fundamental to everything - Perhaps the most important decisions you will make will be whether you minimise how long a process step takes, or whether you minimise the number of steps in the process. But the number of different operations on the CPU - the stove - is the main driver of complexity.

Everyone underestimates the problems of deployment - How will your recipe work in another kitchen, or in the same kitchen under different circumstances?

The hacker ethos - If you have to know what line 348 will be before you open a text editor, you'll never get started. Similarly, you will get nowhere by wondering what temperature in degrees you should saute onions. Chuck your code in the oven, and see if it returns a roast chicken! Also, the fun of having a secret recipe is actually the fun of sharing it with others.

Junk food is bad for you, but sometimes it is unavoidable - Software produced by huge sinister corporations and stuffed with secret additives is likely to make you fat and stupid. But sometimes you need a pizza, or a really fancy presentation graphic, or a pretty music library application for a mobile device. Everyone does it - the thing is to maintain a balanced diet.

Saturday, March 13, 2010

Sunday SDR data points

Some data points relating to my last Sunday SDR post and specifically the points about the way in which some forms of advanced technology are becoming much more evenly distributed. Here's Armscontrolwonk getting the wrong year, but more importantly, linking to India experimenting with hit-to-kill antiballistic missile/antisatellite technology.

Here's DARPA starting an app store for the military - interestingly, they've settled on Android as the standard operating system, just like major Chinese mobile device vendors have (it's cheap and highly open). I think I said something in Charlie Stross's comments about putting a drone app on your iPhone.

Really interestingly, they've drawn the obvious conclusion - if you're going to use smartphone-like devices based on civilian high technology, you'll need a cellular radio network for them, and as a result, DARPA is also tendering for lightweight, rugged portable GSM/UMTS equipment. Both this lot and the OpenBTS people will be interested. (This post refers, slightly, as does this Bruce Sterling screed.)

Meanwhile, this is interesting:
Its apparent success has prompted some to wonder if the model can be extended to other areas of British space activity.

A recent report prepared jointly by industry, government, and academia said a PFI could be used to develop a national Earth observation programme. The idea has already been dubbed "Skysight".

Tuesday, March 09, 2010

tha mun laike wi't gangling-iron, Councillor Duxbury

Well, well, well.

The Conservative chairman, Eric Pickles, last night appeared to disown the leadership of the Young Britons' Foundation, a rightwing training organisation for young Conservatives whose officials have described the NHS as "the biggest waste of money in the UK" and suggested the waterboarding of prisoners can be justified.

Pickles spoke last week at a YBF rally at the House of Commons and the group is working with Conservative Future, the party's official youth wing, on pre-election training of young Tory activists. But yesterday that relationship came under serious strain.

"We don't agree with these views," a spokesman for Pickles said in a brief statement. "The YBF organisation is independent of the Conservative party."


Sometimes, what you need is a Yorkshireman. But it would be very interesting to know if the Conservative Party actually knows what, legally, the YBF is, how much money it's received, and how much VAT has been paid.

This is interesting, too:

Cutts said that, as an organiser for Conservative Future's East Midlands region, he came under pressure from the national organisation to ensure there were sufficient paying customers for the YBF courses.


If this is a commercial arrangement in the meaning of Section 50 of the Act, it's a slightly unusual one.

(If anyone gets the joke in the title, I'll be delighted.)

Monday, March 08, 2010

marginally less disturbing search request

It's been a while since this referrer's been in:
proxy-ext-76.osd.mil [Srv] [entrypoint #81] 2007/05/falklands-myths-4-american-and-european.html Mar 04, 17:36:34 [0:00:00] views: 1
Google (ascension island 1982 jet fuel)
Proxy: proxy-ext-76.osd.mil - RSRCNISA01


I think that counts as a Disturbing Search Request, if not quite as disturbing as when the S in OSD was Donald Rumsfeld.

a technical stop

Something interesting on the Air Cocaine story. A lurker dropped off this:
El avión, Boeing 727, siglas J5-GCU de Guinea Bissau, aterrizó en el aeropuerto internacional La Chinita de Maracaibo a las 11:00 pm del 16 de octubre, procedente de Panamá. Allí permaneció poco más de una hora, mientras llenaba los tanques de combustible.

Antes de despegar, sus tripulantes presentaron un plan de vuelo con destino en Bamako, capital del país de África Occidental. No obstante, el aparato voló al sur del país. Cuando sobrevolaba Barinas, el piloto se reportó a la torre de control de Barquisimeto e indicó que tuvo que desviarse por malas condiciones atmosféricas. El vuelo nunca fue declarado en emergencia.

I think the point is that not long after leaving Maracaibo, the B727 announced a change to the flight plan and then possibly landed on yet another remote airstrip to load the cocaine, and possibly to refuel, and then continued towards Brazil and Africa.

Sunday, March 07, 2010

Sunday SDR: Chapter 2

OK, so it's time for another chapter of the Strategic Defence Review as a Blog.

Chapter 2 begins as follows:
The use of force as an option is becoming more complicated. It is likely to become more difficult to use force in the way in which we have used it in the last two decades.

This is of course code for Iraq. The Iraq experience is a considerable theme through the chapter.
Many of our assumptions about joint working and expeditionary capabilities have been validated. But experience has shown that our operations have developed in more complex ways than we envisaged. We have sometimes underestimated the intricacy of working in multi-national operations and with non-military actors

To put it another way: We were right to expect we wouldn't spend all our time in Germany. Further, we had to talk to the RAF. But one particular operation turned out to be much more complicated and much more serious than we allowed ourselves to imagine.
Looking ahead, The Future Character of Conflict will grow more complex. We are likely to face a range of simultaneous threats and adversaries in challenging operating areas – such as fighting in urban areas against enemies concealed amongst civilians. We are also likely to be subject to greater scrutiny from the media and public, both in the UK and overseas. Communications is now a key component of any campaign.

That seems to be communications as in "strategic communications" - PR, in other words. Nothing to do with being Better Off With Map And Nokia. Snark aside, again, this is the experience of Iraq glaring through.
Technological development, especially in the fields of cyberspace and space, may further change our understanding of conflict. It is likely to be more difficult to maintain our technological edge over some adversaries, or to bring that edge to bear on others, with a profound effect on the way we operate.

Anything electronic is now cheap, and the big power monopoly of satellite reconnaissance is breaking down.

There follows a list of operations and arguments that tend to support the 1998 SDR and the later New Chapter. They do not include Iraq, and only mention Afghanistan in passing on the grounds that we got there logistically, until we get to this paragraph:
Special Forces have demonstrated their value across a broad spectrum of activity, from operating alongside our conventional forces in Iraq and Afghanistan to capacity-building with partners or hostage rescue.

However, when we get on to the "lessons learned", we get this:
Our assumption that we could “go first, go fast and go home” has proved false. We believed that we could deploy our forces for the most difficult early intervention stage of a conflict, and leave the subsequent stabilisation and development tasks to partners. But we have not been able – or wished – to disengage as we had planned. We have therefore further improved our ability to sustain deployed forces, including, for example, through additional procurement of strategic lift.

I think this is important. Going first, going fast, and going home was very close to the early Rumsfeld view - airpower, strategic mobility, force protection, and an almost neurotic self-assertion towards allies. It's rather what the European Council on Foreign Relations says here; Europe was meant to do the boring stuff. It reminds me of the old line about "America cooks and Europe washes up". Well, if you never wash up, eventually you get typhoid. The reference to additional lift was the decision to lease, at vast cost, and eventually buy the RAF some C-17 transports - a sort of shadow of the concurrent procurement train crash around the Future Strategic Transport Aircraft.
The international and national policy and legal framework is having an increasing impact on our operations. Defence continues to make an important contribution to tackling terrorism overseas, following the lines set out in the SDR New Chapter in 2002. The role of Defence in working with other departments to tackle the drivers of terrorism, and to build security capacity, is crucial – although the scope for conducting overseas counterterrorism operations is narrower than envisaged in 2002.

Indeed - 2002, and the spirit of 2002, are a long time ago, and:
In many cases, our operations have developed in much more complex and dynamic ways than we envisaged and planned for, and we have not been able to adapt as rapidly as we would have liked.

Indeed.
In particular, in our focus on our geographical area of responsibility, for example in Basra, we may have placed insufficient emphasis on the multi-national operational level. In the later stages of operations in Iraq, the full integration of UK staff into US and coalition headquarters significantly improved the coordination of our contribution. We are taking that lesson forward in Afghanistan.

To put it another way: We thought we could ignore what was going on in Baghdad, Anbar, and Multinational Division South-Centre, and just crack on in Basra without rocking the boat. But it's impossible to divide the problems of war, whether between land, sea, and air or between geographies within the same theatre. When they wanted war elsewhere, we opted out of the big decisions and lost the ability to say no effectively.
Our deployment of formed headquarters and formations for limited periods has not reflected the need for “campaign continuity”. We have now extended the tour lengths for key headquarters personnel and are looking at options that would ensure greater continuity throughout the headquarters. We are clear that we need to go further to produce better campaign continuity.

This was a problem for the Americans in Vietnam and also for the British Army in various counterinsurgencies. It's probably common to all armies involved in a long war that isn't utterly central to their worldview, because it's driven by career structures. To be a general, you must have a general's command, and why would you be a general if not to command? Further, what they usually command is a formation, and formations usually rotate. Ad-hoc geographical or functional commands are against the bureaucratic structures involved - perhaps it's because of this that they are always necessary.
We have found it challenging to identify and rapidly implement lessons in doctrine. This is inherently difficult, but in some areas we have already moved a long way. The Army recently issued a new Counter-Insurgency Doctrine, and we now have a dedicated training facility for counter-insurgency in the UK.

Well, they're right, really.
Often, innovation within the operation has minimised the adverse impact of these weaknesses. In our current operations, we have incorporated those insights into our strategic policy. Our Afghanistan/ Pakistan Strategy, and General McChrystal’s strategy, are based on a clear understanding of the challenges we face, a long-term vision founded on integrated political, development and military action and an overarching regional approach. Our Urgent Operational Requirements (UORs) procedures are delivering the equipment our forces need as the requirements evolve. The Government has approved over £5.5 billion of UORs in Afghanistan since the operation began.

To put it another way - we muddled through, sort of. The reference to UORs corroborates this - part of the reason why the armed forces don't buy all they need through UORs is that they go around much of the procurement process, in order to be urgent. This means, of course, that they may end up paying more, or getting less. On the other hand, the very fact that they needed to raise £5.5 billion worth of them for Afghanistan - that's as much as the carriers - suggests that the normal procurement process is dysfunctional.

What about the future? It's likely to be:
* Contested – access and freedom of manoeuvre – even as we attempt to deploy into the regional theatre – will have to be fought for;
* Congested – we are likely to be unavoidably drawn into urban areas, the littoral and lower airspace;
* Cluttered – we will find it difficult to discriminate between a mass of ambiguous targets – friendly forces, other international actors such as non-governmental organisations or development agencies, media representatives, local civilians and our adversaries;
* Connected – key lines of communication, including critical military infrastructure, maritime chokepoints and computer networks, will be vulnerable to attack and disruption; and
* Constrained – legal and social changes will place additional limits on our actions.

This is true, but hardly original. This is a good book.
Our preferred way of warfare – concentrating force, bringing technology to bear and seeking rapid defeat of our adversaries – may not be as effective as it has been in the past.

Is that our preferred way of warfare? I think this might need a debate. It sounds a lot like a classic statement of the American way of war, which may be the problem. Of course, nobody wants to disperse force, fail to use technology to best advantage, and seek endless, inconclusive struggle - but if Rupert Smith is right and struggle tends to be endless and inconclusive, and technology less decisive than expected, perhaps this should have some bearing on our preferences.

And here comes the dread word: "cyber".

Cyber Space, in particular, poses serious and complex challenges for UK security and for the Armed Forces’ operations. Our increasing dependence on cyber capabilities creates opportunities but also serious vulnerabilities. Cyber attacks are already an important element of the security environment and are growing in seriousness and frequency. The most sophisticated threat is from established and capable states but cyber eliminates the importance of distance, is low cost and is anonymous in nature, making it an important domain, not just for hostile states, but terrorists, and criminals alike. Cyber space is critical to much of our military effort here and overseas and to our national infrastructure.

Note that the most sophisticated threat comes from states - not the main or the most serious threat. Of course, if the feared attack involves an electron microscope or a quantum computer, a state is the most likely attacker. But it's in the very nature of information security that the great overwhelming majority of threats come from a huge diversity of tiny actors, and they are just as capable of doing serious damage as anyone else.

Further, defence against these threats tends to be the same - basically, sensible network management. The good news here is that there is no talk of giant firewalls or of "cyberdeterrence" - just of sensible security precautions. Further, the realities of the threat environment are taken seriously. No Dr. Evil plots here, nor cold war fantasies, just a space rather like the sea. The upshot of this is that the UK has far greater interests in keeping the infrastructure up, working, and open to all than it could possibly have in disrupting it. Very like the sea. The White House appears also to be heading this way, as Kings of War points out.

The National Security Strategy also set out the increasing challenges we face in Space. The Armed Forces’ dependence on space has grown rapidly over recent years. Access to space-derived information is now critical to our ability to conduct operations. This makes us vulnerable. The development of offensive counter-space capabilities is a particular concern. But, given our reliance on assets we do not control, there is also a risk of loss of access in periods of high demand – such as during large-scale operations or in the event of a sudden reduction in existing capacity. A continued close relationship with the US underpins our access to space capabilities. But we intend to look closely at how we contribute to allied programmes or develop national capabilities.

This is probably the most significant paragraph in the chapter. After 1971, the UK hasn't tried to maintain its own reconnaissance satellite capability, nor has it participated in multilateral projects. It is thought, although as with everything between the UK and the USA, it is not written down in anything subject to ratification, that there is an understanding that the USA would share its overhead imagery with the UK. We know that this was turned down at least once during the Falklands War.

Tellingly, during the Iraq war, European countries fell in three groups. Those who had their own imagery - France and Germany. They didn't participate at all. Those who got such a capability after the spring of 2003 - Spain and Italy. They left early. Those who had nothing at all - everyone else, basically. The outlier is Turkey, which didn't have such a capability (although they did have representatives at the EU Satellite Centre) but didn't get involved. Then, the Turks probably had good human sources in Iraq. They've since ordered a high-resolution photographic satellite from Telespazio of Italy.

Exactly what the US chose to share with us out of the wealth of imagery its national technical means, as they say, produced remains one of the great questions about the UK's involvement in Iraq.

Research and development investment in defence technology in emerging nations has been increasing significantly over the past decade. Some key equipment produced by these countries is already as capable as equivalent equipment produced by the UK and our key allies and partners.

Civil investment in research and development, both nationally and globally, is now much larger than equivalent defence spending. Much of this research is developing technology – for example in communications, materials or biomedical science – which could be used in a military or wider security context. But the Ministry of Defence and our international partners in defence can expect to have less visibility of and expertise in such cutting edge technology than we have had in the past.

Loss of our technological edge in significant areas of military capability would have a profound effect on the way we operate.

This is the Arduino question; the proliferation of what used to be technology confined to the superpowers, or as Phil Hunt put it, what happens when a Congolese workshop with a RepRap can make a surface to air missile? Arguably, the key point here is that there is nothing we can do about it except for getting more like that ourselves - which comes back to the procurement economy.

As Kings of War's David Betz says, this is an argument for general-purpose forces more than anything else. He also quotes the Navy as follows:
* Firstly, what do you want to defend and what are the Standing Commitments for Defence?
* Secondly, we need to have a clear idea about what we as a country would aspire to do on our own.
* Third, where the UK is operating as a coalition member, how do we want to influence our partners?

Sunday Stasi HQ Bloggin'

I recently had the opportunity to look round the Stasi's old head office (as visitors to my photo blog will be bored beyond belief by). A couple of things - first of all, here's something in the spirit of the last post.

A real conspirator's radio

That - as you may be able to make out from the brass plate - is a model of the concealed shortwave transmitter used by none other than Richard Sorge to send his reports from Tokyo so Stalin could comprehensively ignore most of them. (I don't think I've blogged this interview with his Japanese mistress in Die Zeit before, so there you go.) The East Germans made a bit of a cult out of Sorge (and you bet we would if he'd been a Brit) - as you can see from this photo.

The building has a sort of Stalinised Royal Festival Hall chic to it; this is the private meeting room inside the Minister's offices.

Communist executive-suite luxury

They could make anything, as long as they could make it out of wood.

Hypermodern marquetry

Talking on this device was probably unwise.

East German personal tech

But more to the point, here's the display in Chief Directorate VII (Counterespionage and Police Internal Affairs)'s unit hall of fame about their campaign against Amnesty International.

You know you're indecent when the Stasi put your photo on the wall

The text is in a truly awful bureaucratic German; I will try to render it faithfully.
Amnesty International - a "bourgeois (or civil - the German word is fundamentally ambiguous) human rights organisation" - is strongly oriented towards slander of the socialist states. The colleagues of Chief Directorate VII contributed successfully to identifying the enemy efforts to create AI operational bases in the GDR and to rendering their attempts at discrimination ineffectual.

The exhibits are a collection of Amnesty leaflets, reports, letters and the like, which they presumably collected by slipping over to the West and going to their street stalls. These are described at the bottom as "sichergestellte Hetzschriften der Amnesty International", which translates as something like "securely recovered hate-sheets" and usually refers to something like Der Stürmer.

who hid a radio under the stairs?

Here's some more Iran sense. I especially like the bit where the Content is Free comments pit tries to explain to the Iranian all about Mohammed Mossadegh and all. that. jazz. Of course, he's essentially reiterating the former head of the IRGC's line.

I've said before that the practicalities of this depend largely on what kit is available inside Iran. If there are a lot of Thurayas/other satellite terminals about, it might be possible to buy airtime and send the vouchers, or better, just the numbers off the vouchers. How many minutes could you fit in one encrypted e-mail? Quite a lot. But they are expensive, trying to buy one openly there would be the height of stupidity, and some of the biggest users are the military and the oil'n'gas sector.

He refers to this Japanese comsat project, which is indeed cool (in fact, pretty much everything JAXA does is quite cool) and is designed to use the classic footy'n'porn 45cm satellite dish antenna. The technology - phased-array beamforming antennas and on-satellite packet switching! - is impressive. However, as comments here pointed out way back in June, the problem with this is that dishes are quite visible and 'totherside has already got used to the idea of harassing and intimidating satellite TV users. You don't need to do electronic warfare to detect dishes - you just need your thugs to go around looking for dishes and stomping people.

Clearly, a key element in designing something like this will be minimising the antenna requirement. (Actually, the antenna is the key element in designing any radio whatsoever.) Again, the land-mobile satellite systems offer a good model - for things like the Globalstar/Iridium/Thuraya handsets, which do voice and messaging and up to 144Kbps of IP, the form factor is rather like a clunky early 90s GSM device, and the Inmarsat BGAN broadband terminals use a rectangular folding antenna, about the same size and rough design as a netbook. (The smallest available is the Thrane & Thrane Explorer 110, which breaks down in two parts, measures 20 by 10 cm, does burst speeds of 384kbps and Tx power of 10dBw.)

Similarly, I'm dubious of the idea of having a specific satellite - they are predictable, after all. Again, it would probably be better to hide in the "legitimate" traffic and the existing satellite constellation. So, it looks like the solution would be a low cost radio board that drives whatever antenna you feel you can hide, using one of the land mobile satellite services or possibly the radio hams' AMSAT. It must do at least 10dBw and not require any fixed installation. (The AMSAT people have a linkbudget calculator here; clearly, that Thrane & Thrane device must be doing something clever.)

The Inmarsat stuff works in the L-band, around 1550MHz (check out the coffee mug antenna while you're there), which is handy as the other band they use is too high for the Universal Software Radio Peripheral, the open-source radio kit. Which currently costs about $700 - and it's illegal to sell them to Iranians if you're in the US, which the manufacturer is. Not that buying one would be the smartest move either, but there you go.

So, it needs to be cheaper than the USRP, it needs to be feasible to make it from parts available in - say - Iran. On the up side, it doesn't need to be as smart as the USRP because it's specialised to task, rather than a general purpose device that has to be capable of a lot of different radio profiles.

a donation of minus £25

Before moving on to more serious issues, does anyone know what, exactly, the Young Britons' Foundation is? I mean, if you were to sign a cheque to it, what kind of a legal entity is it that cashes it? (I'm tempted to cut them a £10 cheque, just to see what comes up on my bank statement. Or possibly a whole lot of 1p cheques. But that would be infantile. Wouldn't it?) This is an interesting question. Is it a commercial company? Is it a charity? Is it a club of some sort? Is it an unincorporated association? If it was a company limited by guarantee, the usual set-up for a wanktank, it would have to file accounts somewhere, but the funds passing through it would be tax deductible.

I ask because despite its aims:
The Young Britons’ Foundation is recognized as the leading provider of state-of-the-art political training to conservative activists in Britain.

YBF offers a number of one-day specialist training workshops in London as well as bespoke half-day campus campaign workshops throughout Britain.


It is conspicuous by its absence from here. Neither is it a company or a registered charity.

I can see a couple of possibilities. One is that whatever-it-is claims that by charging participants a fee, it's providing services to the Conservative Party on commercial terms and is therefore exempt from registration under Section 50(2) f of the Political Parties, Elections, and Referendums Act 2000. This has some interesting consequences - if this is commercial activity, then surely someone is making a profit from it. If these services were supplied at a loss, the terms would not be commercial ones.

Another is that Section 52(1) g applies, which excludes from registration
the provision by any individual of his own services which he provides voluntarily in his own time and free of charge;


It's possible that the YBF is merely a collection of activists who voluntarily provide these services to the Tories. But the services are not provided free of charge (and in that case, what becomes of the money?).

My favourite explanation, however, is that Section 50(4) b states that:
(4) In determining—

(a) for the purposes of subsection (2)(e), whether any money lent to a registered party is so lent otherwise than on commercial terms, or

(b) for the purposes of subsection (2)(f), whether any property, services or facilities provided for the use or benefit of a registered party is or are so provided otherwise than on such terms,

regard shall be had to the total value in monetary terms of the consideration provided by or on behalf of the party in respect of the loan or the provision of the property, services or facilities.


I think I see the pattern. The argument is clearly that his courses are worthless, and therefore have no market value whatsoever, and therefore, cannot possibly constitute a political donation subject to registration under the 2000 Act?

This is not impossible in practical terms. Consider the YouGov polling of marginal seats; the difference between the famous Ashcroft-targeted marginals and the uniform national swing has narrowed sharply over the last 18 months, which seems to suggest it's possible to spend and spend on a Tory campaign and actually do worse.

Saturday, March 06, 2010

blogging ethics panel around me, please!

It may be of interest that Donal Blaney is the only blogger I've ever heard of whose blog set a robots.txt file to stop the Internet Archive Foundation from indexing it.

Google's Web-in-RAM systems are rather less fastidious than Brewster Kahle's outfit (which invented quite a bit of the technology behind Google and other big clouds, like the containerised data centre). The site: command, for example, remains informative.

kostenloser Counter