i haz bin in yr AR standardz, facilitatin yr interop. kthx!

So I had the opportunity to take part in an Augmented Reality standardisation meeting on the fringe of this year's 3GSM Mobile World Congress. First of all, it was the year the heavens opened (someone on twitter said it was as if the show had turned into Glastonbury) and I got drenched and my shoes went bad, and my cab didn't take me to the Telefonica R&D building in Via Augusta but instead to the main switching centre, this amazingly domineering building...

Telcos - they live in places like this, they know where your dog goes to school, but can they tell you if it's really your bank on the line?

So I got soaked again, and eventually arrived, and spent the first session listening to my shoes rotting. I acted as scribe for the session on AR browser implementations, markup language vs. JSON, native application vs. browser plugin and the like. I hope I contributed something of value. I have a Flickr set of the annotated flip charts here; I've been asked to help prepare the final report. Which just goes to show the enduring truth that if you want to influence something, wait until the very end and sum up with a balanced account. Supposedly this used to be the way to pass the Diplomatic Service exams - buy a pipe, puff on it occasionally during the team exercise, then "sum up with a balanced account". But you're not allowed to smoke these days.

tasks

I'm not quite as sceptical as some about this. However, it's not clear to me how this differs from the sort of thing UNOSAT does all the time - here's their analysis of imagery over Abyei, the key border area between North and South Sudan. Actually it looks like the "Enough Project" is going to be using UNOSAT imagery itself, going by UNOSAT's own website.

If you follow the link you'll see that they have more than reasonable capability (50cm resolution) and that they routinely observe the presence of refugees/displaced persons and returnees, construction, and the like. There's obvious relevance to an effort to monitor potential conflict along the border, especially as oil prospecting is an issue. You can't easily hide oil exploration from a satellite that can resolve objects 50cm across.

However, the downside is that the UNOSAT report is comparing images over a two-year period. I would suspect that they will need much more frequent passes to be operationally responsive, which is where the costs get interesting.

Also, I've just been over to the website and it's a bit of an unstructured clickaround. What I've always liked about MySociety sites is that they all have a function - FixMyStreet reports things in your street that need fixing, WDTK issues Freedom of Information Act requests, TWFY looks up information on MPs, TheStraightChoice logged what candidates promised and said about each other during their campaigns. DemocracyClub, for example, worked because as soon as you logged in it gave you something to do and some feedback about doing it, and then it hassled you to do something more. It had structure.

Notoriously, if you don't give volunteers something to do as soon as they show up, they'll wander off. It is nowhere easier to wander off than on the Internet. And so there's a button to twitbookspace it and a donation link. There isn't, however, a to-do list or, say, a list of pairs of images that need comparing.

what could possibly go wrong?

So someone's trying to raise $150,000 to buy a satellite from the bankruptcy of TerreStar, in order to "Connect Everyone". I admire the aim, but I'm concerned that this is going to be a round of forgetting that a lot of perfectly good GSM operators are doing just that. Also, I can't find any reference to what they intend to use for the customer-premises equipment except that "we're building an open source low cost modem", which would be better if it came with a link to the source repo, right, or at least some requirements documentation? I'm also a little concerned that the team includes this guy:

Fabian is a NYC based Swiss wanna-be-entrepreneur who spends all his time trying to make meaningful connections between ourselves and business.

(and I chose charitably) but not anyone whose potted bio mentions being an RF engineer.

Actually, I think that it would be more worthwhile to start off with the low-cost open source satellite radio, as this may be the difficult bit and would be highly reuseable in other projects. A lot of Indian or African GSM people would find a cheap satellite radio very useful for their backhaul requirements. Depending on the spec it could be used with things like the amateur radio AMSATs, the transponders on the ISS, and the spare US Navy FLTSATCOMs. USRP is way too expensive at the moment (they cost more than a cheap netbook) so that one's out.

so you want to know who's lobbying?

So I was moaning about the Government and the release of lists of meetings with external organisations. Well, what about some action? I've written a scraper that aggregates all the existing data and sticks it in a sinister database. At the moment, the Cabinet Office, DEFRA, and the Scottish Office have coughed up the files and are all included. I'm going to add more departments as they become available. Scraperwiki seems to be a bit sporky this evening; the whole thing has run to completion, although for some reason you can't see all the data, and I've added the link to the UK Open Government Licence twice without it being saved.

A couple of technical points: to start with, I'd like to thank this guy who wrote an alternative to Python's csv module's wonderful DictReader class. DictReader is lovely because it lets you open a CSV (or indeed anything-separated value) file and keep the rows of data linked to their column headers as python dictionaries. Unfortunately, it won't handle Unicode or anything except UTF-8. Which is a problem if you're Chinese, or as it happens, if you want to read documents produced by Windows users, as they tend to use Really Strange characters for trivial things like apostrophes (\x92, can you believe it?). This, however, will process whatever encoding you give it and will still give you dictionaries. Thanks!

I also discovered something fun about ScraperWiki itself. It's surprisingly clever under the bonnet - I was aware of various smart things with User Mode Linux and heavy parallelisation going on, and I recall Julian Todd talking about his plans to design a new scaling architecture based on lots of SQLite databases in RAM as read-slaves. Anyway, I had kept some URIs in a list, which I was then planning to loop through, retrieving the data and processing it. One of the URIs, DEFRA's, ended like so: oct2010.csv.

Obviously, I liked the idea of generating the filename programmatically, in the expectation of future releases of data. For some reason, though, the parsing kept failing as soon as it got to the DEFRA page. Weirdly, what was happening was that the parser would run into a chunk of HTML and, obviously enough, choke. But there was no HTML. Bizarre. Eventually I thought to look in the Scraperwiki debugger's Sources tab. To my considerable surprise, all the URIs were being loaded at once, in parallel, before the processing of the first file began. This was entirely different from the flow of control in my program, and as a result, the filename was not generated before the HTTP request was issued. DEFRA was 404ing, and because the csv module takes a file object rather than a string, I was using urllib.urlretrieve() rather than urlopen() or scraperwiki.scrape(). Hence the HTML.

So, Scraperwiki does a silent optimisation and loads all your data sources in parallel on startup. Quite cool, but I have to say that some documentation of this feature might be nice, as multithreading is usually meant to be voluntary:-)

TODO, meanwhile: at the moment, all the organisations that take part in a given meeting are lumped together. I want to break them out, to facilitate counting the heaviest lobbyists and feeding visualisation tools. Also, I'd like to clean up the "Purpose of meeting" field so as to be able to do the same for subject matter.

Update: Slight return. Fixed the unique keying requirement by creating a unique meeting id.

Update Update: Would anyone prefer if the data output schema was link-oriented rather than event-oriented? At the moment it preserves the underlying structure of the data releases, which have one row for each meeting. It might be better, when I come to expand the Name of External Org field, to have a row per relationship, i.e. edge in the network. This would help a lot with visualisation. In that case, I'd create a non-unique meeting identifier to make it possible to recreate the meetings by grouping on that key, and instead have a unique constraint on an identifier for each link.

Update Update Update: So I made one.

oiling the steel to sharpen the blade to shave the yak

Progress update on fixing the Vfeed.

Dubai Airport has done something awful to their Web site; where once flights were organised in table rows with class names like "data-row2", now, exactly half the flights are like that, they've been split between separate arrival, departure, and cargo-only pages, they only show the latest dozen or so movements each, and the rows that aren't "data-row2" don't have any class attributes but random HTML colours.

And the airline names have disappeared, replaced by their logos as GIFs. Unhelpful, but then, why should they want to help me?

Anyway, I've solved the parsing issue with following horrible hack.
output = [[td.string or td.img["src"] for td in tr.findAll(True) if td.string or td.img] for tr in soup.findAll('tr', bgcolor=lambda(value): value == 'White' or value == '#F7F7DE')]

As it happened, I later realised I didn't need to bother grabbing the logo filenames in order to extract airline identifiers from them, so the td.img["src"] bit can be dropped.

But it looks like I'm going to need to do the lookup from ICAO or IATA identifiers to airline names, which is necessary to avoid having to remake the whitelist and the database and the stats script, myself. Fortunately, there's a list on wikipedia. The good news is that I've come up with a way of differentiating the ICAO and the IATA names in the flight numbers. ICAOs are always three alphabetical characters; IATAs are two alphanumeric characters, which aren't necessarily globally unique. In a flight number, they can be followed by a number of variable length.

But if the third character in the flight number is a digit, the first two must be an IATA identifier; if a string, it must be an ICAO identifier.

links, the light alternative to writing

Ill-coordinated links. Great news in RepRapping - South Korean scientists have succeeded in getting bacteria to make polylactic acid. PLA is the RepRap project's favourite feedstock because it's a reasonably tractable, general purpose plastic that can be synthesised from starch. The synthesis is not exactly simple, which is why outsourcing the job to germs is interesting. As the kit of parts now costs about £395, I really ought to get started with one of these. Now there's a Christmas present for you. "Engineered bacteria not included." MUM! YOU FORGOT THE GERMS!

The uranium-enrichment deal with Iran is still on, but they are looking for stronger guarantees of getting the promised fuel for their research reactor. I reckon this is going to come down to the exact number of kilos that leave at a time, and therefore to a fine judgment about the efficiency of their centrifuges.

Spencer Ackerman mourns a great Mod shop. I remember that Klass Clothing in Leeds was about the first business of any kind in town to have a Web site, apart from these guys for obvious reasons. That's gone, as is Sam Walker in Covent Garden...and possibly even the SL1200!

twitbook: book of twits

Wired reviews a book on the media of the Middle East, The Media Relations Department of Hizbollah Wishes You a Happy Birthday. Well, even pirates have press spokesmen these days. It sounds like it could be interesting, but it strikes me that this piece by Tom Griffin about trolls sponsored by various Middle Eastern actors is its critical, rebellious twin.

The GLORIA Center at IDC gathered about thirty Israeli bloggers and members of Israel’s foreign and defense ministries for an informal gathering to evaluate the blogging effort during the Gaza war, new techniques and future challenges. Topics discussed included lessons of the Gaza battle for blogalogical warfare, live-blogging, new technologies and interactions with government. Bloggers delivered short presentations on their personal experiences and discussed future plans for cooperation....

Who wouldn't want to be a fly on the wall? It practically glows with a radioactive mixture of trollishness, self-righteousness, and raging, thinktank/intern ambition. A weaponised version of MessageSpace. You'll laugh; you'll cry; you'll read up on freeze-distilling your own hydrogen peroxide to escape all this hideousness!

As always, if you want a practical policy recommendation, make tools. A little investment in annoying javascript thingies pays off hugely by improving the productivity of your trolls; and it doesn't have to be technically very interesting.

In Italy, meanwhile, they've got a truly impressive legislation tracker going.

It allows one to follow an act in its path across the two perfectly symmetrical chambers (La Camera and Il Senato), from its presentation as a proposal, to its final approval.

It tracks all the votations, highlighting rebel voters. It tracks who presented an act, and wether as a first-signer or a co-signer. It also tracks speeches of officials on given acts.

Access to textual documents related to an act is easy and documents can be emended by users online, using an innovative shared comments system (eMend), that allows discussions on a particular act to take place.

Users can describe the acts, using their own words, in a wiki subsystem, acts are ratable and commentable, too.

All acts are tagged with consistent arguments by an editorial board, and that allows to know what’s going on and who’s doing what in relation to a subject.

An event-handling subsystem allows the generation of news. Whenever an act is presented, it moves towards approval or refusal, a votation takes place, someone gives a speech or anything worth noticing happens, news are generated. A dedicated web page and a customized daily e-mail, containing just the news related to those acts, politicians or arguments monitored by the user, allows him/her to follow almost in real time what’s going on.

Pretty cool; better than anything we've got. And, I think, that's much more a piece of real citizen technology than any of the TwitBook propaganda apps, which are all about creating a sense of participation; possibly, they actually exist in order to provide that sense as a substitute for real participation, in order to prevent it.

If that's not hardcore enough for you, the Make blog has a HOWTO on listening to satellites.

free our bills: hardcore wonk/geek out

So what do we need to know about a parliamentary bill?

First of all, as soon as a piece of legislation is published, it has certain meta-data. Date originated; originating department; originating MP; originating house; type - primary legislation, order in council, statutory instrument; current status (pre-legislative/Green/White Paper, first reading, committee, report, second, third, Royal Assent, repealed/superseded). And, of course, a unique identifier. But they aren't isolated; they amend, supersede, or repeal other legislation, so every Bill object needs to keep this information as well.

And if it's secondary legislation, it has dependencies on at least one past Act of Parliament, so anything with the types order-in-council or statutory instrument has to track which Acts it inherits from. Similarly, a primary Bill may create possible secondary legislation.

Now we need to look at the revisions. Once the bill is published, it starts to attract changes; but it remains the same bill. So we need to have further rows which are permanently associated with the original bill, but uniquely identifiable in themselves. It's probably simplest to keep only the changes at each step, because much of the point of the whole project is to monitor the changes. It feels right to me, if nothing else, to consider all the texts of a bill to be revisions, contained within the bill wrapper.

So a revision contains the title, the text in its sections, the status of the text, the originating organisation, if possible the originating MPs, the timestamp, and the amend/supersede/repeal/inherit information, and a revision ID. At each revision stage, a new item is added, until the final version gets Royal Assent; it would make sense to sort them in reverse chronological order and make the most recent version the default that is retrieved when that bill is requested.

This gives us a reasonable database of legislation, but it's not going to be much use; for that we need some more comprehensible semantics. So each bill needs both a summary and some category tags, and both the bills and revisions will need to have users specify their own tags and notes. Add those fields as well... And we'll need links to the debates at each stage, as well. Chuck in a URI field for Hansard in each Revision.

Summing up in object oriented terms, we've got a class called Bill, which has instance methods for the various metadata we've described, and a subclass called Revision, whose instance methods provide all the fields for each revision, but which always inherits the metadata and unique identifier of the Bill that created it, and possibly a further subclass of Revision called Comment to contain user notes. Further, the Bill needs a method Amend that creates a new Revision with the amending text, which remains provisional (inheriting the amending Bill's current status) until the amending Bill is finalised. Of course, if we implemented it in something like Django the code could be precisely that.

In database terms, each Bill is a row with a primary key that uniquely identifies the bill and all its revisions and comments; each Revision and Comment is a row which has the same key as its parent Bill and a key which identifies it in the context of that Bill.

Update: Comments point out that a Comment shouldn't be a subclass of Bill, for because it's not legislation itself and it should be an is-a relationship not a has-a relationship. Good point; actually, commentary should probably be logically parallel to the actual text of legislation, but related to it - Commentary, with subclass UserComment, linked by the bill and revision IDs to the actual text.

And Dsquared tells us that the German Bundestag already has a public version control system for legislation! Here it is; it's very complete and logical, I'll say that for it, but there is no facility to annotate anything. But if you want to know precisely what the Baden-Württemburg delegation wanted to change in the law on modernisation of accounting requirements in the Federal Council's Committee stage, it gets you there in two clicks from the search page. User experience design does not mean making things pretty.

don't leave me hanging on the telephone

An idea, seeing as no-one is very interested in ORGANISE and it looks like I'll have to learn erlang to make any impact on it.

Observation 1: The price of voice telephony is falling fast. Mobile operators provide some truly huge bundles of minutes, and there's Skype and Co.
Observation 2: Political campaigns of all kinds often need either outbound or inbound phone banks.
Observation 3: Asterisk rocks.

Conclusion: Wouldn't a distributed phone bank, based on Asterisk's AGI interface, be cool?

You could: Register volunteers and their availability. Create a campaign. Send talking points to participants as they become available. Dial them up, then dial the target number, and bridge them in. Log the results of the call.

You could also use it for inbound calls - for example, to take statements after a G20-like event, to provide advice, to register participants. And you could initiate and route calls intelligently, for example, to put callers through to people near them, or to send notifications to groups of volunteers.

Anyone interested? I raised this on the MySociety list and we've been discussing use cases.

firefox

Whining about Firefox crashes. Here's one day last week:

Start 0930
1515 - 5 groups, 111 tabs. Pressed page down key; CRASH. Resume successful.
1538 - Hang. RAM usage peaks at 66%, CPU 1 goes to 100%
1541 - Running, very slowly. Resource utilisation still very high
1542 - Hang
1550 - Cache cleared, normal ops resumed
1813 - Hang. RAM goes from 23% to 40%, CPU 1 to 100%
1828 - Memory leak - top shows RAM usage at 55%, but the system is using the swapfile heavily
1845 - Memory leak - RAM 84%. CPU 1 2%
1910 - RAM usage down to 63%, but still crappy
1917 - Memory leak - RAM 80%, CPU 1 3%

Next day:
1110 - Hang. CPU 1 103%, RAM 44.2%
1115 - Hang. Firefox process killed from command line. Fails to launch, "existing process already running" error message. Kill -9 from command line.
1335 - Hang CPU 1 99%, RAM 33%
1337 - Wow; it's recovered to tolerably normal functioning.

I agree 111 tabs is a lot, but you can see why I'm pissed off.

ORGANISE: a reappraisal

What with things like this, the LibDems' drive to collect members of the scientific-technical intelligentsia at a secret location in Kazakhstan (surely it must be a....), it's surely time to shake up my ORGANISE project - a messaging and task scheduling system for organisations of all kinds that implements a Stafford Beer viable system approach. (Unofficially, "like a bulletin board that actually encourages people to do something".)

Before Christmas, we'd gone through a couple of iterations of the spec, I'd vaguely decided to implement it as a Django web application, and I'd written a first version of the models.py (the file that defines the database schema in Django) that (I hope) embodies Duane Griffin's design of the data model. I also managed to discuss it at some length with Chris "Chris" Williams...

Now, it's unlikely to get anywhere by the 17th of March, so Rewired State is out as far as our methodology of conference-driven development goes. But I want to shake the thing up, and I'm increasingly interested in alternative routes; specifically, should it perhaps be more explicitly about messaging/communications rather than being a read-heavy Web site?

I've been reading up on XMPP messaging, and specifically on its publish-subscribe protocol. Each group in an ORGANISE instance would look like a virtual user. This would make sense - the defining characteristic of membership is the right to send to that group, after all. Percolation rules would determine whether a message from a group would be extended into the next biggest group; thematic groups would work in a similar way. In fact, the XMPP Standards Foundation has the notion of "collection nodes" in XEP 0248 which look very much like ORGANISE groups.

Among other advantages - federation with a lot of IM communities and social networks, the ability to interact with the ORGANISE server via other XMPP networks (which could be handy), a generally more real-time system, and the possibility of interconnecting different ORGANISations. And, y'know, it *feels* more appropriate for various reasons summarised here. Unfortunately it looks like 0248 isn't well supported yet...but what say you?

brrrains!

Suddenly, an awful wet crunching and groaning and sick heavy breathing. It's...huge...festooned in the rags of a once-respectable suit, waving a bladeserver torn from a rack like a child's toy...dripping with stale blood. No! The NHS IT Zombie has escaped, and it's fortified itself by eating BT's brains. Now it's coming for us. DAATA! it groans. WAAAANT YOUR DAATA! Run!

Seriously; BT has recently had to spook the stock market by warning of a huge hit to profits from its Global Services big-IT division. But reading this FT story carefully, it seems that a lot of that or maybe even the whole thing is down to the NHS National Programme for IT, and specifically the London Region patient management contract. (The other bits are the ones that haven't gone to ratshit yet.)

The regional patient-management segments were always the most challenging bits of the NHS NPfIT; partly this was natural, because their function - a workflow, documentation, and management information system for the entirety of a major hospital's operations - was by far the most complex in the project. The NHS National Network is a big VPN; the Spine needs to authenticate users, validate input, write to the DB, synchronise, and retrieve; but the patient management system needs to deal with all the possible pathways patients take through the hospital.

Partly, however, it is unnatural and caused by the politics of the project. The regions don't actually correspond to any organisational entity in the NHS - they exist only for the IT project. They therefore have to replace existing systems that vary widely inside each region and cope with organisations in different chains of command. And each region was originally meant to be implemented by a different company; now, most of them have either given up or gone bust, and BT is doing much more work than previously planned, and this of course means that it has to deal with radically varying solutions already installed.

Worst of all, though, the regions mainly exist because the Government wanted to have the job done by the Big Consultancies - Accenture, EDS, and friends - that it was used to dealing with. Assuming that they wouldn't be interested in small contracts, the Government invented a completely new organisational level in order to sweeten the deal. They further insisted on the contracts being covered by intense secrecy, which cut off any possibility of talking to the users. And the Big Consultants proceeded to move the actual development to the US and India to save money, thus avoiding any institutional knowledge that might somehow have seeped in.

Now, it looks like BT is planning to offer a "more tailored service" to the hospitals - which sounds a lot like "doing the requirements exercise we should have done back in 2001". Of course, it's going to cost money and nobody knows how much yet, but I suppose it's progress, especially as the sacking of Fujitsu from the project means that it looks more and more like a BT job (London, the ex-Fujitsu South, the national projects, and perhaps more besides).

But it's still not too late to take radical action. Part of the original plan involved using a common data exchange standard for the whole NHS; if this exists, there's no need for much of the rest, especially not the regions and possibly not the Spine. We could define some goals and a set of data formats, then break out the cash to the individual hospitals, trusts etc to use themselves. In fact, when various US, Australian and Finnish hospital sysadmins tried that, they came up with the best healthcare IT system yet. The problem with the NHS NPfIT is quite simply that it didn't listen to the bureaucrats.

Which is why this is sense. I have no idea what such a sensible set of ideas is doing in George Osborne's in-tray, and I suspect the Tories may think it's a way of preventing IT development in the public sector. But I think a cross-government requirement for common data standards, as much open source as possible, and perhaps even building everything with a sensible API for further development would do nothing but good. And perhaps the project cap might help - after all, the way to deal with zombies is to destroy the brain.

be your own Stanislas Petrov

Remember this post? Well, Geoff Forden at Arms Control Wonk has a brilliant series on the system that Stanislav Petrov was monitoring, how it worked, what went wrong, and how to draw conclusions from pictures of a missile launch with Google Earth. And why you should worry now.

wikileaks in a jam

Vexation about the publication on Wikileaks of some US Army documents with details of the counter-IED radio jammers. Well, you can see why they're concerned; but I very much doubt this is particularly important.

Recap: the New-Old Iraqi Army was in the habit of using command-detonated IEDs to blow up Coalition and Iraqi government road convoys. To begin with, the command element was often either a GSM device or else some sort of el cheapo radio device like a garage-door opener, RF thermostat, bits and pieces from an industrial process-control rig or the like. After much spending and much fuss, the US Department of Defense deployed "secret" but much hyped jammers on the lead vehicles in the convoys.

Now, there was almost certainly no reason to spend anywhere as much as they did. This is directly linked to the non-fuss about Wikileaks. The devices we have just mentioned have an internationally-standardised frequency band to chatter away in - the so-called Industrial Scientific Medical band, which is unlicensed spectrum - anyone can use it for anything, so long as they don't use too much power. Among other things, all the world's WLAN access points work in the ISM 2.4GHz band, as do wireless hi-fi speakers, baby monitors, cheap CCTV cams, etc, etc. So right back in 2003, it was blindingly obvious which frequencies were involved and what an upper bound on the power output would be. Which made the problem of jamming it pretty simple - just hammer away in the ISM with noise at a significantly higher Tx wattage.

Radio waves are electromagnetic radiation, and therefore their intensity changes with the inverse square of the distance from the source. So you could trivially calculate how much power you need to trigger the device a given distance away from the target. All you need is something that will radiate in the ISM band on command, like...a WLAN card, which now costs about five quid (or, perhaps, a door opener with a better antenna...). I have to say, I suspect that Donald Rumsfeld got played terribly over this. And, of course, nothing radio-frequency stays secret once you start transmitting; everyone can hear you.

There are cleverer things you can do; regarding the GSM ones, you could carry a malicious base station around with you, and therefore blackhole all traffic to and from phones in range. Or you could tap the phones and find out whodunnit (we know the other side do it to us). If I was really serious about this, I'd use one of these, which can be programmed to emulate pretty much anything radio.

So, like so much government secrecy, this is much more to do with security from embarrassment (we spent $billions on technology that would have been cutting edge in 1940!) than security from anything else.

HOWTO spoof the Pakistani Foreign Ministry

Reader Chris "Chris" Williams recently came up with an interesting idea regarding the unnerving incident during the Mumbai terrorist assault when somebody called Mr 10%'s office and, posing as the Indian foreign minister, threatened war. Chris reckons that not only are the two linked, but the phone call was actually the main effort; in fact, the commando speedboat assault on the city centre was a sort of privilege-escalation attack intended to boost the effectiveness of the fake telephone call.

As Scott Sagan wrote in The Command and Control of Strategic Nuclear Forces, you can't go on alert without reducing the reaction times and increasing the sensitivity of your command system, because that's what the word "alert" means. Going on alert involves system risk as well as mitigating the risk of surprise attack.

That doesn't just work for radars and satellites, or even for organisations; it happens in your head as well. As well as forcing an Indian military alert and, more importantly, a Pakistani alert, the attack on Mumbai forced an emotional, psychological, and biochemical alert on both parties. Among other things, this tends to mean that even peaceful precautions are overlooked; just pick up the phone!

What is really worrying here, though, is that the Pakistanis still apparently believe that the call came from the Indian foreign ministry, because the Caller ID display said so. This is truly frightening; caller-line identification, CLI, is a surprisingly flaky feature of the public-switched telephone network (see the many, many discussions on uk.telecom). It's the kind of thing that Bellheads like to accuse Internet people of, just it's the Bellheads' work. Rather, CLI works quite well within one telephone network; it's when you start interworking that things happen.

And obviously, international calls involve interworking. CLI works like this (there's a nice explanation on the Asterisk developers' list: a message including the e164 telephone number originating the call and two flags is generated either by the subscriber's equipment or by the carrier's local exchange. The first flag has a value of 0, 1, or 2, which correspond to OK, "Not available due to user action", and "Not available due to interworking". If you choose to withhold your number, BT will set a flag of 1 on the CLI, so standards-compliant equipment at the far end will not show it. Note that the number is still sent. 2 is sent if the carrier (or the user) doesn't have a valid CLI for this call, or doesn't send CLI to the terminating network.

The second flag can be 0, 1, 2, or 3. 0 means the CLI was passed by the originator's equipment and this has not been checked. 1 means the CLI was passed by the originator's equipment and the network setting the flag verified the number is correct. 2 means it was checked and found to be incorrect. 3 means that the setting network's local exchange assigned the CLI and therefore it is known to be good.

In the case of calls that are coming in from another network, then, the first flag should be 0 only if the second is 1 or 3, else 2. It's left as a question of policy what you do with the others - for example, if someone is making unwanted calls to subscribers in a network that you provide service to, using the withhold flag and CLI set by their own PBX, do you honour the withhold flag, do you send the dodgy CLIs in case the subscribers want them to find out who's doing it, or do you treat the CLI as probably all lies?

In practice, this is resolved by keeping lists of networks by degrees of trust. After all, if someone is really dodgy, there's probably no point in relying on their own claim that they checked the CLI, so you might as well treat them all as nonsense. Some you can trust. Some you can trust partially, perhaps only the 1s and 3s.

But this, of course, only works if other people can trust you. If you don't care, you might just pass every little thing, or if you're really irresponsible you might set 1 flags on everything. Eventually this sort of behaviour will get you on everyone's Do Not Trust list, but it's always possible that the people you're trying to send dodgy CLIs to aren't great either.

So, first of all, you need an el cheapo phone company. Specifically, what you're after is someone who provides really cheap and flaky bulk SIP interconnection, because what we need is a way of connecting a device that's going to send someone else's CLI into the traditional phone network, and this is much the easiest via the Internet. Of course, our target might be on a VoIP system, but the important bit is that it turns up at the far end with a traditional phone number as the CLI. We might start here, or - why not? - even here, or of course here.

Now, all we need is a copy of Asterisk, the open-source PBX and general phone toolkit; we change the zapata.conf file to set the CLI to our desired number and whatever flags we like, set "usecallingpres=yes", put the details we were given by the dodgy SIP operator in the trunks section, and we're good to go. There's an obvious way to test if they're passing the arbitrary CLI; call your mobile phone and see what comes up on the screen. To finish the job, we tell Asterisk to route incoming calls from the mobile numbers (or VoIP clients) we want to use, according to the dialplan we just specified.

We can now call the president of Pakistan and say whatever we damn well like, from whereever we damn well like. And caller ID will say exactly what we want it to. You may stroke a white cat while you do it; it's not mandatory.

Scared yet? Right, we have learned a couple of things here. First up, interconnected networks imply netizenship. Keep your gear clean and know your customers, and you'll save people downstream a lot of trouble, and if everyone behaves like that, what a wonderful world it would be.

Second, Caller ID is no kind of security for anything vaguely serious. Telcos don't bother with it for really important purposes, like "working out your phone bill" - they use a different and secret billing identifier. It is truly astonishing that there was apparently neither any technical security beyond that, nor any authentication procedures either. (After all, an alternative to this would have been to have someone walk into the Foreign Ministry and just pick up the phone.) No security questions, no pre-arrangement, no passwords, no crypto, no shared secret. Nothing.

is there anything stupid going on at Tim Worstall's?

We've blogged before about the NHS's computer project. So I'm not at all happy about this remarkably silly post at Timmeh's. He takes issue with a post of Richard Murphy's about bank nationalisation:

Yup, the people who brought you the NHS Spine are to be put in charge of developing all banking software in Britain.

Well, this is a strawman to begin with. Is Murphy the Chancellor now? But let that pass. Really? A group of mostly American healthcare computing specialists? Several of which no longer exist? Or does he mean the big IT consulting firms involved - like IBM, BT Global Services, and Accenture? Because I'm pretty sure they do a hell of a lot of financial work as it stands; in fact, everyone was worrying last week about IBM's third quarter results precisely because banks are big customers. (They turned out to be OK, in that mysterious IBM way.*)

But perhaps he thinks the NHS NPfIT was developed by teh government bureaucrats? Or at least, he's willing to pretend it was to suit ideology? The whole problem with NPfIT, as we've said before, is that the system was developed completely in isolation from NHS bureaucrats or indeed anyone else who would have to use it. The NHS trust IT departments were kept well out of it. The upshot was that the developers knew literally nothing of the NHS's requirements, its business processes, or the data the system was meant to handle.

No wonder it was a disaster. In fact, when a group of US hospital bureaucrats had a go at designing a medical IT system, they came up with a beauty - there's even a satisfied customer in the comments. Why? Because they knew what it was meant to do and how. Compare this comment:

I met a guy who works for this company. I cannot repeat what he said, since he has a family to feed. But suffice to say he was deeply worried about the implications for safety of life. That was a few months ago.

The whole thing is rotten to the core, and desperately needs to be scrapped. Now.

The good news is that the thing still doesn't work well enough to turn it on even as a pilot project, so we're safe for a while yet. But what did happen the last time the Government took on a really challenging in-house IT project? You ask Daniel Davies.

(* probably something to do with asking the fucking users - that or the staple Nazi market, or wearing a lot of pale blue shirts.)

Organise

I've been reading the oohing and aahing about the Obama web operation (for example, here at the Linux Journal); all those individual pages and RSS feeds and iCals for every event. It strikes me that some of this can be generalised, and that we can probably improve on it. After all, as LJ points out, the difference between this and the Republican view of the Internet (Talk radio! With porn!), which also happens to be the Government view, is that it's amenable to individual activists and groups of activists doing things other than sending shitty chain e-mail to their relatives and shouting a lot.

But it still doesn't really allow for them to participate in the direction of the campaign; and it's a one-off tailormade job. If you want to change something with design and engineering, you've got to think mass production - or better, lean production, being able to quickly change the product and still rip them out as if they were standard stampings, and mass customisation, designing to let the users alter the product before and after they get it. Perhaps the crucial factor in this is modularity; you break it down into lumps subject to old-fashioned mass production and configure them as desired.

Further, it's quite common to have an Internet-enabled campaign that sends messages down from headquarters to the mob; not just the organisational model of the 20th century mass party, as originally invented by the Tories in the late 19th century, but even more so, as the party members always had a significant influence on policy and personnel, whether formally (like the Labour and Liberal parties) or informally (like the Tories pre-1965 and pre-1998). However, the "Labour Supporters' Network" (copyright - Zack Ecksley) and the world of nicely on-message duckspeakers around Iain Dale's blog have about as much influence on their party headquarters as a passing slug. Donal Blaney, for all he's the most contemptible arsewit (copy-pasting early 90s Clinton-murder smears? changing the world, Don!) on the Internet, has got the right end of the stick there. But he's still just a one-way blowhard.

What do you need to campaign? You need to know what is happening at the top level, in your rough region, and in your locality. You need to have a locality - to join a group or form one. You need to tell others when and where things will happen. And, I think, you need to be able to escalate things up the organisation. We already have functions like this for various rather crappy geek newssites, but what is important is that the members of the N19 group or the Fisheries special-interest group can break the point out into the London or the Economic or the Environment group, and they can break it out into the main broadcast to everyone - completing the loop.

If this reminds you of Stafford Beer, it's entirely deliberate. Information should percolate up as the members want it to, and "perk" ought to be a better word than "digg".

----------MEMBERS-----------------------------
|LOCALITY 1|LOCALITY 2|LOCALITY 3|
REGION 1 |REGION 2 |REGION 3
:THEME 2 :THEME 1 :THEME 3
|CAMPAIGN|
| = UNIQUE
: = NON-UNIQUE

We have pretty good standards for all the information exchange involved; RSS for the various local, regional, thematic, and main broadcast messages, iCal for calendar events, GeoRSS for messages with location content. SMS or MMS for mobile alerts. And the relationships involved are all ones that can go in a database schema. Members are subsets of the campaign (the campaign, better, is a superset of the membership); they are also part of groups. Messages and events are held in their originating table, until escalated into the next one up or across. By default, each member page has the local, regional and main broadcast feeds, and all the links you need to join or create other groups, start events, subscribe to them, set up alerts, and recommend anything for escalation. All groups, locations, etc create public and password-protected feeds. GeoRSS, with iCal enclosures, should be as MVC as it gets.

Whilst working on the Viktorfeed, I never quite grokked what the various Python Web frameworks (y'know - django, zope, cherrypy, webpy) were for. Now, however, I've actually bothered to read the Django documentation and it looks like the perfect solution. Essentially, it lets you build all your database tables in Python classes, set up all the views of the data you might want, and fit them in whatever HTML chrome you like, as well as creating RSS feeds of any view you can create of the shared data. (*I know I'm years behind the kool kidz here, but, well.)

I think it would be a cracking idea to have a deployable, pythonic, hackable platform for weird political action; with options for resource control, you might be able to use it to run almost anything.

dive! dive! dive!

Der Spiegel has an interesting story regarding the Colombian drug smugglers' homemade submarines. These have so far been considered a curiosity, but apparently they are becoming more and more common, and the technology is developing fast. The biggest vessel captured so far displaced 46 tonnes, presumably surfaced, with a payload of 10 tonnes. Apparently the current ones, described as the third generation, have a radius of action of 2,500 kilometres; an American admiral is quoted as expecting them to eventually start crossing the Atlantic from northeastern Brazil to West Africa, linking up with the emerging drugs route from there to Europe.

The ViktorFeed: Documentation

Here is the presentation I delivered at OpenTech 2008:

| View | Upload your own

I'd publish the text, but I didn't prepare a text:-)

Anyway, the ViktorFeed is a development of basic python scripts I've been using for some time to collect data on certain aircraft movements through Sharjah and Dubai Airports. Both of these place all movements on the Web, but neither of them provide anything like an RSS feed, which is why I began scripting, in order to save checking them myself. (You can read about this phase in the Political Pathetic Python posts on this blog.)

The current version works as follows: the web pages involved are loaded and BeautifulSoup instances created for each one. If a page fails to load and an IOError occurs, this stage is skipped for that one and a default message added. Data is extracted using BeautifulSoup's find method in list comprehensions. Each flight is represented by a tuple of values in a list. For each flight, the tuple is unpacked and each item in it assigned to a standard variable. If the airline name is found in a whitelist, the tuple is discarded. Otherwise, various standard items - for example, the name of the airport the flight arrived at or departed from - are added, the time variable is processed to provide both a readable time and a time in seconds since the epoch, and a database is queried to provide the geographical locations of the source and destination.

In the event that a location is not given or not found, a default value is specified and a message added. The default location is in the Bermuda Triangle, thanks to Soizick. The values are reassembled as a dictionary and appended to a list. When all pages have been processed, the content of this list is decorated with the time values in seconds since the origin, and sorted into reverse chronological order. This version is then undecorated, and the individual flights are used to create a Simple GeoRSS file through Python string formatting, which is encoded as Unicode and written out to disk.

Items in the file consist of the time and data group, in the title field, the source, destination, airline, and flight number in the description field, a GeoRSS Line tag with the source and destination geocodes, and the current time and date in the pubDate field. This data can be visualised in Google Maps more than simply. The test version was served from my laptop, using the SimpleHTTPRequestHandler and ForkingTCPServer methods and port forwarding.

Things to do: get it going on a permanent Web presence, refactor the code into a slightly less ugly mess, keep all the flights in a database, make it possible to query past movements.

OpenTech 2008: It's like the ESF, with code...

Back in 2004, this blog went to the European Social Forum - we weren't that impressed, but we did call it "the Caesar's Palace of Ranting". I'm not sure what the equivalent for the UKUUG's OpenTech 2008 would be; there was plenty of ranting, but a sight less committee wank, more practicality, even if no-one can answer the question of what any of this stuff stands for. I ran into, among others, Liz Henry, most of MySociety, the author of Spyblog (who has some damn good war stories), various readers including Duane Griffin, and a small galaxy of assorted hackers, militants, gawpers, freaks and mutants. Good People, as the Doctor would say.

And they are, too; even if the live demonstration of the ViktorFeed didn't happen due to the lack of a routable IP address (or even working connectivity for that matter), there was the loan of another laptop when OpenSUSE didn't want to speak to the projector. When I'd finished the show and dealt with all the questions, I was faced with at least two offers of colocated server capacity, and the services of at least three professional software developers, as well as an interview for the BBC World Service, a spare USB key, and a pint of lager. All of which would have come in handy the night before, when I foolishly attempted to change something in the code after midnight and borked the whole thing, forcing me to get up at six the next morning to fix it.

As it turns out, having met Francis Irving, I'm probably going to be assimilated by MySociety, or at least my project is. I was also very interested in some of the green/geek crossover projects - I missed the session on solar power and IT, but I did get to the AMEE presentation on their automated carbon dioxide profiler and Hotmapping's show of their IR surveying work, intended to classify buildings by the rate at which they lose heat. Apparently they'd already found one urban cannabis farm.

And BT Osmosoft's TiddlyWiki - a wiki in a single file - may not sound all that much; but I really liked the idea of a zoomable, pseudo 3D interface for wikis. I'm quite keen on the idea of using this to organise contacts - who puts their friends in alphabetical order after all?