HOWNOTTO build a computer system

Ziff-Davis Baseline carries a huge report on exactly why the NHS National Programme for IT is a disaster. (Thanks, Charles.)

First of all, a reiteration of a past point: they mention as one of the few successes the N3 tranche of the project, which turns out to be the deployment of DSL lines to GP surgeries and the national VPN-over-MPLS backbone. Well, if BT couldn't get that right, it would have been astonishing. Why it even needed a project is worth asking - why not give each doc £25 a month and tell them to get their idle legs over to Carphone Warehouse, then get the VPN client from the NHS website? (Answer: because a nontrivial proportion probably think computers are for secretaries.)

Anyway, it looks like Blair's inability to critically assess the statements of powerful actors has got us into another fine mess.

"The inspiration to digitize this far-flung bureaucracy first surfaced in late 2001, when Microsoft's Bill Gates paid a visit to British Prime Minister Tony Blair at No. 10 Downing St. The subject of the meeting, as reported by The Guardian, was what could be done to improve the National Health Service. At the time, much of the service was paper-based and severely lagging in its use of technology. A long-term review of NHS funding that was issued just before the Blair-Gates meeting had concluded: "The U.K. health service has a poor record on the use of information and communications technology—the result of many years of serious under-investment."

Coming off a landslide victory in the 2001 general election, Blair was eager to move Britain's health services out of technology's dark ages. Gates, who had come to England to tell the CEOs of the NHS trusts how to develop integrated systems that could enhance health care, was happy to point the way. "Blair was dazzled by what he saw as the success of Microsoft," says Black Sheep Research's Brampton. Their meeting gave rise to what would become the NPfIT."

Couldn't they have introduced him to Richard Stallman? But, as ever, the one eye was shining and we were all off on a happy crusade.

After a February 2002 meeting at 10 Downing St. chaired by Blair and attended by U.K. health-care and Treasury officials as well as Microsoft executives, the NPfIT program was launched.
In quick order, a unit was established to purchase and deliver I.T. systems centrally. To run the entire show, NHS tapped Richard Granger, a former Deloitte and Andersen management consultant. Granger signed on in October 2002 at close to $500,000 a year, making him the highest-paid civil servant in the U.K., according to The Guardian.

In one of his first acts, Granger commissioned the management consulting company McKinsey to do a study of the massive health-care system in England. Though the study was never published, it concluded, according to The Guardian, that no single existing vendor was big enough to act as prime contractor on the countrywide, multibillion-dollar initiative the NHS was proposing. Still, Granger wanted to attract global players to the project, which meant he needed to offer up sizable pieces of the overall effort as incentives.

[snip]

The process for selecting vendors began in the late fall of 2002. It was centralized and standardized, and was conducted, Brennan and others say, in great secrecy. To avoid negative publicity, NHS insisted that contractors not reveal any details about contracts, a May 2005 story in ComputerWeekly noted. As a byproduct of these hush-hush negotiations, front-line clinicians, except at the most senior levels, were largely excluded from the selection and early planning process, according to Brennan.

I've bolded the key failures here. First of all, letting the producer interest poison the well. Microsoft execs, eh? The big centralised-bureaucratic proprietary system vendor Microsoft was permitted to influence the whole process towards a big centralised-bureaucratic proprietary system from the very beginning. This occurred at a time when Health Secretary Alan Milburn was constantly railing against "producer interests" blocking his "modernising reforms". This was code for the trade unions that represented low-waged nurses and cleaners, and the British Medical Association that represented doctors. Can anyone spot the difference between the two groups of producer interests? One of these things is not like the other..

The managerialists inevitably called on a management consultant to run the show - as we all know, we are living in a new world, and the status quo is not an option, so nobody who actually knew anything about the NHS, hospitals, or for that matter computers could be considered. (Granger failed his CS degree.) With equal inevitability, he called on management consultants to tell him what to do. The great global consulting firm McKinsey duly concluded that only great, global consulting firms could do the job.

Choosing which ones was clearly a job only central authority could undertake, and the intervention of the press, the unions, competitors or elected representatives would only get in the way, so the whole thing vanished behind a cloud of secrecy. Secrecy enhances power. It does this by exclusion. The groups excluded included the doctors, nurses, technicians and administrators of the NHS - which means that the canonical mistake, the original sin of systems design was predetermined before the first requirements document was drawn up or the first line of code written. Secrecy specifically excluded the end users from the design process. There are two kinds of technologies - the ones that benefit the end-user directly, and the ones that are designed by people who think they know what they want. They can also be described as the ones that succeed and the ones that fail. Ignore the users, and you're heading for Lysenkoism.

Among the "problems" of the NHS system was that most hospitals had their own computer systems, developed either by small IT firms or in-house. The contracts stated that each of the five new regional service providers and the "spine" (BT) would have to replace them, design a single regional system, but also maintain "common standards" nationally. The sharp will spot the contradiction. If you have common standards for information exchange, why can't you have them within the region as well as between regions? Why do you need the regional system at all? Why do you need the big global consulting firm - standards, after all, are for everyone, from Google to the hobby programmer cranking out a few lines of Python or such. In fact, almost all developments in computing in the last 10 years have been in the direction of separating levels of abstraction. It doesn't matter if the web server runs Linux and the database Windows Server if they both speak XML at the application layer.

This was actually recognised for some purposes. The NHS bought 900,000 desktop licences for MS Windows and further commissioned Microsoft to develop a common interface for the NPfIT, thus ensuring that any common interface would be proprietary and unalterable except by Microsoft. But no-one seems to have thought through the implications of common standards. Instead, the contracts specified that the old systems must be torn out and the data transferred to the new, thus adding a huge sysadmin nightmare to the costs.

Trying to keep down the costs, iSoft outsourced the development to India. But the Thomas Friedman dream of hordes of crack coders as cheap as chips showed some flaws - specifically:

the programmers, systems developers and architects involved didn't comprehend some of the terminology used by the British health system and, more important, how the system actually operated, the CfH conceded.

Neither did IDX's developers working with Microsoft in Seattle know anything about the NHS. This choice, like the secrecy, ensured that no NHS institutional memory would be available to the developers. So, 100 medics were shipped off to the coder farm to explain. Naturally, this effort to fix fundamental architecture problems by tinkering just added complexity and cost, as Pareto's theory of the second best bit. Eventually, one of the regional systems contractors decided to take iSoft's off-the-shelf product and hack it into something vaguely suitable, and another walked away. IDX and GE Healthcare's product was so dire that even BT couldn't make more than one implementation work in two and a half years, and then sacked them.

But, there is no sign any of this will affect policy whatsoever. Instead, the managers content themselves with intermediate statistical targets (apparently they are installing 600 N3 lines a month, a rather poor performance for any normal ISP), rigged definitions (the deal with Microsoft is said to have saved £1.5 billion - compared to what? certainly not open-source..) and bully rhetoric about feeding the slower huskies to the faster ones (I am not joking). The inevitable signs of failure, meanwhile, emerge - it doesn't work.

"As an example, in July, mission-critical computer services such as patient administration systems, holding millions of patient records being provided by the CSC alliance across the Northwest and West Midlands region, were disrupted because of a network equipment failure, according to the CfH. As a result, some 80 trusts in the region were unable to access patient records stored at what was supposed to be either a foolproof data center or a disaster recovery facility with a full backup system. Every NPfIT system in the area was down for three days or longer. Service was fully restored and no patient data was lost, the CfH says.

That was not the first such failure. In fact, in the past five months more than 110 major incident failures having to do with NHS systems and the network have been reported to the CfH, according to ComputerWeekly."

But, of course, the users are lying and everything is wonderful.

"The CfH responded in an e-mail to Baseline: "It is easy to misinterpret the expression 'major incident.' Some of these could have been, for example, individual users experiencing "slow running." We encourage reporting of incidents, and we are open and transparent about service availability levels, which we publish on our Web site."

Perhaps they'll put the chocolate ration up there too. But guess who is driving the march into the marshes?

Still, for every setback, Granger, CfH and Tony Blair's Labour Government announce a step forward. Blair, in fact, is CfH's biggest ally. Addressing some 80 senior doctors earlier this year earlier and, according to The London Times, sweating profusely under the bright lights, Blair said, "The truth is that we have now reached crunch point where the process of transition from the old system to a new way of work in the NHS is taking place. Each reform was in its time opposed. Each is now considered the norm. The lesson, especially at the point of difficulty, is if it's right, do it. In fact, do more of it."

I remember thinking, when I first heard of the project, that Palm had just confessed to a huge stockpile of unsold PDAs in a warehouse in Long Beach, and that we ought to buy the lot at firesale prices and turn loose the programmers at local level, with a common data exchange standard. Standards, not standardisation, as David Berlind says.

Oh yes, guess who's still running the world?

Update: This post has been approvingly linked by the Adam Smith Institute's blog, which positively scares me. But I think it's worth pointing something out here, which is that this story is not really about planning versus markets or private versus public. The Government brought plenty of stupidity to the table, but so did the Big Consultants, and so did little iSoft. Commercial motives led to as much stupidity as planning did. Very likely, had there been 10 more bidders for the Regional Service Provider contracts and therefore more competition, the same institutional factors would have entrained the same stupidity.