The slow death of your family history

As I embark on the building of the Stevenson family tree, Sarah Williams, editor of Who Do You Think You Are, issues a warning. “It’s quite addictive,” she said. “It’s like a serious eBay habit, but you know, instead of buying yet more pots… [you’re] buying ancestors, collecting them.”

This proves to be true. Over the following weeks, my family tree at Ancestry.co.uk takes ever more elaborate shape as I pluck great aunts and uncles from obscurity, and pore over birth indices and military records. It replaces Twitter; faced with five spare minutes, I find myself shoring up the particulars of my great-great grandfather or despairing at the Stevensons’ frustrating habit of using Sydney as a middle name, but not being able to decide whether to spell it with a Y or an I.

Irregular names are far from the only frustration for amateur genealogists, and certainly not the most irritating. Despite the best part of £10 million being spent on digitising Britain’s vast archives of birth, death and marriage certificates, they remain offline more than a decade after the project began.

Instead, you’re forced to rely on expensive private archives or copies sent from paper archives. How has Britain got to 2016 with its family history trapped on paper and microfiche?

Fractured families

The complexity of building a decent family tree soon becomes clear. Discovering that an ancestor existed is simple; indices for birth, death and marriage certificates are easy to find. However, the indices are far from fleshed-out. The birth index gives you a person’s birth name, place and, after 1911, their mother’s maiden name, but nothing else. Even the precise date of an event is missing – you’re left with a three-month range. Likewise, a marriage index provides the names of the couple and venue, but details such as their parents and occupations are saved for the official marriage certificate.

These certificates – the most solid, useful evidence about our ancestors – are available, but the process by which you obtain them is old-fashioned. Birth, death or marriage certificates can only be ordered from the General Register Office (GRO) – part of the Passport Office. You’ll need the details from the relevant index so that GRO can locate the record, and then, £9.25 and a week later, you’ll receive a copy of the certificate. I work out that a single certificate for each of my relatives would set me back almost £200.

Instantly searchable, transcribed records wouldn’t only make researching family trees much easier, they’d also bring wider benefits. Professor Chris Dibben is the principal investigator at Digitising Scotland, a vast project designed to process 14 million birth records, four million marriage records and 11 million death records between 1855 and 1973. “The arguments we made to the funder [the Economic and Social Research Council] was for the research potential of this data,” he told me. “It could be used to facilitate social science and health research in the future.”

Records can help researchers and governments understand patterns of disease and social mobility, with lessons that can be applied globally. Scotland’s records cover a period of vast industrialisation and air pollution. “In many industrialising countries around the world – for example, in India or China – their populations are being exposed to air pollution,” Dibben noted. “They don’t know what’s going to be the impact on their populations in 20 or 30 years’ time, but with this historical Scottish data we can look forward.”

The cost of the Digitising Scotland project is surprisingly modest – not including the scanning of documents, the project’s budget was only £2.5 million. And, although the project is designed to benefit researchers rather than individuals, Dibben says there is a crossover. “What’s of interest to the genealogical public is also of key interest to health and demography research.”

Broken records

“There’s a lot of envy for Scotland,” admits Williams. In the rest of the UK, the practice of dispatching a government worker to dig up paper certificates, or generate copies from microfiche, doesn’t seem to have an end in sight. People have tried to modernise the system but failed, and at some considerable cost. In 2005, the Office of National Statistics awarded a contract to Siemens to digitise 250 million birth, death and marriage records. Dubbed DoVE (Digitisation of Vital Events), the project planned to scan 80,000 microfilms. The resultant data would be encrypted and wired to Chennai, India, where an army of 800 employees would set to work transcribing certificates dating back to 1837. Siemens proudly announced it expected to complete the work by 2008.

But by 2008, Siemens had scanned only half of the records available and was duly paid £8.33 million for the work it completed, according to an ONS response to a Freedom of Information request. A year later, the government renamed the project the Digitisation and Indexing project. It never progressed beyond “pre-procurement”; it was suspended in 2011.

Finding a precise reason for the project’s failure proved tricky. It seems likely that Siemens underestimated the complexity – and cost – of the project. The Passport Office estimated that the scanning and transcription of the remaining 130 million records, plus the creation of a system to access them, would cost up to £30 million.

A source with knowledge of the industry at the time – but not of the DoVE project specifically – speculated that DoVE’s failure could have been the joint result of Siemens over-promising and the government failing to appreciate the complexity of the project.

Jon Elliott of the Archive and Records Association recounts a story of an MP who believed record digitisation could be achieved with nothing more sophisticated than a smartphone. “People who don’t know the issues make an assumption that it should be simple to just take a picture,” he said. “Doing it on a national scale, or even on a regional, local scale, brings with it all kinds of legal obligations, data-protection obligations and search-ability obligations that won’t apply at home.”

Simply scanning documents is daunting enough, but Williams tells me it’s only half the story. Without transcription, all a digitisation project achieves is several hundred million images that can’t be searched by content. The UK’s plan was to outsource to India, an approach Digitising Scotland’s Chris Dibben says can work well, but public liaison officer Jon Elliot sounds a note of caution. Transcription is “obviously very tricky,” he said. “It’s not simply some mechanical motor task – it requires interpretation. Also, the further back you go, the more likely you are to have records in different languages.”

The GRO has said it recognises “the potential benefits for our customers and stakeholders having online access to civil registration records,” and told Alphr in a statement that it “continues to explore options for modernisation of record access” – but there appears to be nothing even faintly resembling a concrete plan.

It’s a lamentable situation, according to Elliott, who claims digitisation is a “very low” priority for the British government. Apart from the benefits for individuals and scientists, he says a well-digitised archive of public records could have enormous benefits for society. “If you look at issues such as Hillsborough, or the scandals over the Foreign Office misplacing records from the Kenyan insurgency [documents detailing British abuse of Mau Mau insurgents were deliberately destroyed], we’re finding ourselves in an age where public inquiries and public scrutiny of previous government decisions has never been higher.

“People tend to see archives and records in that family history box,” he added, “but increasingly, we’ve seen movement in the mainstream of public accountability.”

Private data?

If government can’t complete record digitisation, the private sector seems willing to step in. Since the GRO’s digitisation efforts have stalled, Williams says users are starting to bypass their records. “They’re getting a subscription to Ancestry or Findmypast, and finding there are enough records becoming digitised.”

The London Metropolitan Archives is a collection of more than 20 million parish records – if an ancestor was married in a London church, there may be no need to pay for an official GRO marriage certificate. “The information the church takes and the certificate you get is exactly the same as the civil registration one,” said Williams.

Another example is a register taken in 1939. This de facto census was designed to help co-ordinate the war effort and resulted in 1.2 million pages detailing the lives of 30 million people. A deal between genealogy site Findmypast and The National Archives created a digitised, transcribed, searchable database made all the more important by the lack of official censuses in 1941 (war) and 1931 (destroyed by fire).

The problem is the cost of accessing these private stores. To search the 1939 register at Findmypast requires signing up for a year’s membership from £120, or paying £6.95 to unlock each household’s records. Ancestry’s Premium subscription also costs £120 a year. If your ancestors emigrated to the UK, you’ll need the £180 worldwide membership.

Williams says that most of their readers understand the cost of digitising records. Plus, the only other way to get hold of historical archives is to traipse across the country. “You used to have to go to every individual archive to research your family. It was expensive and time-consuming.”

However, while privatisation may keep hobbyists happy, there’s a risk that only the most commercially viable records will be indexed. “We want to avoid a two-tier approach, where the stuff that people can make money out of is digitised, and the stuff that’s still enormously valuable and in the public good… is not,” said Elliott.

As for the question of physical security, experts agree that the UK doesn’t risk losing its invaluable paper archives. If anything, problems with long-term accessibility are more likely to come from digital archives. “Some of these paper records are hundreds of years old,” Dibben argued, “and yet some of the electronic media from the 1980s is now at threat. Thinking about how to preserve these is obviously important.”

Elliott agrees. “We already have cases of early digital data that can no longer be read because technology [and] machinery has moved on. I think one day we’ll be able to crack it,” he added. “As with any kind of major project, it requires the right questions to be asked at the outset.”