These AR glasses aim to transform the theatre

When Dave Finch lost his hearing, it came on suddenly and deteriorated rapidly. He used to love going to the theatre, but not being able to catch dialogue meant plays became a mish-mash of half-heard sounds.

Closed caption performances gave him a way to keep up with the action, but they were often frustrating – textboards tucked to one side of the stage, forcing him to flick eyes back and forth between actors and words.

He just couldn’t shake the feeling he was missing out. So Finch joined a trial of a new technology, spearheaded by the National Theatre (NT): augmented-reality glasses that make it seem as if subtitles are hovering in front of the viewer’s eyes.

READ NEXT: VR, AR and MR, what’s the difference?

“I will never get my hearing back,” he told an audience of journalists in the NT’s vast Olivier space. “That’s gone. But the glasses are the next best thing to regaining that sense of involvement you have; when you’re looking at something and are really absorbed by the performance.”

The technology, dubbed Open Access Smart Capture, has been developed by the NT in collaboration with live-subtitling expert Professor Andrew Lambourne and the Accenture Extended Reality group. It’s based around pairs of Epson Moverio BT-350 smart glasses, which relay captions for a play in real-time over Wi-Fi. The idea is that audience members will be able to order the headsets for any production for free, pick them up ahead of the performance and hand them back in when it’s finished.

At the launch of the device, I was able to try them out for myself during a short extract of the theatre’s current production – Exit the King, starring Rhys Ifans. I was handed a pair of clunky glasses, attached to a controller, and told I could sit anywhere I liked. With the glasses on my face, I clicked through a short on-screen tutorial about tweaking text positioning, size and colour, then I was all set.

The aim had clearly been to reproduce the look and feel of TV and film subtitles. As characters spoke on-stage, lines of text seemed to appear beneath them (or above, if I preferred). Sound effects were likewise given the subtitle treatment, all of which I’m told was relayed not through a live captioner, but through automated voice and cue tracking.

READ NEXT: A rare disease robbed me of my sight. VR brought it back

“It works by using speech following software, which follows the phonetics of the performance, alongside technical information such as lighting cues and sound cues,” explained the NT’s Jonathan Suffolk. “It combines all that information to provide an automated service that responds to the performance as it’s going on onstage.”

There were a few times when this didn’t quite work. Dialogue would appear a beat too late or linger too long before glitching through the rest of the script. When it came together, though, the effect was impressive. Instead of needing to constantly dart a glance to captions at the side of the stage, the viewer can look at what’s happening right in front of them. 

“It’s centre stage,” said Finch. “You can focus forward. That may not sound like a big deal, but for someone like me that means you can look at the performance and line the text above the performers. That means I can still lip read. I can still see all the movement. I can still see the facial gestures and expressions.”

According to the charity Action on Hearing Loss UK, there are currently 11 million people with hearing loss in the UK, and that number is expected to rise to 15.6 million by 2035. With potentially one in five people suffering in the not-too-distant future, technologies such as this aren’t just a flash addition to cultural institutions, but a lifeline to audiences that might otherwise never step foot in a theatre. Instead of a handful of captioned performances, AR subtitling could open the doors for people with hearing loss to come to any performance, and to sit anywhere in the theatre.

READ NEXT: Augmented reality’s future lies in the 1930s avant-garde

The iteration I tried, however, showed there’s still room for improvement. Not only did text sometimes mismatch what was happening onstage, but the focus on the headsets were a headache to get comfortable. I couldn’t imagine wearing the glasses for longer than half an hour, which isn’t ideal if you have the whole of King Lear to get through. These were definite irritations, but they are also issues that can be fixed and should be if the technology is to live up to its promise.

Because there really is a lot of promise. It was mentioned during the launch that the glasses could be used for captioning in languages other than English, effectively turning them into visual translation machines. There’s also scope for the headsets to be used with live captioning for talks and improvised shows. Away from theatres, they could also be used in galleries and museums for reactive captioning of artworks and artefacts. There are all sorts of ways this hardware could be put to use.

To begin with, 90 pairs of glasses will be available for the next stage of testing in 2019 between the NT and Leeds Playhouse. “If there’s a surge in demand from people who don’t think of themselves as hard of hearing but want to use the service, that will ultimately mean it grows from 90 to whatever number,” said George Marcotte, managing director for Accenture Digital.  “That scaling is a challenge for any digital project,” he admitted.

READ NEXT: What virtual reality can teach us about life with brain damage

If the technology does scale, and finds itself in theatres other than the relatively well-heeled NT, then it could be a revelatory addition to the artform’s accessibility toolkit. The NT was keen to press that this wouldn’t replace pre-existing provisions, from stage captions to infra-red audio loop systems for hearing aids, but would hopefully be a way to expand its options for audiences in the future – opening doors that might otherwise be shut.

“The biggest advantage of this – so big that it’s easy to miss – is the fact that without these glasses I wouldn’t be in the theatre,” said Finch. “I’d be sat at home wondering about what I was missing.”