Meet BugFinders, the British company crowdsourcing the art of app testing

Barry Collins Read more October 19, 2016

If you’re charged with finding critical bugs in the apps and websites of major clients such as the Met Office, Tesco and Bang & Olufsen, enlisting the help of testers sourced via the internet might not seem like a smart idea. According to BugFinders CTO and founder Martin Mudge, however, it’s the only sensible way of doing it.

BugFinders describes its job as “true crowdsourced testing”, employing freelancer testers from all over the globe to spot and report the flaws in its clients’ software. A community of 55,000 testers sit waiting for their next job, some earning as much as £800 per week, many doing it purely for the challenge.

Why does online outsourcing work so successfully for software testing? And how does BugFinders prevent sensitive flaws in big brands’ software becoming public knowledge? How might you, if you have the skills, join the team? We met with Martin Mudge to find out.

Numbers game

bugfinders_martin_mudge M Mudge, CTO & founder, BugFinders

Mudge started testing apps in the days before crowdsourcing was even heard of, let alone desirable. He was head of the testing team at mobile network Orange at around the turn of the decade, when the number of devices you needed to test against was tiny compared to today. “At that time, you only really needed to cover 15 devices to get good coverage, and we’d use a team of two to do that,” he said.

Before long, however, app developers began complaining that bugs were being missed and Mudge couldn’t work out why. “It transpired that, in the same way spot-the-difference puzzles have only two pictures, if you give the tester more than three devices or three browsers to check the same application, they basically get bored by the fourth one and don’t see bugs unless they’re really big ones,” he said.

There was another problem: with the app boom triggered by the emergence of the iPhone and Android, Mudge’s team simply couldn’t keep up. “I and my co-founder [who happens to be Mudge’s wife, Donna] came up with the idea that what we needed to do was use a bigger team of people who are remote to do the work.” Donna already owned her own recruitment firm, giving them a pre-existing network of freelancers to tap, and consequently BugFinders was born in 2011.

Experienced testers

“If you give the tester more than three devices or three browsers to check the same application, they basically get bored by the fourth.”

BugFinders isn’t like other crowdsourced labour projects, such as Amazon’s Mechanical Turk, where essentially anyone can earn a few pounds by completing a relatively menial task. BugFinders specifically recruits experienced testers, ideally those who already have at least three years’ experience working as a tester. They might top up their existing skills and salary, earning freelance income in their spare time, or they might be one of the elite testers who can earn a full-time salary by testing from home.

Finding the number of testers required to make crowdsourcing effective proved to be more difficult – and expensive – than Mudge thought, even with Donna’s experience in the recruitment industry.

Although they quickly enlisted a decent-sized community of part-time testers in the UK, Mudge needed testers available in multiple time zones, so that when clients dropped an app that required testing within a short time frame, BugFinders had people who were able to work around the clock. “We sent people all over the world at one stage to recruit people,” he said. “That worked really well.”

Indeed, it pushed tester numbers past the 25,000 mark – what Mudge described as the “critical point whereby we could then effectively take a project at any time, day or night, 365 days a year and deliver it”. That number has since swelled to 55,000, largely helped by existing testers referring colleagues, although anyone can sign up via the company’s website (head to bugfinders.com/register).

A team of testers that would come close to filling Arsenal’s Emirates Stadium sounds like one heck of a workforce, but when you consider that you now need to be testing apps/websites on 152 devices to cover only 80% of traffic, you begin to see the necessity of having so many people to call upon.

“Most of our clients, when we start talking to them, are covering five or so devices internally,” said Mudge. “They’re missing out on massive opportunities to improve their mobile experience. Effectively, with 300 testers we’re covering sometimes 400 or more mobile devices – so way over the 80%.”

In-house validation

BugFinders doesn’t rely purely on the output of its freelance testers. An in-house team will check and collate the bugs identified by the freelance testers, before they’re reported back to the client in a clear and consistent fashion. And testers have to prove their mettle before they’re allowed to work on live projects. “They firstly come in through a vetting process,” said Mudge. “They’ll go on a sample project, they’ll raise some bugs, we’ll adjudicate those.” If they pass that hurdle, the next step is some entry-level live projects. “These will typically be things like e-commerce, which are a little less complex or people would commonly know how they work.”

Those who deliver accurate work on a consistent basis can be promoted through the ranks to the elite team, where the earnings really do become more than pocket money. “Sometimes, they’ll earn £800 a week,” said Mudge. “In different parts of the world, even in the UK, that’s pretty good money for sitting in your pyjamas all day, right?”

As lucrative as the work may be, Mudge insists that around 20% of the testers aren’t in it for the money at all. He cites the example of a tester in London, who works as a contractor “on about £1,000 a day” as the head of testing in a bank. “His wife goes out on a Tuesday and a Thursday and he says, ‘I like to keep my skills fresh, so rather than watching Coronation Street, I want to do some testing and work on the latest apps’.” Anyone who’s ever sat through an entire episode of Coronation Street can doubtless empathise.

Strictest confidence

Surely, I put it to Mudge, one of the biggest fears of clients such as banks is that devastating bugs in their software are going to be publicly exposed or sold to hackers? He insists not, likening it to the early days of the internet, where you were told not to enter personal details into websites through fear that everyone was a wolf in sheep’s clothing. “Things have changed,” he said. “It just isn’t suitable to say we can test something internally, because you’re going to deliver an app that won’t meet your customers’ needs. You have to get over that problem.”

BugFinders doesn’t rely purely on a leap of faith from clients. Highly confidential launches will be tested only by the elite team of testers who’ve proved their ability and loyalty to the company over a number of years. All testers are also asked to sign a non-disclosure agreement, “and thus far, we’ve never had any confidentiality issues at all”, Mudge insisted.

“There’s a professional element within that community that kind of self-polices.”

“Remember: the testers aren’t doing it to get people’s secrets; they’re doing it because they enjoy testing and… they’re professionals. There’s a professional element within that community that kind of self-polices,” he added.

Second chances

Despite that enormous workforce of 55,000 freelance testers, BugFinders has only 21 in-house staff, although there are plans to swell that number to 30 or 35 by the end of the year. About half of the current workforce is dedicated to validating the bugs from the freelance team, which is a big responsibility on the shoulders of less than a dozen staff.

The validation is also such a unique job (BugFinders uses its own bespoke software to collate and report bugs) that staff have to be trained in-house. “Because it’s never a job that anyone has experience of, we bring in people who have been long-term unemployed,” said Mudge, describing the company as a social enterprise. “We give them a trial and, if they’re good at it, we take them on. From a Cheltenham perspective [where BugFinders is based], the Job Centre love us.”

However, there’s no pathway from freelance tester to in-house staff; they’re two different disciplines requiring two different skillsets. “We’re into sending 747s to Australia in two hours, with extra rockets on the wings,” he said, while testers are more likely to go “down to Gloucester Airport and take a Cessna out for half a day”. Either way, it sure beats watching Ken Barlow in the Rovers.