Beta Boom

In 2001, when 33-year-old Josh Crawford was a freshman in college, he began taking online surveys to earn extra cash. One company sent him household products to try beforehand. Soon, he began testing online PC games, pleased to get not just a sneak peek at new products, but to help shape their design.

"It's a great feeling seeing one of my suggestions added to the retail version of a product," he explains.

Crawford—who directs human resources at the Arkansas-based Ridout Lumber Company—is part of a network of volunteer beta testers that companies from Apple to eHarmony have tapped to explore new features, identify flaws, and refine designs.

Testers might receive a gift card or get to keep the product, but they are not usually paid for their time. Instead, like Crawford, they are usually motivated by an interest in making their mark on new technologies. Many end up specializing in specific types of products; Crawford says he's currently interested in testing home automation devices. Other testers might focus on mobile apps or software.

Though beta testing has been around for decades, the rise of Internet crowdsourcing has opened up new opportunities. For some companies, recruiting testers is as simple as starting a thread on the AlphaAndBetaUsers subreddit. Others have set up dedicated beta testing programs: Apple's Beta Software Program recruits people to test pre-release versions of its operating systems, as does Microsoft's Windows Insider. And many simply turn to one of the industry's only dedicated beta testing companies, a Laguna Hills, CA-based enterprise called Centercode.

Centercode was co-founded in 2001 by 38-year-old Luke Freiler, an engineer who, one year earlier, had been asked to manage a beta test for his then-employer, Ericsson. The company's client list reads like a Who's Who of consumer tech, from Adobe and Amazon to TiVo, Time Warner, and Waze. According to Freiler, around 50% of Centercode clients license its software platform, which offers a single-source tool for building a beta testing portal, recruiting and selecting testers, tracking bugs, managing non-disclosure agreements (NDAs), and capturing and evaluating feedback. The other 50% hire its services team to manage beta testing programs from start to finish.

"The SaaS model of beta testing really struck a nerve," says Sharon Rylander, director of User Research & Beta at Anki, a Centercode client.

The Centercode community includes around 170,000 volunteer testers, all screened and vetted according to what interests them and where they might be most successful.

Keeping testers engaged is a big priority, says Freiler, which is why there are no plans to dramatically increase their number. "We want to put every moment our testers invest to good use." Josh Crawford, for example, got involved with Centercode last year, and has already participated in nine tests. Another Centercode tester and home automation enthusiast, 45-year-old Brad Flowers, has been a part of the community for six years and participated in seven tests. "I can do a much better job if it's something I'm interested in," says Flowers.

Community engagement is also important to John Gruen, product manager of Firefox's 10-month-old Test Pilot program. "We wanted to take this idea of contribution"—long a part of Mozilla's ethos—"and expand it to a more general population. You don't have to be technical to help build Firefox," he says.

Test Pilot participants can opt in to a number of "experiments": an in-browser screenshotting tool called Page Shot, for instance, or Tab Center, a feature that lets you move tabs from the top to the left of your screen. Feedback is gathered via in-browser surveys and dedicated online forums; experiments are also instrumented so the Test Pilot team can understand usage trends.

"We try not to be overly prescriptive with the feedback mechanism," says Gruen, who estimates that around 80,000 people participate in an experiment each day.

According to Gruen, Test Pilot has helped the company resolve the tension between maintaining a stable, first-class browser and being able to rapidly test and get feedback on products. Anki's Rylander rattles off additional benefits: doing quality assurance (QA) across more devices than developers could test in a lab, preparing customer support for the issues they're likely to encounter, and even gauging the effectiveness of marketing messages.

Every once in a while, beta testing catches a show-stopper, as it did when a media start-up that Rylander once worked for prepared for its European launch. There, testers uncovered a small but critical difference between out-of-the-box European and American router settings—one that would have made the company's product unusable.

Even so, according to Rylander, beta testing often gets squeezed as companies race to meet aggressive release dates, or is outsourced to friends of the developers (as it was to disastrous effect on HBO's Silicon Valley). "It's not a habit," she says. Still, many companies are starting to get it. The focus on data has helped. "Beta testing is a tool to mitigate risk and bridge the gap between companies and customers," Rylander explains.

The growth of IoT devices—which, as Centercode's Freiler points out, can't be tested in a lab—has also boosted interest in beta testing. Of course, IoT also increases the complexities of the testing environment. Brad Flowers, the Centercode tester, is married with young children, and admits his wife isn't always enthusiastic is about the devices that he tests. "She's used to things like the remote control and the thermostat working a certain way. If they change, she might not be into it."

Leah Hoffmann is a technology writer based in Piermont, NY.