Beta testing in the web ageThu, 2008-01-03 09:29
During the first 9 months of Nokia Beta Labs existence, there's one thing we have learned the hard way: beta testing in the web age is not what it used to be. Nowadays, everything in the web gets labeled as "beta", and stays such seemingly forever - despite being technically very mature from the users' point of view. Consequently, people's expectation level has been raised significantly: they tend to expect - and demand - public betas to be technically reasonably mature = stable and not-too-many-bugs.
This has left quite a many people scratching their heads, or bitterly proclaiming that the word "beta" has become meaningless. I tend to disagree. I believe the meaning has just changed, due to the fundamental changes underway in the software industry and the web.
Let me tell you a simplified story of what I think has happened and why.
The origin of "beta"
According to Wikipedia, it all started in IBM hardware development:
"The term beta test applied to software comes from an early IBM hardware product test convention dating back to punched card tabulating and sorting machines. Hardware first went through an alpha test for preliminary functionality and small scale manufacturing feasibility. Then came a beta test to verify that it actually correctly performed the functions it was supposed to and could be manufactured at scales necessary for the market, and then a c test to verify safety."
Now, if you think about the beta testing of clunky punch-card machines, life probably wasn't easy. The marginal costs of adding beta testers must have been unbearable (material costs, shipping&logistics, etc).
The economic curve must have looked something like this:
I believe beta testing wasn't too popular back then, and if you did it, you didn't invite too many beta testers to the loop.
Golden age of shrink-wrapped software
Eventually, PC software became unbundled from the hardware, and so began the golden age of "shrink-wrapped software". Companies wrote software, wrapped them into convenient boxes, and sold them in retail stores to the masses. Beta testing such stuff became much more feasible:
- reasonable fixed costs
- reasonably low marginal costs (shipping CDs, etc)
- relatively high lost-opportunity costs (if beta testers didn't pay for the final software)
I believe the economics looked something like this:
And so beta testing became mainstream, as a widely accepted part of software development cycle. But as your business depended on charging hard cash for the final release, you had to manage the software distribution carefully, and limit the number of beta testers. And it was probably no accident that the typical size of a beta test was about 100 active testers.
Side note: if you are interested in doing beta testing for shrink-wrapped software, personally I found these sources useful:
- Joel Spolsky: Top Twelve Tips for Running a Beta Test
- Michael Fine: Beta Testing for Better Software (Michael is founder of the interesting beta testing company Centercode)
Beta testing in the web era
With the advent of the web (supported by the open source movement), something big happened: software and web stuff started becoming "free". The users didn't pay directly for the software anymore, as it was funded by indirect business models, such as:
- advertising (e.g. Google)
- selling hardware (e.g. Nokia)
- selling services (e.g. Red Hat)
- selling your company and user base (Web 2.0 startups)
- + non-monetary incentives (e.g. the "free software" ideology)
Suddenly, the marginal costs (and lost-opportunity costs) of adding beta testers dropped close to zero.
Suddenly, adding beta testers started creating direct value, as your business was dependent on the usage, not on paying customers.
Suddenly, it started making sense of offering your stuff as perpetual beta.
Changing definition of "Beta"
Before diving into the new definition, let's take a look at the traditional ones:
"A beta version is the first version released outside the organization or community that develops the software, for the purpose of evaluation or real-world black/grey-box testing. The process of delivering a beta version to the users is called beta release. Beta level software generally includes all features, but may also include known issues and bugs of a less serious variety."
Not quite. In the web age, you don't need to have all the features, but you should try to minimize the bugs.
ZDNet / Computer Desktop Encyclopedia:
"A pre-shipping release of hardware or software that has gone through alpha test. A beta version of software is supposed to be very close to the final product, but, in practice, it is more a way of getting users to test the software in the first place under real conditions."
Almost, but doesn't fully cut it... What is the "final product" in the world of perpetual betas and continuous improvement?
Michael Fine: Beta Testing for Better Software, page 11:
"Beta testing is the managed distribution of a product to its target market; the gathering of feedback from the market; the evaluation of the feedback into manageable data forms; and the integration of the data into the organization it affect"
Getting closer. However, in the web age, it's less managed. And it's about co-creation, not about 1-way feedback.
Let me try to define it myself:
Beta testing in the web age is distributing not-guaranteed work-in-progress stuff to your target market, and co-creating further improvements. - Tommi Vilkamo
Hmm. Something like that.
What do you make out of this all?