Beta testing in the web age
During the first 9 months of Nokia Beta Labs existence, there’s one thing we have learned the hard way: beta testing in the web age is not what it used to be. Nowadays, everything in the web gets labeled as “beta”, and stays such seemingly forever - despite being technically very mature from the users’ point of view. Consequently, people’s expectation level has been raised significantly: they tend to expect - and demand - public betas to be technically reasonably mature = stable and not-too-many-bugs.
This has left quite a many people scratching their heads, or bitterly proclaiming that the word “beta” has become meaningless. I tend to disagree. I believe the meaning has just changed, due to the fundamental changes underway in the software industry and the web.
Let me tell you a simplified story of what I think has happened and why.
The origin of “beta”
According to Wikipedia, it all started in IBM hardware development:
“The term beta test applied to software comes from an early IBM hardware product test convention dating back to punched card tabulating and sorting machines. Hardware first went through an alpha test for preliminary functionality and small scale manufacturing feasibility. Then came a beta test to verify that it actually correctly performed the functions it was supposed to and could be manufactured at scales necessary for the market, and then a c test to verify safety.”
Now, if you think about the beta testing of clunky punch-card machines, life probably wasn’t easy. The marginal costs of adding beta testers must have been unbearable (material costs, shipping&logistics, etc).
The economic curve must have looked something like this:
I believe beta testing wasn’t too popular back then, and if you did it, you didn’t invite too many beta testers to the loop.
Golden age of shrink-wrapped software
Eventually, PC software became unbundled from the hardware, and so began the golden age of “shrink-wrapped software”. Companies wrote software, wrapped them into convenient boxes, and sold them in retail stores to the masses. Beta testing such stuff became much more feasible:
- reasonable fixed costs
- reasonably low marginal costs (shipping CDs, etc)
- relatively high lost-opportunity costs (if beta testers didn’t pay for the final software)
I believe the economics looked something like this:
And so beta testing became mainstream, as a widely accepted part of software development cycle. But as your business depended on charging hard cash for the final release, you had to manage the software distribution carefully, and limit the number of beta testers. And it was probably no accident that the typical size of a beta test was about 100 active testers.
Side note: if you are interested in doing beta testing for shrink-wrapped software, personally I found these sources useful:
- Joel Spolsky: Top Twelve Tips for Running a Beta Test
- Michael Fine: Beta Testing for Better Software (Michael is founder of the interesting beta testing company Centercode)
Beta testing in the web era
With the advent of the web (supported by the open source movement), something big happened: software and web stuff started becoming “free”. The users didn’t pay directly for the software anymore, as it was funded by indirect business models, such as:
- advertising (e.g. Google)
- selling hardware (e.g. Nokia)
- selling services (e.g. Red Hat)
- selling your company and user base (Web 2.0 startups)
- + non-monetary incentives (e.g. the “free software” ideology)
Suddenly, the marginal costs (and lost-opportunity costs) of adding beta testers dropped close to zero.
Suddenly, adding beta testers started creating direct value, as your business was dependent on the usage, not on paying customers.
Suddenly, it started making sense of offering your stuff as perpetual beta.
Changing definition of “Beta”
Before diving into the new definition, let’s take a look at the traditional ones:
“A beta version is the first version released outside the organization or community that develops the software, for the purpose of evaluation or real-world black/grey-box testing. The process of delivering a beta version to the users is called beta release. Beta level software generally includes all features, but may also include known issues and bugs of a less serious variety.”
Not quite. In the web age, you don’t need to have all the features, but you should try to minimize the bugs.
ZDNet / Computer Desktop Encyclopedia:
“A pre-shipping release of hardware or software that has gone through alpha test. A beta version of software is supposed to be very close to the final product, but, in practice, it is more a way of getting users to test the software in the first place under real conditions.”
Almost, but doesn’t fully cut it… What is the “final product” in the world of perpetual betas and continuous improvement?
Michael Fine: Beta Testing for Better Software, page 11:
“Beta testing is the managed distribution of a product to its target market; the gathering of feedback from the market; the evaluation of the feedback into manageable data forms; and the integration of the data into the organization it affect”
Getting closer. However, in the web age, it’s less managed. And it’s about co-creation, not about 1-way feedback.
Let me try to define it myself:
Beta testing in the web age is distributing not-guaranteed work-in-progress stuff to your target market, and co-creating further improvements. - Tommi Vilkamo
Hmm. Something like that.
What do you make out of this all?
You might find this post by Faris and the link to Russell interesting.
http://farisyakob.typepad.com/blog/2007/06/people_are_alwa.html
Comment by Charles Frith — January 3, 2008 @ 3:28 pm
That was quite a history about “beta”. I feel your definition is the best, but if you think about it it’s not that different from the other ones. From what I’ve understood all beta tests, from the very beginning, have had the purpose to see the good and the bad in a product and help the manufacturer improve it, which is what is going on in the present. Sure you can’t compare testing a card puncher the size of a fridge to testing a few megabytes of software, but we’re talking about the general process and it’s objective, not the thing that is being tested.
All in all, it was an interesting read. Can’t wait for the first beta card puncher….er, I mean application, of 2008.
Comment by Bogdan Galiceanu — January 3, 2008 @ 5:54 pm
This is why I prefer the open source terms stable and unstable. The word unstable scares normal people, but tempt bleeding edge folks like myself to try it out. Beta as a word has been ruined by Web 2.0.
Everything is a beta, oh well.
Comment by Stefan Constantinescu — January 3, 2008 @ 6:44 pm
Stefan has a point. You could have your vocabulary. Add prefixes or have other warning sings to sw which is very unstable.
I think Tommi has explained every time quite well in what kind of stage the sw is. However, these could be made simple concepts which could tell people what to expect of the maturity.
I think it could help you getting the feedback from those users you want to have feedback from.
Just a thought.
Comment by Henrikki — January 4, 2008 @ 12:30 am
Following your own resolution, you’ve earned yourself a kick in the nuts courtesy of yourself!
“Web startups” reads just as well as “Web 2.0 startups”.
http://blogs.s60.com/tommi/2008/01/time_to_figure_out_a_new_box_a.html
Comment by Hugo — January 4, 2008 @ 11:45 am
Heh, you’re right. *ouch*
Comment by Tommi Vilkamo — January 4, 2008 @ 12:34 pm
> You might find this post by Faris and the link to Russell interesting.
> http://farisyakob.typepad.com/blog/2007/06/people_are_alwa.html
I sure did, thanks!
> This is why I prefer the open source terms stable and unstable.
> Add prefixes or have other warning sings to sw which is very unstable.
Good point, we’ll discuss how that should be done.
Comment by Tommi Vilkamo — January 4, 2008 @ 12:40 pm
I have an issue with the third diagram, bet testing value in the 2000s. When my company betad we had so many testers, we began having beta-spam issues. So the curve does begin to go south after some point, albeit a much more eastern one then before…
Comment by Ran — January 4, 2008 @ 6:34 pm
i prefer the term living software to beta software. i mean seriously, beta back in the day meant you got a CD with some bits that were not final enough to put on a store shelf. now a days with most of the software sitting on a server somewhere spitting out information in your browser of choice, a user doesn’t see the hourly edits you make to code.
Comment by Stefan Constantinescu — January 5, 2008 @ 12:20 am
Hi Tommi, great post on beta testing…just few comments to your definition.
I agree on the not-guaranteed and work-in-progress but do not agree on the target market and co-creating further improvements since they are also relevant during other development stages. The target market might not be really known during beta stages and co-creating nowadays is ongoing during and after beta.
I’d say that going to beta is the chance for a company to *viably include the community (as in customer, consumer, vendor…) to the concept creation and development stages*.
I think that in your definition you should at least mention that is more viable, considering that you added 3 graphs on the current scalability of beta testing
btw, Happy New Year to everyone…
-Reda
Comment by Reda — January 5, 2008 @ 10:07 pm
[…] the Nokia Beta Labs blog there’s an interesting post and discussion on the meaning of ‘beta’. Many people accept that in the web world ‘beta’ […]
Pingback by Mobile Phone Development » Blog Archive » Perpetual Beta — January 8, 2008 @ 4:45 pm
> beta testing value in the 2000s. When my company betas
> we had so many testers, we began having beta-spam issues.
> So the curve does begin to go south after some point
My point was that if the business value of your initiative increases in proportion to the number users (= if you rely on indirect business models like advertising or aquisition baits), it might be higher than the costs for an additional beta “tester”. Therefore, it might never go south.
The only limitation then, as pointed by Simon Judge in Mobile Phone Development, is that not everybody is comfortable installing and using beta-titled stuff. In which case you start feeling pressure of taking your perpetual beta finally out of beta.
Comment by Tommi Vilkamo — January 10, 2008 @ 3:48 pm
nice information. excellent for students & researchers in this area. regards from Peru.
any beta for playing different kind of formats? e.g: wma
Comment by Mariano — January 11, 2008 @ 12:15 am
We had an interesting discussion about this here at work today.
The conclusion was that a “private beta” is a closed, invitation-only, limited volume program with the goal the get user feedback, try out different UIs and core technologies.
Later moving into a public beta it is still possible to limit user volume by territory or simply volume, just that the volumes are a lot bigger than in the private case.
Final, non-beta status would then be reached when the service is released in an unlimited fashion. This will only be done when infrastructure is in place, you know your core servers scale as needed etc.
The above of course isn’t the single truth, but one way of viewing things.
Comment by Göran — January 18, 2008 @ 10:17 pm
cinquefoiled forlornly scolytus eradicate undertie affricate jasmined butcherous
Campus Chronicle
http://www.catamounttrail.org/Association/fAssociation.html
Comment by Gayle Lowery — March 25, 2008 @ 6:19 am