Bug-o-meter

< Previous: Gizmo for S60 upd Next: Beta Labs Archive >

Thu, 2008-03-06 13:13

Quite a many of you have asked us to indicate the technical maturity of the apps and services available here at Beta Labs. We liked the idea. When people have the right expectation level, everybody wins:

diehard-users can play around with early releases

others can wait until the app is "mature enough"

feedback becomes much more constructive

Here's the technical maturity scale we came up with:

How do you like it?

Tags:

Comments

Submitted by on 06-03-2008 18:38.

Love the 3 stage sign. Think they should be linked to the actual bugs (or the disclaimer if RED :D lol!

Submitted by on 06-03-2008 18:47.

> Well, [fixing any known issues before a public release] would postpone all launches by months. Do you really want that?

Yes.

However I can't believe it would take months to fix the obvious issues that anybody can notice immediately when first starting the application.

Just look at the Maps feedback. About 300 comments mainly reporting the same dozen obvious major faults in it - crashing, not working anymore on a bit older phones, eating phone memory, less-than-before detailed map data, license keys not working, bad search, bad screen layout, reversed keys, etc. All of them something that everybody will notice within the first 2 minutes of using the software, giving an extremely bad first impression of your company and product quality.

If you would have fixed just those few issues before release, and tested that it actually does run on every Nokia phone model it's supposed to, your products and your company would have greatly better image within your paying customer base.

But no, instead of a relatively bug-free application you released - and let be quote Jukka Nurminen, Nokia-D - "an early R&D version made by summer trainees." And he even added "Releasing that kind of R&D version is very irresponsible and harmfully for the company." There's not much to add to those words.

Applications like Maps are not something of trend or anything that would go out of fashion in a few weeks. They are services that are supposed to be used for years to come. What does it change if they are released a few weeks or months later - with much better quality, giving an excellent first impression in the public?

Submitted by on 06-03-2008 13:30.

I think that's a great idea! Will help less advanced users stay away from earlier releases

Submitted by on 06-03-2008 13:35.

I agree with having three levels of maturity, and would focus the descriptions on what the user may expect and which feedback will be appreciated by you. So, respectively:
1) Work in progress: we know it's buggy but first of all we want to know whether the features and execution are those that you expect from the software once it's done
2) Getting there: the features are complete and final, there are bugs to be squashed, please report
3) Almost done: should work well and we need many people to test it in order to figure out low-frequency bugs

Submitted by on 06-03-2008 15:37.

I like the idea of setting expectations!

Consistently making the "known issues" available would go a long way in making people aware of the problems to expect. The list exist from some projects, like the Location Tagger under the "More Information" bar on the project page, but not others, like Sports Tracker. This will also minimize the "It does not work" posting, or at least change them to "It does not work due to known issue ____".

I like Heiko's idea of linking the symbol to the "Known issues".

Thank you.

Paul Spencer

Submitted by on 12-03-2008 15:48.

kanta - you don't speak for me. I'm a very satisfied user of Maps 2, it has faults but these are outweighed by the advantages for me. I like the idea of getting access to applications early - if they work reasonably ok I keep them, if they're not good enough I remove them and wait for the next version. I don't burst into tears and start snivelling to Nokia that a beta release isn't perfect.

Submitted by on 06-03-2008 13:35.

It's a good step towards knowing where apps are whilst still in beta phase.

Good system, hopefully it makes it clearer.

Submitted by on 06-03-2008 13:46.

Looks great, good idea :D

Submitted by on 06-03-2008 13:54.

A great idea. I think one of the problems may be the overuse of the word 'beta'.

Many end users perceive beta to be almost finished code that contains a few bugs. They expect it to be feature frozen and the UI to be almost complete.

The word 'alpha' or something could be used to make them realise this is not like all the near-perfect beta releases that Google keep doing. Remember how long Gmail was in beta?

Submitted by on 06-03-2008 13:57.

The term "known issue" has no meaning, since you don't list them, i.e. the issues are not known by the user. In any case, you should fix any known issues before a public release, releasing junk just gives a bad image of your products. At minimum you should before release fix the issues any user can see in 2 minutes of actually using the software.

"Should work"... So on your almost ready level you still don't even know if it actually works at all or just fails miserably. Like Maps.

Submitted by on 06-03-2008 14:02.

Gmail still is beta... Or maybe its actual name is "Gmail beta"...?

Submitted by on 06-03-2008 14:16.

"Early release many knwon issues" - App should not be in beta stage. Move it back to alpha labs.

"Some known issues" - Every app has issues even graduated ones. This message has no meaning. Like kanta said you have to list them.

I stick to Ben Rose opinion, that many users perceive beta apps to work well on their device, but have not the full set of promised features or look a bit ugly.

There should not be an extra evaluation system. Declaring something as beta should be meaninfull enough. There is alpha, beta, release. There should not be alpha, beta, a-bit-less-beta, now-a-bit-more-beta,.....

Submitted by on 06-03-2008 14:17.

Glad to hear you liked it :)

> The word

Submitted by on 06-03-2008 14:49.

> you should fix any known issues before a public release

Well, that would postpone all launches by months. Do you really want that?

do it like before.i never found a app here the doesn'T worek completly so stay like before, works good i think.

and the staus symbols are great like they are now.but it hink most of the guys here are "diehard-user's" so it doesn't matter ;)

Submitted by on 06-03-2008 14:54.

But Tommi u guys can link the Symbols:

So when u click the symbol that a list with known issues appear, it's a easier and faster way to see what bug's exist.

heiko

Submitted by on 06-03-2008 17:02.

Like it! Now, how about a digg-style list of bugs so that we can check the frequency of a specific "inconvenience" ;-)

Submitted by on 06-03-2008 19:21.

Kanta, listening what you say, it seems you are not in the target user group of Nokia Beta Labs. This is a place for bleeding-edge innovators who are willing to tolerate rough edges and a reasonable amount of bugs etc. The earlier we hear their thoughts, the more time we have to take it into account in the final release and forthcoming releases.

You make a good point, though. Sometimes, people don't understand the beta nature of this website and get upset about lower quality and incompleteness. I hope the traffic lights will help reduce the confusion.

Submitted by on 06-03-2008 20:00.

Tommi, maybe I'm not a member of your image of the target user group of Nokia Beta Labs. However I'm here, a happy user of some of the products available (only) here.

If you really think all the people here are innovators who are willing to give all their time and skills to improve your products, the first thing you should do is to hire them to get the product done as efficiently and fast as possible. You would get pretty darn impressive and innovative products out, something to really kick your competitors butt with.

However - sorry to break your dreamworld - most of the people here are just normal users of your phones, people who are looking for any interesting applications to run and take the most out of their (expensive) phones. For those people, Beta == more features. Like many people have said and referred to Gmail, the word Beta has ages ago lost the meaning which you use it for.

Especially Maps must be getting the attention of a lots of ordinary people, since you advertise the 2.0 Beta on the Nokia Maps home page. But ordinary people don't tolerate bugs. Especially from a big company like Nokia. It just hurts your public image to release junk.

It just isn't your (paying) customers' job to do your job, i.e. find and fix the bugs. Hire some experienced software professionals (if you can find some) and finish the product internally before releasing to the public. Your customers will be happy, which makes your company happy, too.

Submitted by on 06-03-2008 20:49.

katana this is called "Beta Labs" for a reason, Yes the Maps was not the best Beta test but it proves that this does work. The test went to a small group of "testers" (i.e. us lot here) the company got good feedback on the product. The Product (although had bugs) was still functional.

Note: I've been running it since it was released here and have not came across most of the bugs mentioned!

To be honest this is the day-to-day "Grunt" work you have to do and put up with as a beta tester!

What you think should be an "Alpha" in your eyes may be an acceptable "Beta" candidate to a larger organisation. There is only so much they can do before the get "Tunnel Vision".

I've worked in program design and testing for a small company and I know it's easy to get a set of test that you think puts a program through it's paces but it's not until you let it out in the real world where you find the bugs that you didn't see.

There is only so much a group of designers can test on a program, they need to release it to a larger group of testers who are not so blinded by the code.