- Mon, 2009-09-28 15:29Comments:19
- Mon, 2009-09-07 13:10Comments:23
- Thu, 2009-09-03 12:21Comments:22
Nokia Beta Labs
Contacts on Ovi beta database failed - my deepest and most sincere apologies
Thu, 2009-02-12 09:12Written by: Kristian Luoma, Product Manager, Contacts on Ovi.
Hey,
Something bad happened yesterday. A cooler broke down in the hosting center that we run the Chat service in. This event lead in to two catastrophic consequences from our point of view. Firstly we had to ramp down the service down for a very long period, in fact most of the yesterday morning and afternoon we had a service break. Secondly, our database broke down. Despite the fact that we had regular back-ups, we were not able to set it right.
What we had to eventually do, was to return a back-up copy from our previous hosting center, created on the 23rd January 2009. As a consequence, anything that you’ve done since that (profile details, images, friendships) are gone. The problem has now been fixed and the server is now back and working. You should now be able to log back to your Contacts and start using it as normal. You do, however, need to invite your friends again etc.
Words can’t explain how incredibly sorry we are for the inconvenience. We’re sorry for the lost contacts in your phonebooks. We’re sorry that the profile pictures you love, and we love too are gone. Nothing can make this right, we know, but we’re hoping that you can forgive us and give another chance to give you good service.
We’ll make every effort to avoid this problem in the future. I am sorry.
- Kristian
Im agree hamei, unprofessional...
Well, this will become another famous example in all admin textbooks.
Let me guess, to be extra safe you kept your backups in another building, and to be really, really safe, you installed metal detectors in that building?:P
Hi, Im from venezuela and I have a friend that has installed CoO in a nokia 6110 navigator but he now his Pc doesnt recognize his phone and he cant shutdown the antenna of the GPS... how can I fix his phone!? so please help me! we are Nokia fans and I hope that this new software will give to NOKIA the first place in the market
Well done - *this* is how to handle events such as these. Anyone who has worked in (or worked with) IT for any length of time knows that as great as technology is, it is not infallible.
No company can promise 100% availability and the one that does will do so at a cost that is not practical for users of the system.
The real lesson here is - own your information: know that your data is your responsibility, and always plan for the worst. Keep backups, in several locations, of everything that you couldn't afford to lose.
Years ago, people would store their photographs in an old shoebox at the top of a cupboard, where they would stay safely for 20 years. Sure, they would be a little yellow, but they would be there :-) Nowadays, we keep our entire lives on one $100 hard drive: any bets whether it will still be running smoothly in 20 years?
Well handled Nokia labs.
Mark.
its cool, coz its beta. (And I hadn't started using it yet ;-)
But backups didn't work, is a bad thing, a very BAD thing. Things go wrong and they will, and our only fall back are backups (trying to save as much as possible).
I suggest you should look more into that .................... I'm sure u already have ;-)
Have a Beer, Cheers,
Rushi
Well, such things happen... nevertheless, such things are incredible for reputation - especially if the backup is that old... Users can get pissed quite easily :(
However, keep going your great work at betalabs
Learn it.. upse a mirror
This is simply unprofessional and beta or not (it's at least a public beta from the worlds leading mobile manufacturer) fact is: no mirrored server, no working backups, 3 weeks data lost... unprofessional.
I'll survive. If the airco goes bust, the machines simply shutdown without dataloss. So you mean a watercooler filled with water? What the hell is a cooler doing in a server space? And why are there no picutures of the disaster flooding?
Your honesty is well appreciated... It is unfortunate that data was lost but as is often the case incidents don't conveniently happen one at a time.
Keep up with the beta testing and let us know what redundancy you put in place when OVI leaves the lab and goes into "production".
Also continue being honest like this.... another reason why I appreciate Nokia...
Tommi Vilkamo I said working...
Live and learn. Mistakes will be made. There is a huge difference between constructive criticism and just plain sour grapes b*tching. Dave and R. Melvin apparently have an axe to grind for whatever reason.
We live, we learn. With Ovi we will need to learn how to have adequate backups, now that we will be handling peoples' contacts, e-mail, etc.. Get online with the Red Hat people if you need external help.
Thanks for the notice, we all know the betalabs teams work really hard, so keep up the good work!
That is why it is called "beta", thank you Nokia for allowing us "Geeks" to trial this service pre-release. Since I have no friends I did not loose much info....
No probs Kristian, It does happen. Has happened with others too.
Lets be glad it was just a beta, not many people were using it. Otherwise, it would have been catastrophic.
> How many people works there??? one???
Two. Me and Ilkka.
In addition, there are several (typically very small) R&D teams from all around Nokia, who are behind the actual application and service prototypes showcased at Beta Labs.
*HUG!*
Don't beat yourselves up for this. If this was a production service then you'd have a different set of SLAs to stick too, and obviously would have provisions that this couldn't happen.
These things happen, i'm sure you've all learnt from it too, and it takes things like this to figure it all out! Don't beat yourselves up over it!
Plus, that apology deserves an award!
Me rakastamme teitä edelleen!
Meh, hey, it happens. I mean, the devil never sleeps, especially when it has to do with computers and stuffs. So yeah...annoying maybe, but it sounds like it's not exactly your fault directly, so... :)
Wow, a public apology? That's incredible, kudos to you! Not many companies would be brave enough to do such a thing, and it's nice to see a little honesty once in a while... Great move!
It's understandable that something like this may have happened with a beta product, and I am not mad at all about the incident since I only had a few things saved, but the term "beta" is used rather frivolously lately.
A great example of this is Google's Gmail - it's still "beta", and if something like this were to happen with Gmail, it would be an absolute catastrophe. People would lose GB's of email, contacts, etc. since they rely on this service as their primary means of email. Google puts a ton of effort into their "beta" services, since they're truly not "beta" to begin with. I understand that Nokia's Beta Labs can't be compared to Google, but it's a step forward that Nokia may want to take if they want Beta Labs to be taken just as seriously.
The way I see it, if an application is marketed and available publicly for download, it is not beta and should be considered a release. Perhaps not a "final" release, but when thousands of users are using a product like this it should probably be watched a little more carefully.
Contacts on Ovi is an amazing product, and one that I will continue to use regardless of any hiccups that happen along the way.
Thanks for fixing the issue, and best of luck in the future.
its all cool.
To bad it gives more ammunition to the silicon valley bloggers ;P.
What can I say? You cannot count on the reliability of the machine. Catastrophe happens anytime you cannot anticipate. What you should do is to make hot backup as well as regular backup. Seems your regular backup isn't that regular though. Anyway, do think seriously about the reliability, security, and scalability. Even it is only a beta service.
No probs, so for the next time (sgrat :D) you will be prepared :)
what crash today??
Beta System Admins, too :-)
Good for owning up to it. Tell us the truth and the way it is we can deal. Nice pug, if you ever lose 3 months of data I have a heart wrenching picture of my daugthe at 3 months you can use instead of the cutie pug. Now bad to work.
Live and learn? The lesson of backups has been well taught over a century. Certainly IT person living today hasn't heard of them. At this point a failure to backup is the sign of the truly lazy or incompetent.
it sounds like another chapter of nokia beta labs terror movie.
How many people works there??? one??? with many nicks???
when are u gonna make something usefull?
Right, right.. thanks for your support.
It just seems that this was a sum of things, lots of other things happened (between the cooler breaking and the database replicas not working) that ended up in this outcome.
Again, we're very sorry, and will be making every effort to make sure that this doesn't take place anymore.
-Kristian
This *is* Ovi "Beta", isn't it? That's the point. It's cool.
And the A-hole of the Day (hmm.. Year?) award goes to.. ***drum roll*** Dave!
Poop happens, as they say. :)
Kristian.
The pictured dog it's my French Bulldog, called Dexter, please bring it back to me, don't you dare to make another catastrophy.
Nice apology anyway that's how bad happenings must be PR handled.
WTH, is only a beta, thanks for your honesty. We'll trying and testing. Regards and hug Kristian.
Hi , don't take it too hard , reading your words , I'm picturing you having tears in your eyes , ooh my poor fellow on Contacts , you got nothing left , hey it's a beta , it's prone to break down , all I want to know what the cooler has to do with it was it a an Air cooler or a Water Cooler , some one let the tap running when drinking water and short circuit the whole lab , do give details , I'd been hard on you in my blog but that's only me :)
Farhan
People simply DO NOT UNDERSTAND THE MEANING OF THE WORD BETA!
The word beta doesn't refer to poorly designed/tested data storage environment. Bad marks to the hosting center team.
Kudos on an apology like that. Not every company would have done that, shows how much you guys care about what you provide.
First of all, it's a beta. Things go wrong. Yeah, there should have been a closer back up but there wasn't, and there's nothing we can do about it so lets not beat ourselves up.
Secondly...did anyone really use Contacts on Ovi that much? I mean...REALLY? Thought not.
So why didn't you use Amazon other cloud storage solution rather than depending upon internal (lack of) expertise to implement the service?
No problemos
http://www.facebook.com/ext/sh
It would be cool to have a simple JABBER CLIENT to connect to any server, not only nokia's!
What the fuck is wrong with you guys? Is there no thing as regular backups? Your last backup was on 23rd Jan. You have more serious issues to attend that releasing stupid new features.
Honesty is so refreshing. It makes it a Beta World..
go Dave ! more rants
Nice one, Kristian. Simple uncomplicated honesty. And QuITe Perfect for a Core Team into AI Control of Communicating Computers in Cloud.
Are Nokia into such Big Picture Great Game Plays?
A database crash is always the least desirable thing to experience, however, it is pointless to tell the Nokia guys what they already know. They made a mistake. Get over it. It's not like it is a paid service or you have a service availability contract with Nokia.
emm. luv it :)