Yesterday evening all problems started: due to some hardware malfunctioning on the storage unit, all services started failing. We worked all day today to try and restore all data and get our services up and running. Because restoring the data is taking longer then expected we decided to enable the website with an older backup. This backup was made last thursday  (CET 2010-09-02 18:12:49).

I take full blame of this problem and offer my sincere apologies. It’s very stupid to keep the backup system and the database on the same physical storage node. This is a serious misjudgement. Sorry !

Below you’ll find some answers to questions you might have :

Why is this backup so old ?

The reason why this backup is old is because the backup machine as well as trackmypeople database were located on the same storage unit. And that’s the storage unit that is failing.

Why did your answer take so long ?

We opted to try and restore all the data right untill the last tracked timeblock. As we tried all day without a result, we decided not to wait anymore and enable the website with an older backup. We figured it’s better to activate all services now, and worry about the data loss as soon as we have a working database backup.

What will you do in the future to prevent this ?

As soon as everything is back to normal, i will create a more solid backup strategy. With both online & offline backups every hour at least.

So for now everything is back online with all data from last thursday. As soon as we recover the more recent data, we will import these timeblocks for you.

I will contact all owners of a subscription plans with a commercial proposal to try and make up. Sorry.

Comments are welcome here => https://trackmypeople.zendesk.com/entries/255075-sorry-we-screwed-up-again