RWG Downtime - Major disk array "poisoning"

[Admin]

Guys, I am sorry for the extended downtime, but something that is not supposed to happen, happened last night. Since I have been working all day trying to recover lost data (without succeeding), I will just make this short post and get back with a better explanation tomorrow. In short, we managed to get all of our raid disk arrays "poisoned" with corrupt data that spread to all disks. How this happened, I have no idea, but this effectively killed our web and DB-servers. I have been trying all day, on a shitty cell connection, to restore data, but a couple of hours ago, the techs at the datacenter basically told me to give up. So the forum has now been restored to our last "healthy" backup and all of our servers have been reconfigured and set up from backups. Because of this, about 12-24 hours worth of content is gone. So sorry for any inconvenience when it comes to PM's posts, etc. Go ahead and resend your PM's and repost your great posts. I'm off to bed

[Admin]

Just to add; Lots of content is on the way back from our backup-servers, so all images, avatars, etc should be visible again within the next 1-2 hours.

[Mike on a bike]

I will go collect the boys from other sites!

[CptTripps]

Go to bed. I've been there. It's a royal pita, but it happens.

We'll live. :-)

[freddy333]

Thanks for all your hard work.

[EbolaVirus]

It's always more likely for these types of things happen to us admins on a weekend.

Thanks for trying to save everything.!

Go to sleep! =)

[KB]

Once again we are so very lucky to have the Admin we do, who can keep our downtime from such a major crash to such a min.

 

Ken

[Admin]

hehe...yeah...always on a weekend, when you are away and all you have is your cell phone and laptop. Great fun

 

But NOW I am off to bed.

hehey...my avatar just came back OK, NOW I am really going to go to bed!

[hiker01]

Damn......I thought ZK finally did his thing......so much for RAID? When you have it figured out, please share your experience, it would be nice to know for us GEEKYS!

[sgtguk]

Fuck me thank god    do you know how much gardening I got through yesterday..................fucking hate gardening, still the wife was happy     Thanks for getting the site back up again and saving me from more chores today! 

[Aero_Dave]

Thanks for all your hard work to recover the site quickly.

As a IT guy, I understand how bad things can get quickly and how hard it

can be recovering data.

Thanks,

Dave...

[cougar1]

I learned how to bake a cake, and change nappies, fucckin awful stuff like, don't ever go down again

 

Good work admin!

[panerai153]

Thanks very, very much for your hard work. I'm glad that we are back up and running.

[billyfras]

Good work getting  the site back up. Maybe you could start a twitter account for the site so we could get updates whenever stuff like this happens? Just an idea.

[jpz5142]

WhooHoo! Now I can get back to feeding my addiction!

Thanks Admin!

[Rolexman]

Thanks for all the effort. Glad we Are up and running again.

[markiemark]

Im not trying to be a smart ass, but this should not have happened with raid...

But thank you for your time and effort! I finally can spend my days again here!

Thanks!

[SSTEEL]

Good to be back, thanks for the update, I too was wondering what had happened.

[KB]

@markiemark the boss will be back to explain later but the problem arose from the hosts not setting up everything they way our Admin asked them to.

 

 

Ken

[martijnp]

I was wondering what was going in but great to be back online 

[Deckard1979]

Thanks for Your hard work. I thought that the Ariel guy got us..

[chronology1066]

Glad to see things getting back to normal. Thanks to T for all the effort, gonna have to get you a 4G dongle just in case

Also to echo other member it would be interesting for the geeks, techs and BOFHs to know what happened in a bit more detail.

[indywatchguy]

Thanks admin! Great job!

[fraggle42]

Big thanks for your efforts in getting the site back up and running, a dead RAID array is really no fun at all. Been there, done that. (Of course, at the blasted weekend)

I thought RWG had been DDOSed off the face of the web, thought I'd never talk to any of you again!