Admin Posted January 11, 2011 Report Share Posted January 11, 2011 Finally back online, but unfortunately on a week old backup (Well, since the board went down on friday and the backup is from tuesday, it's more like 3 days old) I've still got a bunch of errors to figure out, but never in my entire computer life have I ever had this run of bad. I'll get back to you with more details when I find the time, but just to give you a hint; The server died on Friday because of a fire in the server Link to comment Share on other sites More sharing options...
alligoat Posted January 11, 2011 Report Share Posted January 11, 2011 Good to see you back, T! Fire in the server, that's way beyond me..., it's good to finally have RWG up and running again- even if you are double posting! Link to comment Share on other sites More sharing options...
a german in denmark Posted January 11, 2011 Report Share Posted January 11, 2011 Great, our beacon of stability is back online! Good work, mate... Volker Link to comment Share on other sites More sharing options...
chubbchubb Posted January 11, 2011 Report Share Posted January 11, 2011 Hey big boss. Good to see the board up and running again. I was developing cold turkey.... Link to comment Share on other sites More sharing options...
Migge Posted January 11, 2011 Report Share Posted January 11, 2011 Great work Admin! Glad its up and runing again! Link to comment Share on other sites More sharing options...
TeeJay Posted January 11, 2011 Report Share Posted January 11, 2011 Thanks for all the hard work to get things back, T It's good to be back Link to comment Share on other sites More sharing options...
valerian Posted January 11, 2011 Report Share Posted January 11, 2011 Burning server?! Sounds bad. Anyway, thanks very much for your hard work, T! I felt like standing in front of my home while hearing somebody redecorating the interiors inside. Valerian Link to comment Share on other sites More sharing options...
panerai153 Posted January 11, 2011 Report Share Posted January 11, 2011 Thank you,Thank you ,Thank you!!!!! It's good to be back. Link to comment Share on other sites More sharing options...
krpster Posted January 11, 2011 Report Share Posted January 11, 2011 Wow. I heard of burning CDs and burning DVDs but burning a server...that must take forever Thanks for all the hard work. Looking forward to the story. Link to comment Share on other sites More sharing options...
hackR Posted January 11, 2011 Report Share Posted January 11, 2011 thanks, T!...yikes...a fire?!... nice to be back - HEY EVERYONE!... R- Link to comment Share on other sites More sharing options...
Bike Mike Posted January 11, 2011 Report Share Posted January 11, 2011 Now I have some thing to do at work all day. Link to comment Share on other sites More sharing options...
z3k0 Posted January 11, 2011 Report Share Posted January 11, 2011 Yipii! Already started to worry :thumbsupsmileyanim: Link to comment Share on other sites More sharing options...
vlydog Posted January 11, 2011 Report Share Posted January 11, 2011 Great to be back home! Thanks for all the hard work in getting us here. Link to comment Share on other sites More sharing options...
subbiesrock Posted January 11, 2011 Report Share Posted January 11, 2011 And here I was thinking mont riche had finally had enough of the new Cartel 111J and 288K super reps, Concepta Frankens etc and thought it was time to take it all down Good to have RWG back, cheers to the admins Link to comment Share on other sites More sharing options...
KB Posted January 11, 2011 Report Share Posted January 11, 2011 T said to me earlier today that I wouldn't believe the problems he has run into to get us back online. My answer.. "Knowing how good you are at what you do I don't have the slightest doubt of the problems" I think it's fair to say we are all very lucky to have to have him at the helm. Ken Link to comment Share on other sites More sharing options...
modelizer Posted January 11, 2011 Report Share Posted January 11, 2011 Oh boy, am I happy to see this place alive and kickin' again!! Good to see you all! And a big shout out to our brave firefighter Admin! Link to comment Share on other sites More sharing options...
lhooq Posted January 11, 2011 Report Share Posted January 11, 2011 Ahh... Feels like coming home! Great work, Dear Admin! Link to comment Share on other sites More sharing options...
Dizzy Posted January 11, 2011 Report Share Posted January 11, 2011 WOO Hooo!!! Link to comment Share on other sites More sharing options...
capice Posted January 11, 2011 Report Share Posted January 11, 2011 I was online when it went down,.....smoke coming off my keyboard. Fire in the server, I hope you all are ok? Regards for you T, it's becoming a fulltime job so it sounds to keep this running....I will take the liberty of posting a thread to collect some cash for the destroyed server Good luck buddy Link to comment Share on other sites More sharing options...
Admin Posted January 11, 2011 Author Report Share Posted January 11, 2011 OK. I have some stuff that is running in the background that I need to wait for, so here are some more details; When the site went offline Friday, I was going to contact the datacenter where our servers are located, but they beat me to it and contacted me (which is not good, because either I owe them money or there is something VERY wrong somewhere) Well, I knew I did not owe them any money, so this had to be bad news and it was; There had been a fire in the rack where our servers were located. The fire was quickly put out, but not quick enough. Before I continue, here is a little background info for you guys. We run separate web and database servers to speed things up a bit. My regular setup is at least 3 hard drives per server; 2 drives in a RAID1 setup (this will create identical content on 2 disks, so that if one fails, it can easily be replaced and all content is still available on disk 2 and when a new disk is inserted, all content is synced to both disks again) The 3rd disk is used for our nightly backups. All content is then archived (this is why the servers are so unresponsive sometimes. It takes a bit of computing power to put all our content into an archive) and stored on the 3rd disk. On top of this, I also rsync the archived backups on disk 3 to a separate, off site location. OK, so we had a fire in the datacenter and lost all of our servers. "No biggie. I've got backups" Well, unfortunately all of our disks were somehow damaged, and it was simply not possible to get any content from any of the disks. Again..."No biggie. I've got backups". And this is where the "fun" starts. (I seriously thought everything was lost forever when the next thing happened). When I started to restore the latest backups, NONE of them worked. Not sure why, but all backups failed file checks and the archives were corrupt. So basically that means I had no backups left. Bad luck like this should not be possible.....A backup setup like this is usually bulletproof...Our techs at the datacenter had never heard of anything like this! But then Lady Fortuna smiled to me; On Wednesday I downloaded one of the main backups from our server to get a couple of files from it and it was still on my home computer. YAY! So to be on the safe side, I made several copies of this file and started to upload it. Now, this file was about 52GB in size (Every daily backup file is this big!) and I can only tell you that it takes a loooong time to upload such a file on a 5MB outbound connection. After starting the restoration this morning, my time, and running into a few minor problems with the database, everything was once again working again. All content posted since Tuesday is gone and this also goes for PM's and any other "stuff" that you have managed to put on our server. So if you sent me a PM and I did not fix it, PM me again. If you posted something funny, interesting, etc, post it again Oh, and if you upgraded your membership these days, PM me or e-mail me at admin@rwg.cc to get the upgrade applied to your account again. Remember to include your PP-address or transaction number in your PM/email. Link to comment Share on other sites More sharing options...
TeeJay Posted January 11, 2011 Report Share Posted January 11, 2011 What a series of events, but thanks for sharing, T, it really helps to appreciate just how much hard work goes on behind the scenes to keep this place running Kudos indeed Link to comment Share on other sites More sharing options...
Stephane Posted January 11, 2011 Report Share Posted January 11, 2011 Fire in the server, I hope you all are ok? Yes, we are. KB has his arm burned, but nothin serious Link to comment Share on other sites More sharing options...
freddy333 Posted January 11, 2011 Report Share Posted January 11, 2011 Backup loss is actually not all that uncommon. I had a similar problem on a BSD server a few years ago (server got fried & main set of backups were corrupt). Glad to see that good old ingenuity prevailed. Link to comment Share on other sites More sharing options...
coolfire Posted January 11, 2011 Report Share Posted January 11, 2011 Yes, we are. KB has his arm burned, but nothin serious Did he??? On a more serious note.... After reading T explain, I am in awe of the amount of work which goes on behind the scenes... Thanks T & gang, for letting me come home again... These couple of days seemed like years, really was lonely.... Link to comment Share on other sites More sharing options...
ExtraExtra Posted January 11, 2011 Report Share Posted January 11, 2011 Good to see RWG back up and running. Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now