Shoot the breeze, anything goes.
User avatar
!
30%
Posts: 3258
Joined: 2013-02-25 18:36

2015-12-09 21:36 »

OH DEAR GOD, WE DODGED A BULLET AND WE ARE BACK ONLINE!

You have no idea what hell I went through in the past two days! I'm just glad we are now back online again! I will post all the details here soon, I'm going to eat some food first. HAVEN'T HAD FOOD FOR ALMOST TWO FUCKING DAYS! :sad:

User avatar
!
30%
Posts: 3258
Joined: 2013-02-25 18:36

2015-12-10 00:32 »

So... for some reason, the motherfucking Windows Update gives me two optional stuff "SUSE Network driver" and "SUSE Storage controller - block driver" inside our VPS. I knew I shouldn't install them but I had them marked to check them, you know, read the extra information to see what they do, long story short, I clicked "install" and didn't realize this error until after the server rebooted...

...and BAM, just like that, Windows was gone! It went into a reboot loop, stupid piece of shit wouldn't even go into "Safe mode". Having the "C:" drive encrypted by BitLocker, Microsoft didn't think of this at all that OH, WHAT IF THE FUCKING THING BREAKS. No way could the Windows be repaired because it kept nagging the drive is locked but it never gave any option to type the password to unlock the encrypted drive.

Poor men that we are, we can't afford two servers (we used to but I lost my job so yeah), so our only server just fucking dies! I'm a complete moron so I didn't take a backup or a snapshot before I did the Windows Update which I really blame on my cough medication (I have been having this bad cold and been coughing my lungs out a few weeks), so I couldn't take out the files!

DOOMED!

I call the hosting company "Tilaa.com", they tell me because BitLocker isn't a "supported scenario" (WHATEVER THE FUCK THAT EVEN MEANS) they wouldn't be able to help me! So I thought OK, they want money. I called up and said look, help me just attach this drive to another VPS so I can just unlock and get the files... what do they answer? "WE DON'T HAVE TIME FOR THIS, WE ARE TOO BUSY!"... I literally asked "What does that even mean?" "What do you mean you don't have time?"... then he was like "It's not possible to do!"... and I was like "I have been a systemsdeveloper for 20 fucking years, I KNOW it's possible! What are you talking about?"...

...I got angry and asked to talk to a manager, some other guy came on phone and I think he realized I'm a pissed off customer, he was like "Oh no, I mean I don't have time 'NOW' but maybe later"... *lol* WHAT THE FUCK? I guess we aren't a big enough customer for them to give two shits about us. I kept on the phone and kept asking come'on, this is important, I need my files etc. He mellowed a little and then I suggested OK, fine, just give me a raw copy of the disk so I can put it up my ass somehow and fix it myself... he posted some piece of shit crap bullshit Linux commands in the support ticket... I was like "WHAT THE FUCK?"...

...his commands were not even good! He wanted me to use "dd" (Gentoo) to SSH into a fucking Windows, God damn it! Then I said, look man, just fucking do this and they said OK fine but we will charge you for 1 hour €89 euros!!! I said FINE, JUST DO IT. Hours went by and no respond. Another guy replied to the ticket and said "We will get back to you when this is done." LOL! Hours went by and nothing. I was so depressed that I went into bed and just fell to sleep. I wake up, business hours are finished and I check the ticket, NOTHING! They didn't do shit yet!

OH MY GOD, I WAS PISSED OFF BUT ALSO VERY VERY VEYR SAD. So I sat there, easy peasy, mounted a network drive to the new VPS with "mount -t cifs //<ip>/<shared folder> root/somefolder/ -o rw" and then I read up a little (few minutes) and realized that "ddrescue" is a better command to use so I did a "ddrescue -d /dev/vda1 /root/somefolder/disk.img". It took maybe 30 minutes and DONE! I had an image from my disk.

Then I had to found an old tool (WHICH FUCKING MICROSOFT FOR SOME FUCKING REASON REMOVED FROM THEIR WEB SITE) called "VHDTool.exe" to convert the image into a ".VHD" file. Something like "vhdtool.exe /convert disk.img" and it makes the same file into a ".VHD" (doesn't create a new file). Then from within Windows, I just mounted that (renamed from .IMG to .VHD) and yiiihaaa, it asked for the BitLocker password... aaannnddd... it was unlocked! All the files were there! :mrgreen: :clap: :thumbup:

I also add the "vhdtool.exe" here because I'm sure it will save somebody, someday.

vhdtool.zip
(32.51 KiB) Downloaded 324 times

vhdtool.exe
(61.5 KiB) Downloaded 322 times

So I emailed them back and told them I don't need their help, it's fixed. Good thing, they didn't charge me the €89 euros but I had to get an extra VPS to move these files and add extra harddisk space so overall, I have lost about €100 euros.

Two lessons:

1. Never go lazy on the backup and snapshots! (I knew this and we have good backups but because I have been really down about me losing my job, I didn't really pay much attention to the backup, YES, SORRY!)

2. Never ever, NEVER EVER, trust the support for ANY hosting company NO MATTER HOW GOOD THEY ARE. Granted, "Tilaa.com" is really good. I mean, REALLY good. Usually great service but most importantly, GREAT PRODUCTS... but you know, when you are in state of panic, remember that NOBODY will ever care about you when you need help.

Make sure to cover your bases and if you can afford it, have your backup server at a different location AND company than your main one.

Oh and of course, ALWAYS make sure that your restore procedures are in place as well! Once I got the files out, we were up and running within MINUTES! :grin: The only thing I lost is the SSL certificate's private key so I will have COMODO to issue a new SSL file for me. Other than that, we lost ZERO files. :mrgreen:

Last but not least, please accept my apologies for the downtime of about 24 hours.

Post Reply