We’re back up and running as you can see. Our hosting provider’s server farm was hacked and much damage was done. As a result we’ve been offline for more than 48 hours
My provider is Mungo Studios, a small outfit run by a friend of mine from my days at Microsoft. The server farm is located in a big data center near Chicago. My friend Tim owns several machines, which are in racks along with a few thousand others. For a small web hosting outfit, it’s more profitable to own the machines than to rent space on the data center’s machines like many host companies. He pays more up front, but has a slightly better margin on the back end. Everything is striped of course, and if some piece hardware goes bad, there is somebody there 24×7 to replace the parts and restore the data.
In this case there was an attack that reached several servers, it was recognized by the IT staff there, which shut down several racks of machines, Tim said he was not the only one effected, but doesn’t know how extensive it was. Looks like the software that manages the data center was affected, since it wasn’t just data on customer websites that was destroyed, user accounts were deleted as well. When that’s the case you can’t just log-in to your server and run restore from your backup, you can’t even ping your server.
For those that don’t run their own domains and websites, you are responsible for managing and securing your own server space. Management is done through C-Panel, where you setup and administer your mail server, file management and security settings. Of course you are also responsible for backing up everything on your site.
In this particular attack, the server admin was targeted. If it was just my own site, I could easily log back into the server, deleted some or all the data, change passwords, and restore everything from a backup. Worst case, down a few hours at most. In this case the damage was a few layers higher. The host provider accounts were damaged, and many user accounts were deleted.
So bottom line is, accounts for the host clients have to restored first, then all their user accounts. That means individual machines have to be wiped and the admin software reinstalled. Whoever runs that data center did a good job in limiting the damage and rebuilding. By this morning I could get back onto C-Panel, so my account was restored, but it took several hours for the actual files and databases to be restored from the previous weeks backup tapes. Remember, they are restoring from the server farm backup, not my individual website backups.
I’m not positive on this point, but I think the data center has a rolling backup strategy, so each server gets backed up once a week or so. That’s more often than I backup my site, so I waited for that.
One extra tidbit, even though it wasn’t a user’s problem that created this mess, you have to wait your turn for your website to be restored. You can pay extra to move ahead in the line, that costs $200 /hr. Since my site is not a business, no value there, so I was at the bottom of the list. It’s good that they do that, it’s not in any contract that I’m aware of. If your data disappears, it’s all on you. Of course the first thing I did when it came back on was to do a fresh backup. It’s a highly technical procedure called “Closing the barn door…. ” You get the picture. Even though in this case having a fresh backup would not have mattered.
At the end of the day, you have to ask why would anyone go to all the trouble. There are kinds of motives for bad behavior. It was suggested that the miscreants were taking of sites to use their mail servers to send spam. There are lots of affiliate scams, where they send spam email, if someone clicks on the link, the hacker installs an Amazon cookie. If you then buy something from Amazon, the commission goes to the hacker. On a large scale, that’s big money.
Personally, I think the death penalty would be appropriate.
There was some additional downtime Sunday morning. The data center upgraded the server operating systems, MySQL, PHP and Apache services to the latest versions. I addition, I changed my account names and passwords, both for WordPress in C-Panel. Hopefully that will hold back the barbarian hordes for awhile.