Massive #FAIL: Gawker is still down.

So Gawker and it’s family of sites (Jalopnik, Jezebel, io9, Lifehacker, etc) has been down since Monday. This is a site that does between 30-40MM uniques a month. They’ve since shifted to a bunch of tumblr blogs and I even saw a former Gawker writer praise them for monetizing their new tumblr existence with State Farm ads. This is all sorts of crazy and more importantly, it just didn’t need to happen.

(Disclaimer: I don’t work for Gawker, never have had any involvement with them, and have only been following this with some interest because it was so avoidable. Some of my clients were in the direct path of Sandy and while we didn’t experience any downtime, we were prepared.)

Apparently they’re hosted in a data center in downtown Manhattan, which still lacks power, the lower floors flooded, and their pipes might even be cut. BuzzFeed, HuffPo and even Fog Creek all experienced some amount of outage as well, yet it was brief.

Rather obviously, Gawker lacks failover. I’m sure some sort of local redundancy exists, multiple webservers, dB boxes and the like. Maybe even switches. These are more likely in place for load reasons than redundancy given that they’re still down today.

Now, Gawker was hacked back in 2010 which resulted in the release of their source code and database. I’ve not looked at it personally (nor should you ever admit to doing so) but I’ve seen some analysis and it’s a fairly straight-forward set up of PHP and MySQL. This provides some valuable insight. There’s no one-of-a-kind appliances or ornate setups in the mix. It’s basically code + dB, like most sites.

There’s a million mitigation strategies one could use to allow geographic failover without any downtime or data loss. Database clusters, for example. I won’t go into those details here. I’d argue a site as large as Gawker should be using them, but they do increase hosting costs (obviously) but that’d be marginal for the cash-cow that is Gawker. But of course such a plan wasn’t in place. I think it’s rather obvious they didn’t have a fucking clue as even what to do given the days of warnings about what was to happen. The news didn’t overplay this one.

What they should of done (and this is the “they didn’t prepare shit until the skies went grey out the window” scenario):

• When the shit started getting real, put the site into real-only mode.
• Dump the DB and anything else associated that might be user/editor-generated content. (Images, for example.)
• Move the critical data off-site into something safer, say S3.
• Have EC2 instances (or similar) ready to become your backup webservers and databases boxes. This costs almost nothing if they’re not actively running. They’re simply sitting around as AMIs ready to be launched. (And considering we’re 5 days out, they could of even started from scratch and accomplished this on Monday night.)
• I believe most of the data centers warned when they were about to go caput, given their generators were flooding. Spin up your backup instances now. (Better yet, move to them before the inevitable happens as everything below 39th is rapidly becoming part of the east river.)
• Bring your code up to date by pulling from your code repository or using the backup from your primary boxes.
• Load in the latest dB snapshot.
• Change IPs to point to the new site.
• Resume Lohan updates and snark funnel.

So shall we call it incompetence? Probably. That’s completely fair 5 days out. That didn’t bother State Farm apparently but I’d guess advertisers and even employees are wondering why they deal with a place that treats their core product with such lax concern.

Remember Josh Barro is a conservative

Sane gay Republicans?

GoDaddy Refugees: hover, namecheap, name, etc

GoDaddy’s reprehensible: misogynistic advertising, elephant-killing cartoon villain CEO, difficult interface designed to confuse, and of course SOPA support. They’re the evil credit card company of the domain industry, full of tricks n’ traps. And don’t let the eleventh-hour SOPA change of heart deter you: it’s time to switch. (That last minute SOPA change of heart somehow makes it worse, and then there’s this.)

I’ve used name, namecheap, and hover. They’re all more than adequate, although I’ve found hover to my favorite. It’s not the cheapest, but they’re close enough and they’ll save you the funny business, sport an intuitive interface and forgo tricks n’ traps.

Save yourself a few bucks — a few Hover coupons:

• 20EB24BM4PO (save 17%)
• SOPA (save 10%)

Pro tip!

Enter your new DNS settings as soon as you initiate the transfer rather than waiting to afterwards. This insures against possible downtime.

QotD

Richard Clarke on Olbermann referring to today’s Senate Intelligence Report on how the Bush administration lied us into war:

I just don’t think we can let these people back into polite society […] and just pretend this didn’t happen. […] Someone should have to pay.

Um, can I get an AMEN?

Jan-Michael Vincent was Passed Out After a Bender

Link:

Edward dedicates his time to caring for [his 1974 VW Beetle] Vanilla and composing car-themed love songs on his keyboard. He reveals that his taste for mechanical love extends to a fascination with aircraft – and even claims to have made love to Airwolf, the high-tech helicopter from the 1980s TV series of the same name.

Flying Penis

Chess legend cum politician Garry Kasparov is interrupted during a speech.

Cousin Fucking

And the results:

BRAZORIA, TX (KTRK) — A state trooper found a six foot alligator in the back seat of a burglary suspect’s car early Saturday morning.

The trooper told Eyewitness News that William Johnson found the gator in the road near Angleton and loaded it up into his Buick Regal, then broke into a house in Brazoria.

DPS Trooper Steve Stanfield explained, “From my understanding, he tried to get a big screen TV out of the house. He couldn’t get it in the car by himself, so he recruited a neighbor, who, when he saw the alligator, dropped the TV and said, ‘Alright, I ain’t got nothing to do with it.'”

Officers arrested Johnson and a game warden took the alligator.

Officers also told us on Friday, they found Johnson with a four and a half foot water moccasin that had bitten him. Johnson did not suffer ill effects from the snake bite.

In Texas. I know, I’m shocked too.

Happy Thanksgiving!

Memorable:

• Wednesday, half day, goodbye lunch for another employee which shifts ever more responsibility on to me. I will miss her though.

• About 4:30 that afternoon, I’m sitting in the ER having a massive panic attack. (No, the above isn’t a trigger, that scene is played out like twice a week and if I’m at peace with anything, it’s that.) While freaking at the ER — feelings of going crazy, imminent death, profuse sweating, and BP so high I’m unable to articulate full sentences, I wait an hour or two as a colorful parade of street drunks go right through the VIP line, no waiting required. Cuz I “look” normal, nervous pacing and rapid short breathing aside. I keep muttering to Mason, “just 1mg of Ativan and I can turn the corner.”

  

• Nurse finally calls. “Probably panic attack, just need a mg of Ativan,” I’m begging. BP machine doesn’t work; the 5th take finally gets a reading. Much to her surprise, it’s spectacularly high. As is my heart rate. Blood O2 low, too, from all the rapid shallow breathing. Hey, guess he’s not lying! I’m asked to pee in a cup and strip down and get into a gown. I’m ushered into a room.

• Space limited. Room divided by curtain, crazed ramblings coming from the other side. I try to remain as quiet as a mouse. I’m offered a Turkey recipe from behind the curtain, which I don’t even acknowledge. But to her mind-altered credit, she was topical, even if delivered as “Hey, fuck hey there, you, oh my fucking god, I’ve got the best, [hack hack hack], best fucking Turkey recipe.” I kept quiet. She was snoring less than a minute later.

• See Doc. Obviously knows his stuff. Two minutes later Ativan arrives. 2 more minutes and I’m starting to relax and my BP had dropped 40%.

• Given Rx for Ativan. Met with Doc no more than 1 full minute. Discharged, Mason’s obviously terrified about my well being, by taking a nap. I think he’s snooring.

• Since tomorrow’s Thanksgiving, force myself to go Turkey shopping, after a stop by the Walgreens pharmacy to stock up on Ativan.

• So… Thanksgiving was nice. Relaxing. Cooking on Ativan, fun, if not especially efficient. That’s my little 12-pounder to the right. Delicious!

Happy Thanksgiving, y’all!

Toothpaste Tubes

It’s official: the GOP is 100% gay.

After the two men apparently had sex, Castagna allegedly left the hotel room with Curtis’ wallet, the report says. Curtis agreed to give the man $200, which he left at his hotel’s desk, only to be confronted with a demand for an additional $800, the report says.

Spokane Police Detective Tim Madsen wrote in his report that Curtis wanted to keep the whole incident quiet. At one point, Madsen told Curtis that “the toothpaste was already out of the tube.”