Massive #FAIL: Gawker is still down.

So Gawker and it’s family of sites (Jalopnik, Jezebel, io9, Lifehacker, etc) has been down since Monday. This is a site that does between 30-40MM uniques a month. They’ve since shifted to a bunch of tumblr blogs and I even saw a former Gawker writer praise them for monetizing their new tumblr existence with State Farm ads. This is all sorts of crazy and more importantly, it just didn’t need to happen.

(Disclaimer: I don’t work for Gawker, never have had any involvement with them, and have only been following this with some interest because it was so avoidable. Some of my clients were in the direct path of Sandy and while we didn’t experience any downtime, we were prepared.)

Apparently they’re hosted in a data center in downtown Manhattan, which still lacks power, the lower floors flooded, and their pipes might even be cut. BuzzFeed, HuffPo and even Fog Creek all experienced some amount of outage as well, yet it was brief.

Rather obviously, Gawker lacks failover. I’m sure some sort of local redundancy exists, multiple webservers, dB boxes and the like. Maybe even switches. These are more likely in place for load reasons than redundancy given that they’re still down today.

Now, Gawker was hacked back in 2010 which resulted in the release of their source code and database. I’ve not looked at it personally (nor should you ever admit to doing so) but I’ve seen some analysis and it’s a fairly straight-forward set up of PHP and MySQL. This provides some valuable insight. There’s no one-of-a-kind appliances or ornate setups in the mix. It’s basically code + dB, like most sites.

There’s a million mitigation strategies one could use to allow geographic failover without any downtime or data loss. Database clusters, for example. I won’t go into those details here. I’d argue a site as large as Gawker should be using them, but they do increase hosting costs (obviously) but that’d be marginal for the cash-cow that is Gawker. But of course such a plan wasn’t in place. I think it’s rather obvious they didn’t have a fucking clue as even what to do given the days of warnings about what was to happen. The news didn’t overplay this one.

What they should of done (and this is the “they didn’t prepare shit until the skies went grey out the window” scenario):

• When the shit started getting real, put the site into real-only mode.
• Dump the DB and anything else associated that might be user/editor-generated content. (Images, for example.)
• Move the critical data off-site into something safer, say S3.
• Have EC2 instances (or similar) ready to become your backup webservers and databases boxes. This costs almost nothing if they’re not actively running. They’re simply sitting around as AMIs ready to be launched. (And considering we’re 5 days out, they could of even started from scratch and accomplished this on Monday night.)
• I believe most of the data centers warned when they were about to go caput, given their generators were flooding. Spin up your backup instances now. (Better yet, move to them before the inevitable happens as everything below 39th is rapidly becoming part of the east river.)
• Bring your code up to date by pulling from your code repository or using the backup from your primary boxes.
• Load in the latest dB snapshot.
• Change IPs to point to the new site.
• Resume Lohan updates and snark funnel.

So shall we call it incompetence? Probably. That’s completely fair 5 days out. That didn’t bother State Farm apparently but I’d guess advertisers and even employees are wondering why they deal with a place that treats their core product with such lax concern.

GoDaddy Refugees: hover, namecheap, name, etc

GoDaddy’s reprehensible: misogynistic advertising, elephant-killing cartoon villain CEO, difficult interface designed to confuse, and of course SOPA support. They’re the evil credit card company of the domain industry, full of tricks n’ traps. And don’t let the eleventh-hour SOPA change of heart deter you: it’s time to switch. (That last minute SOPA change of heart somehow makes it worse, and then there’s this.)

I’ve used name, namecheap, and hover. They’re all more than adequate, although I’ve found hover to my favorite. It’s not the cheapest, but they’re close enough and they’ll save you the funny business, sport an intuitive interface and forgo tricks n’ traps.

Save yourself a few bucks — a few Hover coupons:

• 20EB24BM4PO (save 17%)
• SOPA (save 10%)

Pro tip!

Enter your new DNS settings as soon as you initiate the transfer rather than waiting to afterwards. This insures against possible downtime.

Tear Down the Wall

I mossied on over to Amazon’s music store today, the latest iTunes “killer,” and I was buying within a minute: my decade old rip of The Wall needed updating, and Amazon wanted me to have it DRM-free, both discs, for $8.99? Okay, sold. (iTunes: $16.99)

Beyond that, I don’t have much to say, dear Amazon. DRM-free? Great. 10c cheaper for most tracks? Great. 2M vs. 6M songs? Well, you can work on that. iTunes vs. a browser and some proprietary software for anything more than a single track? A decent job, I’ll admit, but I’ve already downloaded iTunes. At least my tunes are copied into iTunes.

I’ll be watching. Lord Steve is likely fairly displeased, given the nearly 70% discount on DRM-free tracks and recent spats with the likes of NBC-Universal.

I do find it amusing the #1 song is Feist’s 1234. Yeah, I don’t know who the hell Feist is either, but my mad internet skillz has revealed it be an obscure Canadian emo-indie band featured in, hang on here, the latest iPod commercials.

Technorati Tags: amazon, amazonmp3, apple, ipod

DRM

[ via TorrentFreak ]

Craigslist: no IKEA option plz

For sale is now separated by owners/dealers, cool and all, but it sorely needs a 3rd filter: No Fucking IKEA. It’s inescapable pressboard land now.

At right: some sad fag chases after $15 for the 2nd week in a row.

Things I Don’t Believe

I believe the upstream speeds, but those downstream speeds are kinda nuts. (Reference: typical DSL is 1500Kbs/s, my “enhanced” cable should peak at 8192Kb/s.)

Comcast in SF? What are you getting?

Technorati Tags: comcastic, comcrapstic

Sweet Jesus, I hate NetworkSolutions

…they held one of my domains hostage and sent me through so many hoops, wasting so much time, that now my only option this far out is some bullshit $150 “redemption fee.” What a racket they’re running — no wonder they’ve lost all of their domain business.
For the next several days, I’ll fantasize about revenge and talk to lawyers. I don’t think my urge to kill will subside fast enough for me to let this go onto the back burner.

Now with PowerBoost!

So in what’s perhaps an unholy union between Comcast and Jamba Juice, Comcast has started this thing called PowerBoost. Supposedly this doubles your speed during large downloads — only for short bursts, and only given “network availability.” I *think* I’ve seen this, but am I the only one calling horseshit on this? No way to really verify this or see when you’re getting “PowerBoosted.” No guarantee if it’s gonna kick in during that 123MB download of OpenOffice. It’s gonna skew network speed tests and the average download speed given by your browser. And it’s undoubtedly going to be used to quell the piles of customers complaining about less-than-steller speeds, especially the craptastic upstream speeds (which are Boost-free).

Some marketing weenie at Comcast deserves a promotion. Jamba Juice should sue.

(If this is already deployed in San Francisco, it’s having the desired effect — I’m testing north of 18Mbps right now, but my upload speeds? 690Kbps. C’mon Comcast.)

Plogs

Amazon’s Phabulous Pheature with the Phunny Name has been relegated to the backseat. I previously lamented their lame-o take on blogs, and they’ve been moved off the front page to a link at the top of the page.
Perhaps premature, but Plogs, god-willing, are headed for the trash bin. And good riddance…
Unfortunately, links to Glenn Reynolds and James Lileks are now 2 clicks off the main page vs. the previous 1. Idiocy should not be so prominent, but maybe they’re smarter than I as that vicious hack piece against Kerry’s Vietnam record is currently #1 in books. (I recommend #5, Maureen Dowd’s Bushworld)
Oh, and hi Tim!