In January, Laptoplogic.com submitted a guest post and I posted it. Since then, the post has seen a tremendous amount of traffic, most of it from StumbleUpon.com. The traffic the post has been bringing has been a nice increase from my normal numbers and I’ve been enjoying it. Yesterday was a whole new ball game.
What happened?
Yesterday morning, I was running through my morning routine, which includes a visit to by blog admin area to delete spam and approve valid comments, but it didn’t come up. Whenever this happens, my stomach turns just a little, then I check things out to see how bad it is. The first thing I do is SSH in and poke at logs, etc. This is what I saw:
Upon further inspection, I found that the server was trying to handle about 150 httpd requests per second. I looked at my stats page on MyBlogLog because there I can quickly see the source of a lot of traffic. The source was an article called “64 Things Every Geek Should Know“, which had over 1,200 diggs by time I saw it. The article linked to the guest post I mentioned above. It was the 5th of the 64 things.
Staying online
Keeping the site up was a bit of a struggle. I’m told that no single-server site can survive the “digg effect”, but I did everything I could to keep things going as smoothly as possible. Regardless, tons of people were met with database connection errors or just server timeouts as I tweaked whatever I could to make the server feed web pages to as many people as it could. At the peak of the traffic, I copied the popular post into a new static html page and redirected all traffic for that post. This was done to eliminate the overwhelming amount of MySQL database queries and additional http requests that were previously resulting from being within WordPress. This worked to some extent, but it just resulted in more people seeing less of the site and the server still struggled to keep up. With nothing more to do, I just monitored the server as best I could as I watched the article that linked to me hit the number two spot on the Digg.com homepage and surpass over 4,000 diggs pretty quickly.
Residual Effects
When an article hits the front page of Digg, a common side effect (on top of the surge of traffic) is a lot of inbound links from those Digg readers either on their own sites or in social media. For example, Guy Kawasaki posted the “64 Things” article link on his Twitter account with over 100,000 followers, it turned up in a tweet from @mashable (almost 500,000 followers), and according to Retweetist, he’s not the only one who tweeted it. In fact, according to Twitter’s real-time search, it’s been retweeted 20 times in just the last half hour. The numbers so far look like a little over 26,000 readers yesterday alone. That’s just a little more than my monthly unique reader count. Because I put the post in a flat html file without Google Analytics, I don’t have proper numbers for yesterday from Google, but I do have my local stats which show a clear rise in traffic:
To show the residual traffic effects, I’ll update this image tomorrow. So far, today’s traffic is looking way up still.
What I’ve learned
There’s one really important lesson I’ve learned from this, and it’s that I need to be much more prepared for the unexpected. This was just a link from a post that has hit the front page of digg. If it were an article on my site that hit the front page itself, the results would have been catastrophic to manage. As it is, the traffic received was far more than my server was prepared for. To prepare, I must find an appropriate solution that lets me still maintain just the one server (more than enough for normal operations), but that can somehow immediately live up to the performance needs of the Digg effect when it happens. My first thought was something like Aptana’s Cloud, but I want to talk to my current host first and feel out my options.
UPDATE: It seems the guest post has now made the Delicious “popular” page.
UPDATE 2: It seems the guest post also made the Delicious home page today.
Related posts:
If you liked this Joe Tech article, subscribe to the RSS feed for daily updates.
