I would normally do something like this in Dapur. But heck, it is down now. I know how it feel to have our own blog down. We’ve been working it out on Sunday morning.
It will be up sometime soon. Take it as matter of faith, Maia.
Now, the moment of truth. Blogdetik is infant. We installed blogdetik in a crappy old server about 6 weeks ago for internal blog (I think you should write this fact). And stir was changed. We decide to give it a try for public. After several hacks, that leads to another hacks, we did it. Same old crappy server, 2 new database servers (master-slave), and 1 LDAP server to store account information.
Blogdetik was online at Feb 1st 2008 10.00 PM.
Maybe 9.00 PM.
Maybe 8.00 PM.
You know how unpredictable DNS stuff is. However I knew that it was online since I checked it with my PDA in Taman Menteng that night. Rest of team were working overnight just to make sure everything went allright.
And sleepless nights are still part of our agenda for next couple of days to come.
Last wednesday we replaced that crappy old server with a decent one. It is not good enough, Opteron something with 8GB of memory didn’t help much. We’ve been suffering downtime every morning since Thursday. 2 httpd process take entire processors and then eventually, hungs up application server. We did prefork tune. We put XCache. We give WP-Super-Cache a try. Latency tests are good. Capacity planning are worked out thoroughly. Several stability pattern/antipattern also addressed correctly. Of course, we did compile PHP and Apache ourselves to gain 64bit advantage. Still, these two httpds are such pain in the ass. There’s nothing in pstree. Error log don’t help much.
Sigh. I miss doing post-mortem server analysis in enterprisey JVM through thread dump. If anyone happens to know how to perform httpd thread dumps autopsy, raise your hand please.
Lies, Damn Lies, and Webtrends
Above all, our biggest problem is this: Blogdetik is sold out. Like selling free candy in kindergarten. Free candies run out, while demand stand still.
Blogdetik is chasing Detikforum, Alexa said. Climb, you fool!
Report Range: 02/01/2008 00:00:00 – 02/29/2008 23:59:59
Number of blogs: 2042
Number of users: 1712
Average Page Views Per Day: 44,348
Unique Visitors: 45,491
Visitors Who Visited Once: 30,201
Visitors Who Visited More Than Once: 15,290
These numbers are still small compares to “One million blogs in 2008”. This is not our target, though. We get that somewhere from a great idea created by you.
What do you guys call it, PestaBlogger?
What’s Next?
The idea is to give more candies. Also known as vertical scaling.
Before that, however, I would lie, cheat, steal, suicide, and sell my soul to demon just to get rid of httpds problem first. Soon after everything seems fine, we will moved out to nginx.
Nginx is small-footprint webserver (It has 2MB memory footprint whilst Apache eats 20MB running altogether in my small MacBookPro). It is now our standard Ruby on Rails deployment strategy (i.e: our Adpoint servers). Soon-to-be standard for all deployment platform in our system architecture (except several JVM app server node clusters). As the matter of fact, progressive download in DetikTV and one of DetikNews mirror, since last Monday, are now nginx.
Every network engineers around here are just amazed by nginx. I hope it is also to be a remedy to Blogdetik problem.
Okay now. GTG.
Our God(s) are crying out loud again. Sit tight, and I’ll be back next month for future report.
Update:
It appears that we didn’t properly configure mod_rewrite thingy. Each request yielded multiple threads which was eventually bogged down httpd (Hints: MaxRedirect directive). Silly us.
Blogdetik is now stable, and scalable. httpds sit in 200-300MB memory footprint with XCache enabled. Although it still doesn’t match my personal speed expectation, I believe it is already fast enough for most people. Enjoy!
