Tuesday, September 18, 2007

The silver bullet for zombie mongrels

Ok ok, so they are technically not zombie processes, but how could I resist such a great title.

I'm talking about mongrels that just stop responding to new requests. They are still up, ps does not tell you anything is wrong, monit thinks they are chugging along just fine, you can even telnet to them. But try to get a http response and it times out. Restarting the server fixes the problem every time.

This error state seems to be triggered randomly, however we saw it more with a overburdened mysql server running on the same box as the mongrels.

The trick, install the C based mysql driver. It's that simple, don't question it, just do it. Then curse yourself for not doing it sooner. You may even want to unleash a volley of nerf missiles on your coworker for claiming that we were using the C based driver all along.

Run this from the root of your rails app:
./script/console
Then this:
>> require_library_or_gem 'mysql'
If you don't see this:
=> ["MysqlField", "MysqlError", "Mysql", "MysqlRes"]
Unleash the fury.

How do you install the C based mysql driver?

1. Pray to your deity of choice that all required packages are installed.
2. Offer to trade the life of an office plant for a package management tool. (rpm is not package management, it's the root cause of anger management, see the difference?).
3. sudo gem install mysql
4. Replace that ficus you killed before office mates start asking questions.
5. Restart you rails apps

You do not need to change the rails environment.rb. If the gem is present, rails will find it.

If your god has forsaken you in steps 1-5 above just go grab the mysql RPM's your going to need.

Depending on distro, flavor and fat content, you will need some form of:
Headers and libraries
Shared libraries

Which can both be found on the mysql site.

Happy hunting.

No comments: