Read a slashdot thread on Rails not scaling and now I need to weigh in from the safety of my blog. Let's do bullet points to help break down some of their arguments:
- PHP and Ruby are compariable
Sorry, you just can't do it. Ruby and Perl are comparable, Perl is probably better, who cares, let's move on.
- CakePHP is a great framework.
For me to poop on. Back before my Rails days I really wanted to like CakePHP. Seriously. I was a PHP ninja and the idea of MVC + data abstraction layer was a dream come true. But alas it was total crap. Weak community, annoying syntax (PHP's fault) and bugs. Next song.
- Rails is not thread safe
This is actually true, but rarely cited. Want to impress your friends, whenever they talk about Rails being slow because of Ruby, say "Who the fuck cares about Ruby? The Rails stack is not even thread safe you dumb bastard". What does that mean? It means a whole host of problems when it comes time to use your lovely self tested app. I"m not even talking about massive 2 billions requests per second sites, this will be bite you in the ass if you have 2 requests per second (given that one is significantly slower then the other).
- Ruby is slower then Python
So what, you think the bottle neck of your awesome facebook app is the CPU? How about I/O? You ever think of that? Full table scans, massive correlated queries, missing indicies, yeah, that's your problem chief. Unless you are doing some massive protein folding you better leave Ruby alone. Unless you are going to argue that Rails is not thread safe because of limitations in Ruby. Then you would be on to something.
- Google uses Java
Google is godlike in their programming ways. I seek only to collect what morsels of genius they donate to the community and not understand the terrible and mysterious ways they are contorting the web to their every whim. I am not google, you are not google, no one has ever been google and may never be again.
Wednesday, September 19, 2007
Tuesday, September 18, 2007
The silver bullet for zombie mongrels
Ok ok, so they are technically not zombie processes, but how could I resist such a great title.
I'm talking about mongrels that just stop responding to new requests. They are still up, ps does not tell you anything is wrong, monit thinks they are chugging along just fine, you can even telnet to them. But try to get a http response and it times out. Restarting the server fixes the problem every time.
This error state seems to be triggered randomly, however we saw it more with a overburdened mysql server running on the same box as the mongrels.
The trick, install the C based mysql driver. It's that simple, don't question it, just do it. Then curse yourself for not doing it sooner. You may even want to unleash a volley of nerf missiles on your coworker for claiming that we were using the C based driver all along.
Run this from the root of your rails app:
./script/console
Then this:
>> require_library_or_gem 'mysql'
If you don't see this:
=> ["MysqlField", "MysqlError", "Mysql", "MysqlRes"]
Unleash the fury.
How do you install the C based mysql driver?
1. Pray to your deity of choice that all required packages are installed.
2. Offer to trade the life of an office plant for a package management tool. (rpm is not package management, it's the root cause of anger management, see the difference?).
3. sudo gem install mysql
4. Replace that ficus you killed before office mates start asking questions.
5. Restart you rails apps
You do not need to change the rails environment.rb. If the gem is present, rails will find it.
If your god has forsaken you in steps 1-5 above just go grab the mysql RPM's your going to need.
Depending on distro, flavor and fat content, you will need some form of:
Headers and libraries
Shared libraries
Which can both be found on the mysql site.
Happy hunting.
I'm talking about mongrels that just stop responding to new requests. They are still up, ps does not tell you anything is wrong, monit thinks they are chugging along just fine, you can even telnet to them. But try to get a http response and it times out. Restarting the server fixes the problem every time.
This error state seems to be triggered randomly, however we saw it more with a overburdened mysql server running on the same box as the mongrels.
The trick, install the C based mysql driver. It's that simple, don't question it, just do it. Then curse yourself for not doing it sooner. You may even want to unleash a volley of nerf missiles on your coworker for claiming that we were using the C based driver all along.
Run this from the root of your rails app:
./script/console
Then this:
>> require_library_or_gem 'mysql'
If you don't see this:
=> ["MysqlField", "MysqlError", "Mysql", "MysqlRes"]
Unleash the fury.
How do you install the C based mysql driver?
1. Pray to your deity of choice that all required packages are installed.
2. Offer to trade the life of an office plant for a package management tool. (rpm is not package management, it's the root cause of anger management, see the difference?).
3. sudo gem install mysql
4. Replace that ficus you killed before office mates start asking questions.
5. Restart you rails apps
You do not need to change the rails environment.rb. If the gem is present, rails will find it.
If your god has forsaken you in steps 1-5 above just go grab the mysql RPM's your going to need.
Depending on distro, flavor and fat content, you will need some form of:
Headers and libraries
Shared libraries
Which can both be found on the mysql site.
Happy hunting.
Tuesday, September 11, 2007
When is / not division?
When you are using the wrong Ruby object.
Ruby seems innocent enough, but it's really doing a lot of 'object' work without you knowing.
Take the following rails example.
points = 0
points_possible = 100
points += quiz.points # points is 40, stored as an integer in the db,
percentage = ((points / points_possible) * 100) # evaluates to 0, WTF!
If we just give Ruby a little more direction by changing points to a float.
percentage = ((points.to_f / points_possible) * 100) # evaluates to 40, rock on
Ruby seems innocent enough, but it's really doing a lot of 'object' work without you knowing.
Take the following rails example.
points = 0
points_possible = 100
points += quiz.points # points is 40, stored as an integer in the db,
percentage = ((points / points_possible) * 100) # evaluates to 0, WTF!
If we just give Ruby a little more direction by changing points to a float.
percentage = ((points.to_f / points_possible) * 100) # evaluates to 40, rock on
Wednesday, June 20, 2007
Deploying Rails applications to multiple environments
I love capistrano, however I had some trouble finding solid doc on how to deploy your app to multiple environments (UAT, staging, etc.) So here is a quick and dirty of the solution I can up with.
cap deploy
The old standby, this will configure our deployment based on deploy.rb. We want to conditionally configure our deployment so we need to pass a parameter to this comment, this is done with -S
cap deploy -S stage=uat
This sets a local variable 'stage' to 'uat' in our deploy.rb
Here is our conditional code from deploy.rb
if stage == 'production'
set :application, "foo"
set :repository, "http://svn.agiledog.com/svn/agiledog/foo/trunk"
# Without this $RAILS_ENV on the destination server will be used to determine the server environment
set :rails_env, :staging
role :web, "app.agiledog.com"
role :app, "app.agiledog.com"
role :db, "app.agiledog.com", :primary => true
elsif stage == 'uat'
# Different application names are used to allow different stages to be deployed on the same server if required
set :application, "uat-foo"
set :repository, "http://svn.agiledog.com/svn/agiledog/foo/branches/RB-4.0"
set :rails_env, :uat
role :web, "uat.agiledog.com"
role :app, "uat.agiledog.com"
role :db, "uat.agiledog.com", :primary => true
else # assume app == 'staging'
set :application, "staging-foo"
set :repository, "http://svn.agiledog.com/svn/agiledog/foo/trunk"
set :rails_env, :staging
role :web, "stage.agiledog.com"
role :app, "stage.agiledog.com"
role :db, "stage.agiledog.com", :primary => true
end
We have added two new environments, staging and uat. Both of these will need to be added to the database.yml file. We will also need to create staging.rb and uat.rb to the config/environments folder to declare any specific environment settings.
One gotcha with this solution is that you will always need to set the stage variable anytime deploy.rb is read. That is to say, even though we assume the staging enironment, the following command will fail.
cap deploy
Instead you need to use:
cap deploy -S stage=staging
cap deploy
The old standby, this will configure our deployment based on deploy.rb. We want to conditionally configure our deployment so we need to pass a parameter to this comment, this is done with -S
cap deploy -S stage=uat
This sets a local variable 'stage' to 'uat' in our deploy.rb
Here is our conditional code from deploy.rb
if stage == 'production'
set :application, "foo"
set :repository, "http://svn.agiledog.com/svn/agiledog/foo/trunk"
# Without this $RAILS_ENV on the destination server will be used to determine the server environment
set :rails_env, :staging
role :web, "app.agiledog.com"
role :app, "app.agiledog.com"
role :db, "app.agiledog.com", :primary => true
elsif stage == 'uat'
# Different application names are used to allow different stages to be deployed on the same server if required
set :application, "uat-foo"
set :repository, "http://svn.agiledog.com/svn/agiledog/foo/branches/RB-4.0"
set :rails_env, :uat
role :web, "uat.agiledog.com"
role :app, "uat.agiledog.com"
role :db, "uat.agiledog.com", :primary => true
else # assume app == 'staging'
set :application, "staging-foo"
set :repository, "http://svn.agiledog.com/svn/agiledog/foo/trunk"
set :rails_env, :staging
role :web, "stage.agiledog.com"
role :app, "stage.agiledog.com"
role :db, "stage.agiledog.com", :primary => true
end
We have added two new environments, staging and uat. Both of these will need to be added to the database.yml file. We will also need to create staging.rb and uat.rb to the config/environments folder to declare any specific environment settings.
One gotcha with this solution is that you will always need to set the stage variable anytime deploy.rb is read. That is to say, even though we assume the staging enironment, the following command will fail.
cap deploy
Instead you need to use:
cap deploy -S stage=staging
Tuesday, June 19, 2007
The Butt Kicker
Embarrasing moments at work.
Day: -5
Mood: Very good
Chris sends link to daily woot. As a result of good mood and caffine I purchase woot as a graduation gift for Jimmy. To ensure that I get woot as soon as possible I have it shipped to my office in SF so as not to miss the fed-ex guy.
Day: -4
Mood: Remorseful
Reality that I have just shipped a silly item from woot to my workplace sets in.
Day: -3
Mood: Concerned
Woot has still not shipped, not sure if it is going to make it in time.
Day: -2
Mood: Ambivilant
No woot. It's not going to make it. Oh well.
Day: 0
Mood: Elated
Graduation party rocks, no woot, but horseshoes and smashing a truck with a sledgehammer focuses my mind elsewhere.
Day: +4
Mood: Concerned
Woot has arrived at work. Working from home so cannot retrieve it until next week.
Day +5
Woot sits on desk at work.
Day +9
Mood: Embarassed
Come in to work to find massive box on desk (woot). The clearly labeled package has been aligned like an art piece so that all may revel in it's existance. It seems to saw, 'Behold, this idiot bought a Butt Kicker'.
http://www.thebuttkicker.com/ButtKicker%20Gamer_home.html
Day +433
Mood: Victorious
After more then a month in Twitters office, two weeks in the back of my car, a month and a half in my living room, and the remainder in my garage, the butt kicker was finally delivered to my brother as a college graduation present. After an epic odyssey I am proud to report that the butt kicker is kicking butts. My brother reports that on a medium setting the device delivers enough force to throw him from his office chair. A project is in the works to secretly mount the device inside his couch, allowing the delivery of a brown noise class audible to unsuspecting guests. Kick on butt kicker, kick on.
Day: -5
Mood: Very good
Chris sends link to daily woot. As a result of good mood and caffine I purchase woot as a graduation gift for Jimmy. To ensure that I get woot as soon as possible I have it shipped to my office in SF so as not to miss the fed-ex guy.
Day: -4
Mood: Remorseful
Reality that I have just shipped a silly item from woot to my workplace sets in.
Day: -3
Mood: Concerned
Woot has still not shipped, not sure if it is going to make it in time.
Day: -2
Mood: Ambivilant
No woot. It's not going to make it. Oh well.
Day: 0
Mood: Elated
Graduation party rocks, no woot, but horseshoes and smashing a truck with a sledgehammer focuses my mind elsewhere.
Day: +4
Mood: Concerned
Woot has arrived at work. Working from home so cannot retrieve it until next week.
Day +5
Woot sits on desk at work.
Day +9
Mood: Embarassed
Come in to work to find massive box on desk (woot). The clearly labeled package has been aligned like an art piece so that all may revel in it's existance. It seems to saw, 'Behold, this idiot bought a Butt Kicker'.
http://www.thebuttkicker.com/ButtKicker%20Gamer_home.html
Day +433
Mood: Victorious
After more then a month in Twitters office, two weeks in the back of my car, a month and a half in my living room, and the remainder in my garage, the butt kicker was finally delivered to my brother as a college graduation present. After an epic odyssey I am proud to report that the butt kicker is kicking butts. My brother reports that on a medium setting the device delivers enough force to throw him from his office chair. A project is in the works to secretly mount the device inside his couch, allowing the delivery of a brown noise class audible to unsuspecting guests. Kick on butt kicker, kick on.
Thursday, June 14, 2007
Rails threads vs. the Mongrel pack
Skip to the bottom if you have an aversion to reading.
I was very dissapointed to find out that Rails is not thread safe. Well, that's not totally true but I'll come back to that. What this means is that we cannot put every request to our rails application in it's own thread. Every request needs to line up and wait for rails to proccess it.
Using your out of the box script/server command this can be problamatic.
Imagine you can only buy the newspaper one page at a time, and you have to buy it from your freiendly neigherhood homeless person. Now this may not be so bad assuming you are the only person trying to read the newspaper.
But now the newspaper is getting popular, and there are 5 other people want to read the paper as well. It's still ok until Mary the town gossip spends 30 seconds buying her paper. Now everyone unfortunate enough to try and get a page during this time will have to wait those 30 seconds.
Mongrel to the rescue, sorta. So now instead of one homeless person we have a pack of homeless people (pack of mongrels). If one is slowed down by the town gossip you have other homeless people waiting to sell you that same page of the paper.
But there is a catch, you and all your fellow newspaper readers are blind (ironic, I know) and thus you can't see what homeless person has the shortest line. Lucky for you the native american girl Apache22 is willing to direct you to what she thinks is the best line. Unfortunatly she is not that great at this (through no fault of her own) and she can only tell when a line is in really bad shape.
As a concerned newspaper distributer what do we do? Round up more mongrels, but they come at a price. Adding more mongrels means more memory you will have to allocate for them. Each one is an entire Rails process. Depending on the app you are looking at 50-200 MB each. Whew, that sounds like I have to do some math, and I hate to think, is there an easier way?
If Rails were thread safe we could spawn a thread (homeless newspaper seller) for every request that comes in. Threads use much less memory then processes so we're not so worried about memory consumption.
But Rails is thread safe! Yes, I think it is, however is your application? And all those nifty plugins you installed? Thread safety is something we have to think about when developing, and we don't want our developers to have to think any more then they already do.
Can't we just kill all the people who are too slow? That's sick, but what we can do is make sure that all of our requests are served in a similar amount of time. It should be pretty obvious that most page requests should be returned in a matter of seconds, but if we are using AJAX to run a fatty report or search, that may not have crossed our minds. After all, that's why we have spinner.gif.
Is there a way to run a long process in the background? Of coarse, http://backgroundrb.rubyforge.org to the rescue. You will have to setup an observer on the client side that keeps asking the server if the job is done yet, but those requests will be small and fast.
So that's all. Let's review:
Threads:
- Applications need to be thread safe
- Small memory footprint
Processes (Mongrels!):
- Slow requests can hold up faster ones
- Larger memory footprint (~50-200MB)
- Architect needs to determine how many mongrels to run
Conclustion:
After reviewing the facts, threads are not as cool as I thought they were.
This is a gotcha more then a defect with rails.
Gotcha:
Rails does not run with threads by default. If you have actions that take a long time to process, you may ruin your performance long before anything else (database, disk I/O) starts to effect you application.
I was very dissapointed to find out that Rails is not thread safe. Well, that's not totally true but I'll come back to that. What this means is that we cannot put every request to our rails application in it's own thread. Every request needs to line up and wait for rails to proccess it.
Using your out of the box script/server command this can be problamatic.
Imagine you can only buy the newspaper one page at a time, and you have to buy it from your freiendly neigherhood homeless person. Now this may not be so bad assuming you are the only person trying to read the newspaper.
But now the newspaper is getting popular, and there are 5 other people want to read the paper as well. It's still ok until Mary the town gossip spends 30 seconds buying her paper. Now everyone unfortunate enough to try and get a page during this time will have to wait those 30 seconds.
Mongrel to the rescue, sorta. So now instead of one homeless person we have a pack of homeless people (pack of mongrels). If one is slowed down by the town gossip you have other homeless people waiting to sell you that same page of the paper.
But there is a catch, you and all your fellow newspaper readers are blind (ironic, I know) and thus you can't see what homeless person has the shortest line. Lucky for you the native american girl Apache22 is willing to direct you to what she thinks is the best line. Unfortunatly she is not that great at this (through no fault of her own) and she can only tell when a line is in really bad shape.
As a concerned newspaper distributer what do we do? Round up more mongrels, but they come at a price. Adding more mongrels means more memory you will have to allocate for them. Each one is an entire Rails process. Depending on the app you are looking at 50-200 MB each. Whew, that sounds like I have to do some math, and I hate to think, is there an easier way?
If Rails were thread safe we could spawn a thread (homeless newspaper seller) for every request that comes in. Threads use much less memory then processes so we're not so worried about memory consumption.
But Rails is thread safe! Yes, I think it is, however is your application? And all those nifty plugins you installed? Thread safety is something we have to think about when developing, and we don't want our developers to have to think any more then they already do.
Can't we just kill all the people who are too slow? That's sick, but what we can do is make sure that all of our requests are served in a similar amount of time. It should be pretty obvious that most page requests should be returned in a matter of seconds, but if we are using AJAX to run a fatty report or search, that may not have crossed our minds. After all, that's why we have spinner.gif.
Is there a way to run a long process in the background? Of coarse, http://backgroundrb.rubyforge.org to the rescue. You will have to setup an observer on the client side that keeps asking the server if the job is done yet, but those requests will be small and fast.
So that's all. Let's review:
Threads:
- Applications need to be thread safe
- Small memory footprint
Processes (Mongrels!):
- Slow requests can hold up faster ones
- Larger memory footprint (~50-200MB)
- Architect needs to determine how many mongrels to run
Conclustion:
After reviewing the facts, threads are not as cool as I thought they were.
This is a gotcha more then a defect with rails.
Gotcha:
Rails does not run with threads by default. If you have actions that take a long time to process, you may ruin your performance long before anything else (database, disk I/O) starts to effect you application.
Time to start
It's offical, now that Chris is going to start blogging I must follow suit. Look forward to posts and responses to other people's outlandish statements regarding time traveling and black holes.
Wednesday, April 18, 2007
Project management tools
We're looking for a new project management system at Hotchalk so I've been evaluating what's available in the online arena.
Basecamp
The old and trusted tool but it's lack of a ticket management system is unacceptable for our needs. Since it has an API I hoped to find a ticket tracking app that would create tasks in basecamp and provide some nice integration. The only tool I found was http://bctix.com. While it boasts integration with basecamp BCTix is half baked and not ready for prime time.
Trac
A great tool for defect management and I was disappointed to find there are no plugins to basecamp. Trac has great plugin support but I suppose all those python guys are still bitter about the popularity of Ruby and are not in a hurry to contribute to 37 signal project.
Goplan
It's like basecamp with a few more features. It does have ticket management, but it's so minimal I don't really see the benefit. My major complaint is the inability to link tickets to a milestone or release. This is huge, without it there is almost no difference between tickets and the messages available in basecamp. On the plus side it's stable, and is a viable substitute for Basecamp.
Lighthouse
This is going to be the tool to watch. It is more issue tracking centered and has all the standard API's. It also allows you to create and update tickets via email, which is great for people on mobile devices or just to lazy to log in. It lets you associate tickets with milestones and shows a nice status bar as you close all the tickets related to a milestone.
Basecamp
The old and trusted tool but it's lack of a ticket management system is unacceptable for our needs. Since it has an API I hoped to find a ticket tracking app that would create tasks in basecamp and provide some nice integration. The only tool I found was http://bctix.com. While it boasts integration with basecamp BCTix is half baked and not ready for prime time.
Trac
A great tool for defect management and I was disappointed to find there are no plugins to basecamp. Trac has great plugin support but I suppose all those python guys are still bitter about the popularity of Ruby and are not in a hurry to contribute to 37 signal project.
Goplan
It's like basecamp with a few more features. It does have ticket management, but it's so minimal I don't really see the benefit. My major complaint is the inability to link tickets to a milestone or release. This is huge, without it there is almost no difference between tickets and the messages available in basecamp. On the plus side it's stable, and is a viable substitute for Basecamp.
Lighthouse
This is going to be the tool to watch. It is more issue tracking centered and has all the standard API's. It also allows you to create and update tickets via email, which is great for people on mobile devices or just to lazy to log in. It lets you associate tickets with milestones and shows a nice status bar as you close all the tickets related to a milestone.
Subscribe to:
Posts (Atom)