Deploying during business hours

I recently had to defend a habit I created of doing our production deployments at the start of our day.

Most companies deploy during off hours, or over the weekend. Having myself suffered through many early morning deploys I decided to try a new strategy about a year ago, and it was very successful. Let me lay out my reasoning.

Developers hate to get up at 2am: And we should really care about what developers like and dislike. Not because it's a nice thing to do, but because developers can do massive amounts of damage (purposefully or not) when they are forced to do something they don't want to. Take Zeck for example. Zeck is tired and really wants to go to sleep, but the testing team is taking longer then estimated because the documentation of some new feature is non-existent. After 2 hours of waiting for something to happen a defect is opened against Zeck. Turns out Zeck is not handling exceptions and his code failed a down scenario test (yes some testers actually test this). Rather then think about how to implement this correctly, Zeck just returns what is required to pass this one test (the back end will never actually go down right?) and kicks the defect back to testing. Is this example convoluted? Only slightly. But regardless, what have we accomplished? Exposure of the deploy to the customers was limited. Zeck and the rest of the team don't come in the next morning. Zeck is a little less happy with his job and thinks about what life at Google is like this time of year.

Users are the best testers: Even if your testers are complete sadists when it comes to finding flaws with your developers code, nothing compares to the breaking power of actual usage. And they are going to find most of these defects the morning after a deploy, while your developers are sleeping in, or taking comp time for the previous night. Even if you take the comp hours ahead of time (or not at all), people are going to be worn out from the previous night. A caffeine loaded sleep deprived developer can be the most dangerous of all coding animals. Especially when you are working with live data, which often cannot be restored with out losing new data.

Creatures of habit: If your developers are human (and they are human) then they have a routine. Wake up, drink coffee, write code, go home, or something to that effect. When you introduce a unexpected change (while you may know you have to wake up tomorrow at 2am, your body does not), you throw off the routine, and impair your ability to get in the zone. Now this point is very abstract, but if you have done any reasonable amount of coding you know what I mean. Granted some people can always find the zone, they are a sort of enlightened species that walk among us. If you know someone like this please send me their contact info. It's uh, for my ... research.

It's tough to deploy every two weeks: So you read an article about agile development while getting the oil changed on your Prius. But you just can't imagine doing a deployment every two weeks! Once a month is painful enough. However if you remove the pain (see above) from deployments it's easy to deploy twice a month, even four times. The more often you deploy, the less risk you introduce of something breaking, which means less testing. It's like compound interest, sort of. But if you really want to make it work you have to reduce the fixed pain the night deploy brings.

Bottom line: Happy and rested developers fix code fast, and properly. They also tell funnier jokes. How can you argue with that?

MySQL not using index on simple condition

This was killing me, see if you can find why this query is slow.

The table has about 500,000 records.

SELECT * FROM standards WHERE (`standards`.`RefIdOld` = 1381320) LIMIT 1

describe of RefIdOld
+----------------+-------------+------
| Field | Type | Null | Key | Default
+----------------+-------------+------
| RefIdOld | varchar(32) | YES | MUL | NULL
+----------------+-------------+------

Index on RefIdOld:
Table: standards
Non_unique: 1
Key_name: RefIdsOld
Seq_in_index: 1
Column_name: RefIdOld
Collation: A
Cardinality: 22970
Sub_part: NULL
Packed: NULL
Null: YES
Index_type: BTREE

Hint? Run an explain and you'll see it's not using the index, but it is listed as a possible index, just to mess with you.
Possible Keys: RefIdsOld
Key: null

Turn's our MySQL will ignore an index when the value you are trying to match on is of a different type then the column.

Our RefIdsOld is a VARCHAR, and we are using a integer in our query.

Solution:
Throw single quotes on it and you are gtg.
SELECT * FROM standards WHERE (`standards`.`RefIdOld` = '1381320') LIMIT 1

Fedora 8 ERROR 2003 (HY000): Can't connect to MySQL server

On a fresh install of Fedora 8 with any version of MySQL installed via and installation method (yum, source, rpm) you will get a connection refused error when attempting to connect from a remote server.

Most of the results on google are related to the my.cnf files having skip-networking enabled or using bind-address with 127.0.0.1 or some other incorrect ip address.

The my.cnf on fedora 8 comes default with none of these issues and does not need to be edited.

The problem is a firewall that is installed by default and prevents all incoming connections on 3306.

You can edit or disable the firewall altogether with the following command.

system-config-firewall

I found it incredibly frustrating that a linux distro comes with a enabled firewall out of the box. My past experience has been with Slackware and FreeBSD which don't do anything for you, the later does not even come with bash. It should have at least been a configuration option. Just my two cents.

Rails Scaleability

Read a slashdot thread on Rails not scaling and now I need to weigh in from the safety of my blog. Let's do bullet points to help break down some of their arguments:

- PHP and Ruby are compariable
Sorry, you just can't do it. Ruby and Perl are comparable, Perl is probably better, who cares, let's move on.

- CakePHP is a great framework.
For me to poop on. Back before my Rails days I really wanted to like CakePHP. Seriously. I was a PHP ninja and the idea of MVC + data abstraction layer was a dream come true. But alas it was total crap. Weak community, annoying syntax (PHP's fault) and bugs. Next song.

- Rails is not thread safe
This is actually true, but rarely cited. Want to impress your friends, whenever they talk about Rails being slow because of Ruby, say "Who the fuck cares about Ruby? The Rails stack is not even thread safe you dumb bastard". What does that mean? It means a whole host of problems when it comes time to use your lovely self tested app. I"m not even talking about massive 2 billions requests per second sites, this will be bite you in the ass if you have 2 requests per second (given that one is significantly slower then the other).

- Ruby is slower then Python
So what, you think the bottle neck of your awesome facebook app is the CPU? How about I/O? You ever think of that? Full table scans, massive correlated queries, missing indicies, yeah, that's your problem chief. Unless you are doing some massive protein folding you better leave Ruby alone. Unless you are going to argue that Rails is not thread safe because of limitations in Ruby. Then you would be on to something.

- Google uses Java
Google is godlike in their programming ways. I seek only to collect what morsels of genius they donate to the community and not understand the terrible and mysterious ways they are contorting the web to their every whim. I am not google, you are not google, no one has ever been google and may never be again.

The silver bullet for zombie mongrels

Ok ok, so they are technically not zombie processes, but how could I resist such a great title.

I'm talking about mongrels that just stop responding to new requests. They are still up, ps does not tell you anything is wrong, monit thinks they are chugging along just fine, you can even telnet to them. But try to get a http response and it times out. Restarting the server fixes the problem every time.

This error state seems to be triggered randomly, however we saw it more with a overburdened mysql server running on the same box as the mongrels.

The trick, install the C based mysql driver. It's that simple, don't question it, just do it. Then curse yourself for not doing it sooner. You may even want to unleash a volley of nerf missiles on your coworker for claiming that we were using the C based driver all along.

Run this from the root of your rails app:
./script/console
Then this:
>> require_library_or_gem 'mysql'
If you don't see this:
=> ["MysqlField", "MysqlError", "Mysql", "MysqlRes"]
Unleash the fury.

How do you install the C based mysql driver?

1. Pray to your deity of choice that all required packages are installed.
2. Offer to trade the life of an office plant for a package management tool. (rpm is not package management, it's the root cause of anger management, see the difference?).
3. sudo gem install mysql
4. Replace that ficus you killed before office mates start asking questions.
5. Restart you rails apps

You do not need to change the rails environment.rb. If the gem is present, rails will find it.

If your god has forsaken you in steps 1-5 above just go grab the mysql RPM's your going to need.

Depending on distro, flavor and fat content, you will need some form of:
Headers and libraries
Shared libraries

Which can both be found on the mysql site.

Happy hunting.

When is / not division?

When you are using the wrong Ruby object.

Ruby seems innocent enough, but it's really doing a lot of 'object' work without you knowing.

Take the following rails example.

points = 0
points_possible = 100
points += quiz.points # points is 40, stored as an integer in the db,
percentage = ((points / points_possible) * 100) # evaluates to 0, WTF!

If we just give Ruby a little more direction by changing points to a float.

percentage = ((points.to_f / points_possible) * 100) # evaluates to 40, rock on

Deploying Rails applications to multiple environments

I love capistrano, however I had some trouble finding solid doc on how to deploy your app to multiple environments (UAT, staging, etc.) So here is a quick and dirty of the solution I can up with.

cap deploy

The old standby, this will configure our deployment based on deploy.rb. We want to conditionally configure our deployment so we need to pass a parameter to this comment, this is done with -S

cap deploy -S stage=uat

This sets a local variable 'stage' to 'uat' in our deploy.rb

Here is our conditional code from deploy.rb

if stage == 'production'
set :application, "foo"
set :repository, "http://svn.agiledog.com/svn/agiledog/foo/trunk"

# Without this $RAILS_ENV on the destination server will be used to determine the server environment
set :rails_env, :staging

role :web, "app.agiledog.com"
role :app, "app.agiledog.com"
role :db, "app.agiledog.com", :primary => true
elsif stage == 'uat'
# Different application names are used to allow different stages to be deployed on the same server if required
set :application, "uat-foo"
set :repository, "http://svn.agiledog.com/svn/agiledog/foo/branches/RB-4.0"
set :rails_env, :uat

role :web, "uat.agiledog.com"
role :app, "uat.agiledog.com"
role :db, "uat.agiledog.com", :primary => true
else # assume app == 'staging'
set :application, "staging-foo"
set :repository, "http://svn.agiledog.com/svn/agiledog/foo/trunk"
set :rails_env, :staging

role :web, "stage.agiledog.com"
role :app, "stage.agiledog.com"
role :db, "stage.agiledog.com", :primary => true
end

We have added two new environments, staging and uat. Both of these will need to be added to the database.yml file. We will also need to create staging.rb and uat.rb to the config/environments folder to declare any specific environment settings.

One gotcha with this solution is that you will always need to set the stage variable anytime deploy.rb is read. That is to say, even though we assume the staging enironment, the following command will fail.

cap deploy

Instead you need to use:

cap deploy -S stage=staging