Deploying during business hours

I recently had to defend a habit I created of doing our production deployments at the start of our day.

Most companies deploy during off hours, or over the weekend. Having myself suffered through many early morning deploys I decided to try a new strategy about a year ago, and it was very successful. Let me lay out my reasoning.

Developers hate to get up at 2am: And we should really care about what developers like and dislike. Not because it's a nice thing to do, but because developers can do massive amounts of damage (purposefully or not) when they are forced to do something they don't want to. Take Zeck for example. Zeck is tired and really wants to go to sleep, but the testing team is taking longer then estimated because the documentation of some new feature is non-existent. After 2 hours of waiting for something to happen a defect is opened against Zeck. Turns out Zeck is not handling exceptions and his code failed a down scenario test (yes some testers actually test this). Rather then think about how to implement this correctly, Zeck just returns what is required to pass this one test (the back end will never actually go down right?) and kicks the defect back to testing. Is this example convoluted? Only slightly. But regardless, what have we accomplished? Exposure of the deploy to the customers was limited. Zeck and the rest of the team don't come in the next morning. Zeck is a little less happy with his job and thinks about what life at Google is like this time of year.

Users are the best testers: Even if your testers are complete sadists when it comes to finding flaws with your developers code, nothing compares to the breaking power of actual usage. And they are going to find most of these defects the morning after a deploy, while your developers are sleeping in, or taking comp time for the previous night. Even if you take the comp hours ahead of time (or not at all), people are going to be worn out from the previous night. A caffeine loaded sleep deprived developer can be the most dangerous of all coding animals. Especially when you are working with live data, which often cannot be restored with out losing new data.

Creatures of habit: If your developers are human (and they are human) then they have a routine. Wake up, drink coffee, write code, go home, or something to that effect. When you introduce a unexpected change (while you may know you have to wake up tomorrow at 2am, your body does not), you throw off the routine, and impair your ability to get in the zone. Now this point is very abstract, but if you have done any reasonable amount of coding you know what I mean. Granted some people can always find the zone, they are a sort of enlightened species that walk among us. If you know someone like this please send me their contact info. It's uh, for my ... research.

It's tough to deploy every two weeks: So you read an article about agile development while getting the oil changed on your Prius. But you just can't imagine doing a deployment every two weeks! Once a month is painful enough. However if you remove the pain (see above) from deployments it's easy to deploy twice a month, even four times. The more often you deploy, the less risk you introduce of something breaking, which means less testing. It's like compound interest, sort of. But if you really want to make it work you have to reduce the fixed pain the night deploy brings.

Bottom line: Happy and rested developers fix code fast, and properly. They also tell funnier jokes. How can you argue with that?

MySQL not using index on simple condition

This was killing me, see if you can find why this query is slow.

The table has about 500,000 records.

SELECT * FROM standards WHERE (`standards`.`RefIdOld` = 1381320) LIMIT 1

describe of RefIdOld
+----------------+-------------+------
| Field | Type | Null | Key | Default
+----------------+-------------+------
| RefIdOld | varchar(32) | YES | MUL | NULL
+----------------+-------------+------

Index on RefIdOld:
Table: standards
Non_unique: 1
Key_name: RefIdsOld
Seq_in_index: 1
Column_name: RefIdOld
Collation: A
Cardinality: 22970
Sub_part: NULL
Packed: NULL
Null: YES
Index_type: BTREE

Hint? Run an explain and you'll see it's not using the index, but it is listed as a possible index, just to mess with you.
Possible Keys: RefIdsOld
Key: null

Turn's our MySQL will ignore an index when the value you are trying to match on is of a different type then the column.

Our RefIdsOld is a VARCHAR, and we are using a integer in our query.

Solution:
Throw single quotes on it and you are gtg.
SELECT * FROM standards WHERE (`standards`.`RefIdOld` = '1381320') LIMIT 1

Fedora 8 ERROR 2003 (HY000): Can't connect to MySQL server

On a fresh install of Fedora 8 with any version of MySQL installed via and installation method (yum, source, rpm) you will get a connection refused error when attempting to connect from a remote server.

Most of the results on google are related to the my.cnf files having skip-networking enabled or using bind-address with 127.0.0.1 or some other incorrect ip address.

The my.cnf on fedora 8 comes default with none of these issues and does not need to be edited.

The problem is a firewall that is installed by default and prevents all incoming connections on 3306.

You can edit or disable the firewall altogether with the following command.

system-config-firewall

I found it incredibly frustrating that a linux distro comes with a enabled firewall out of the box. My past experience has been with Slackware and FreeBSD which don't do anything for you, the later does not even come with bash. It should have at least been a configuration option. Just my two cents.