Benchmark Before You Optimize, Dammit!


Donald Knuth said it best:

We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.

Every programmer knows this in their head, but getting that through to their heart can be surprisingly difficult. When it comes to applying lessons like the one above, it’s easy to find yourself in the middle of a conflict between doing what you know you should do and doing what you want to do.

Here’s the thing. Most programmers like to optimize code. It makes us feel smart when we can take some code and, through sheer force of will and brainpower, make it magically run faster. In that sense, optimization is fun.

On the other hand, measuring code performance and benchmarking is not that fun. Done with any sort of rigor, it tends to be fairly labor intensive. Many programmers, especially the less-experienced ones, may start to wonder if the time spent measuring code wouldn’t be better spent writing code instead. After all, wouldn’t we rather all of our code be faster anyway? Why bother measuring code when we know that we can make it faster?

Well, no. Optimization costs time (money). Optimized code is generally harder to read and more bug-prone. Therefore it’s harder to maintain and more expensive. You don’t optimize all of your code because you can’t afford it. Jonathan Blow makes a strong case for this in his talk about independent game programming. That is why you need to make sure the code you’re going to optimize is actually causing a problem. Slow code doesn’t actually cause performance problems unless it is used in a way that makes it performance-sensitive.

You always want to measure your performance before you jump to conclusions about what to optimize. Chances are that the code you think needs optimization is not the primary performance sink in your game. Recently in one of my projects, I started running into performance problems with large numbers of actors on the screen. My mind immediately jumped to the rendering code I had written a few months earlier. There were some parts in there related to visibility handling that I was doing in a simple, but inefficient way. I had implemented it that way because it was faster to get working, but I always kept it in mind as a place where I was basically throwing away performance. I was tempted to go ahead and jump in and refactor that rendering code. After all, I KNEW it was inefficient.

Well, it turns out that it was a good thing I benchmarked my code first (using the benchmarking code from Box2D XNA, incidentally). Once I started measuring my performance, I found that my problems stemmed from my usage of Box2D bodies for collision. By default, my collision-enabled actors were creating their own Box2D bodies, even the money enemies dropped for the player to pick-up. The dropped cash didn’t need to have a sophisticated physics model so I just changed their collision logic to use a simple distance check instead of building Box2D bodies. Voila! Performance fixed. The inefficient rendering code I mentioned was still inefficient, but it accounted for less than 10% of the frametime. It got to live on to be optimized another day.

So, what’s the lesson here? In optimization, it’s best to follow the old proverb: “Measure twice and cut once.

Share this Article:
  • Digg
  • StumbleUpon
  • del.icio.us
  • Facebook
  • Yahoo! Buzz
  • Twitter
  • Google Bookmarks
  • Print