I figured out why my scheme for calibrating the RC oscillator on an Atmega 329 was sometimes failing. I had made the assumption that increasing OSCCAL would increase the oscillator frequence. It seems that is not always the case. I found when going from OSCCAL = 103 to OSCCAL = 104, the frequency dropped slightly.
Examining the datasheet closely I see Atmel observes the same thing.