Warning: Atmega644, 644P and 324P and crystal settings
After spending lots of time tracking down erratic behavior, incl. finding bugs with the Jtagice mkII, I have found that the Atmega324p, Atmega644 and Atmega644p, when used with an external 20 Mhz crystal and lots of serial input interrupts, ALL REQUIRE THAT THE FUSE IS SET TO FULL SWING OSCILLATOR.
I've written a test program that stores incoming serial input in a large buffer using an ISR routine, then empties the buffer in a forever loop. The test program checks explicitly for conditions that can never happen (like identical pointer tests generating different results with all interrupts off). If the fuse is set to "Ext Crystal 8 Mhz-", the test program fails when sending it lots of serial input. When the fuse is set to "Full Swing Osc", the test program never fails. The test program shows clearly invalid values in the internal registers of the AVR.
The problem is particularly insidious because the AVR seems to run fine, until it receives a lot of interrupts (and maybe other conditions too)
I have run these tests at various baudrates (19200, 38400 and 115200) with identical results.
All tests were run with 5V regulated power. A scope shows NO voltage level variations when executing the tests.
The test program fails on the Atmega324p, Atmega644 and Atmega644p, all 40-pin DIP with the fuse set to "Ext Crystal 8Mhz-". However the test program passes on the Atmega328p with the same fuse settings.
The crystal I used is from Mouser, 815-ABL-20-B2, brand Abracom and the 2 caps are 22 pF each. I have used this combination on my Oscilloscope Clock kit for several years using an Atmega168 and Atmega328p. With over 1000 units sold, there never has been a problem.
I did try 2 other 20 Mhz crystals I had (unknown brand) and they showed identical behavior.
Voltage range (p/p) for the oscillator signal was about 800mV on the XTAL1 pin and 500mV on the XTAL2 pin when using the "Ext Crystal 8 Mhz-" fuse setting and 3.56V on XTAL1, 3.36V on XTAL2 (same for all tested AVRs). However I did notice significant distortions on the XTAL2 pins when the serial interrupts where occuring (I used BreyTerminal to send a batch of 50 characters or more, as well as sending text files).
I'll post the test program in another posting.
My conclusion: the Atmega324P, Atmega644 and Atmega644P (DIP versions) are more sensitive to the crystal used then previous AVRs, specifically the Atmega168 and Atmega328p, and require the "Full Swing Oscillator" fuse setting.