Took me hours and hours to find this, so I figured I'd post it. The iox32a4u.h file has an incomplete definition for ADC_CH_t, so it appears 7 bytes long instead of 8, causing ADC_t's CH1, CH2 and CH3 to map to the wrong memory locations. Probably no one noticed because CH0 was in the correct spot.
to the end of the struct definition to fix the problem.
I haven't checked the other header files, but they also may be faulty. I didn't know where to report this (it's not an ASF bug, for once ;)), so hopefully someone with the power to fix it will read this thread and take care of it.