Just posting this as a warning to others to check where the TOSC pins are on their device before laying out the PCB!
The errata for the xmega D3 devices shows that for older rev chips one of the non available functions and options is 'TOSC alternate pin locations, and TOSCSEL bit in FUSEBYTE2'. Those non available items completely disappear in the latest rev chips, leading me to believe that this was now an available option. It seems, however, that the TOSC and XTAL pins are not shared and are instead on dedicated pins, TOSC steals 2 other digital I/O pins PE6 and PE7.
Upshot is D3 (and A3) devices have dedicated pins for XTAL and TOSC, D4 devices have shared pins with the option to place TOSC on different pins via the fuse. Check the TOSC placement in the pinout section and don't, like me' assume a D3 is the same as D4 :-(