I am trying to use the DebugWire interface with an ATMega164, JTAGICEmkII, and STK500. I have the JTAGICEmkII 10 pin header connected to the 6 pin adapter. I have this 6 pin adapter plugged into the 6 pin ISP header of the STK500.
I believe I have everything connected correctly because I can read the correct device signature. I have the reset jumper removed from the STK500 and no lock bits set. I disabled the JTAG enable fuse, and enabled the DebugWire fuse. When I go to Debug -> Start Debugging, I get the following pop-ups.
Failed to identify target. Would you like to retry with external reset enabled? I clicked yes.
Failed to identify target. Would you like to use SPI to enable the JTAG interface fuse? To continue, use the squid cable to attach to the target's SPI pins. If I click yes to this box, it enables the JTAG enable fuse. If I click no, it does nothing.
Does anyone know how to use the debugwire without using JTAG? I want to leave the 6 pin ISP connector plugged in and use the debugwire interface that way. I had no problems doing this on an ATMega48 (although the ATMega48 does not support JTAG, which is what I think is messing things up).