I'm writing a universal (attiny & atmega) bootloader for AVRs with up to 8K of flash that fits in 32 instructions (64 bytes) + a 2-byte virtual reset vector. The current version of the code fits in 31 instructions:
With the 1 leftover instruction I could modify RxByte to work with any pin instead of just pin0. That's if there's nothing else that I've left out that a minimal bootloader must do. I'd like to hear what more experienced AVR programmers have to say.
Another option for the leftover instruction is to do two writes to UART_Tx (0 then 1) instead of toggling with the PINx register. That would allow for support of the Atmega8 which does not toggling a pin with writes to PINx registers.
With the bootloader code mostly done, I've started working on adding picoboot support to avrdude. I plan to use the break signal from the bootloader for flow control, so there will be no need for programming timeouts or knowledge of RWW sections.
The initialization sequence from avrdude will be:
1. send break to trigger reset
2. send xor of 2nd and 3rd signature bytes (from avrdude.conf)
3. send 3 zeros (clear Z command)
4. wait for break
If the target device is not running the bootloader or the signature doesn't match, then avrdude will timeout waiting for the break.
Avrdude will replace address 0 (reset vector) of the downloaded code with an rjmp to the start of the bootloader, and put an rjmp to the application start as the last word in the page before the bootloader.