AVR Freaks

I do not understand why the main application code would be calling the bootloader.  On reset, the bootloader configures the AVR USART and checks to see if any data is arriving within a tiny time period.  If data arrives into the USART before the end of this time period, then the AVR's Store_Program_Memory framework gets activated, and a new program is written into the flash.  This implies that the main application program that is currently in the flash (at power up) is going to be overwritten.  So I can't think of any reason why the main ap would call the bootloader.

It appears that your code is loading  0x0000 into the indirect-reference register (register Z; r30:r31) and  is beginning to execute code there.  At that location is a JMP instruction to the code that is executed when the AVR is powered on and reset-stabled.  It is jumping to the main application's configuration/reset entry point instead of the bootloader's entry point.    It is likely that your AVR has a fuse setting that sends the code execution entry point to the bootloader's starting address when the device is powered-on or reset.  It is unlikely that the this fuse setting will be the factory default.

You need to get a handle on when your bootloader will jump to the app and when it stays in the bootloader. If you are calling the bootloader when conditions are not met to remain in the bootloader, then you will end up back in the app.

Here is an example which stays in the bootloader IF there are no flags set AND app appears valid, AND a bootloader pin not pressed. Which means to call the bootloader from the app- clear the reset flags and call the bootloader as before. If something goes wrong where the app is screwed up, pull the bootloader pin low and power on. In any case if the first byte in the app space is erased, then stay in the bootloader.

                static inline void
    entryCheck  ()
                {

                #define BLPIN   &PORTA,2 // bootloader pin, low = stay in bootloader after a reset

                // get byte address of app fron fuses
                u16 appAddr = FUSE.BOOTEND*256;

                // app first byte ok? (read from memory mapped flash)
                bool appOk = *(u8*)(MAPPED_PROGMEM_START|appAddr) != 0xFF;

                // check boot pin, if pressed (off/low) then user wants bootloader
                pinPullup( BLPIN ); //or similar pin code
                bool noBl = pinIsOn( BLPIN ); //pinIsOn=high=not pressed=true -> noBl

                // if any flag + BLPIN not pressed + app appears valid,
                // go to app (need word address to call)
                if( RSTCTRL.RSTFR && noBl && appOk ) ((void(*)())(appAddr/2))();

                // no flags, app not valid or BLPIN pressed, so return (run bootloader)

                }

                int
    main        ()
                {                    
                entryCheck();

                // if we are here, bootloader is wanted

                while(1){
                    //bootloader
                }

                }