samd21J xplained board: openocd fails to read memory 0xfffffffe, bootloader fails

Go To Last Post
5 posts / 0 new
Author
Message
#1
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Hi,

 

I'm having some weird problems with the samd21J (and G). I have flashed the Arduino bootloader and can upload test programs via EDBG or Bossa.

Up till yesterday I was developing the firmware and flashing it via Bossa and the native USB port no problems. Then it stopped working. I rolled back my changes (everything is in git)

but the same problem persists.

 

The firmware I'm developing is for a keyboard, a fork of the Kaleidoscope open source keyboard firmware. The firmware is here: https://github.com/Dygmalab/Rais..., and depends

on our fork of the Arduino SAM core: https://github.com/Dygmalab/Ardu.... Complication instructions in the Travis CI file: https://github.com/Dygmalab/Rais...

 

Yesterday I was able to use the EDBG debugger to see that the program was hanging at reset here:

https://github.com/Dygmalab/ArduinoCore-samd/blob/cd1eb1a428325f80dec1ab33f660e1191e71707a/cores/arduino/cortex_handlers.c#L150

 

     148       if ((&__data_start__ != &__data_end__) && (pSrc != pDest)) {                                                                                                                                                                                   
     149         for (; pDest < &__data_end__; pDest++, pSrc++)                                                                                                                                                                                               
     150           *pDest = *pSrc;                                                                                                                                                                                                                            
     151       }                                                                                                                                                                                                                                              

 

But I failed to find exactly where the loop failed as the values for pDest and pSrc were optimized out.

       

Today as I continue the investigation, I can no longer get to this point with openocd failing as it tries to read memory at location 0xfffffffe

Which from the datasheet appears to be the last read at the end of the system global memory. I'm not sure if this was happening yesterday.

 

    Debug: 1397 17753 gdb_server.c:2670 gdb_input_inner(): received packet: 'qfThreadInfo'                                                                                                                                                                    
    Debug: 1398 17753 gdb_server.c:2670 gdb_input_inner(): received packet: 'qXfer:memory-map:read::0,fff'                                                                                                                                                    
    Debug: 1399 17753 gdb_server.c:2670 gdb_input_inner(): received packet: 'mfffffffe,2'                                                                                                                                                                     
    Debug: 1400 17753 gdb_server.c:1381 gdb_read_memory_packet(): addr: 0xfffffffe, len: 0x00000002                                                                                                                                                           
    Debug: 1401 17753 target.c:2031 target_read_buffer(): reading buffer of 2 byte at 0xfffffffe                                                                                                                                                              
    Debug: 1402 17753 cmsis_dap_usb.c:522 cmsis_dap_swd_run_queue(): Executing 5 queued transactions                                                                                                                                                          
    Debug: 1403 17753 cmsis_dap_usb.c:545 cmsis_dap_swd_run_queue(): DP write reg 8 0                                                                                                                                                                         
    Debug: 1404 17753 cmsis_dap_usb.c:545 cmsis_dap_swd_run_queue(): AP write reg 4 fffffffe                                                                                                                                                                  
    Debug: 1405 17753 cmsis_dap_usb.c:545 cmsis_dap_swd_run_queue(): AP write reg 0 a2000011                                                                                                                                                                  
    Debug: 1406 17753 cmsis_dap_usb.c:545 cmsis_dap_swd_run_queue(): AP read reg c 0                                                                                                                                                                          
    Debug: 1407 17753 cmsis_dap_usb.c:545 cmsis_dap_swd_run_queue(): DP read reg c 0                                                                                                                                                                          
    Debug: 1408 17755 cmsis_dap_usb.c:583 cmsis_dap_swd_run_queue(): SWD ack not OK: 4 FAULT                                                                                                                                                                  
    Debug: 1409 17755 cmsis_dap_usb.c:731 cmsis_dap_swd_switch_seq(): JTAG-to-SWD                                                                                                                                                                             
    Debug: 1410 17756 cmsis_dap_usb.c:522 cmsis_dap_swd_run_queue(): Executing 2 queued transactions                                                                                                                                                          
    Debug: 1411 17756 cmsis_dap_usb.c:545 cmsis_dap_swd_run_queue(): DP read reg 0 0                                                                                                                                                                          
    Debug: 1412 17756 cmsis_dap_usb.c:545 cmsis_dap_swd_run_queue(): DP write reg 0 1e                                                                                                                                                                        
    Debug: 1413 17757 cmsis_dap_usb.c:600 cmsis_dap_swd_run_queue(): Read result: bc11477                                                                                                                                                                     
    Info : 1414 17757 adi_v5_swd.c:125 swd_connect(): SWD IDCODE 0x0bc11477                                                                                                                                                                                   
    Debug: 1415 17757 cmsis_dap_usb.c:522 cmsis_dap_swd_run_queue(): Executing 2 queued transactions                                                                                                                                                          
    Debug: 1416 17757 cmsis_dap_usb.c:545 cmsis_dap_swd_run_queue(): AP read reg 4 0                                                                                                                                                                          
    Debug: 1417 17757 cmsis_dap_usb.c:545 cmsis_dap_swd_run_queue(): DP read reg c 0                                                                                                                                                                          
    Debug: 1418 17758 cmsis_dap_usb.c:600 cmsis_dap_swd_run_queue(): Read result: fffffffe                                                                                                                                                                    
    Debug: 1419 17758 cmsis_dap_usb.c:600 cmsis_dap_swd_run_queue(): Read result: 0                                                                                                                                                                           
    Error: 1420 17758 arm_adi_v5.c:512 mem_ap_read(): Failed to read memory at 0xfffffffe                                                                                                                                                                     
    Debug: 1421 17758 gdb_server.c:2670 gdb_input_inner(): received packet: 'qSymbol::'                                                                                                                                                                       
    Debug: 1422 17858 cmsis_dap_usb.c:522 cmsis_dap_swd_run_queue(): Executing 5 queued transactions                                                                                                                                                          
                                     

I have 2 boards and after flashing the problem firmware onto a fresh board it now exhibits the same behaviour - can't run or debug programs.

I can still flash programs with openocd (no errors), but they don't run. When I try to debug, I get the above error from openocd and this from gdb:

 

(gdb) target extended-remote :3333
Remote debugging using :3333
0xfffffffe in ?? ()
(gdb) monitor reset init
target state: halted
target halted due to debug-request, current mode: Thread
xPSR: 0xe1000000 pc: 0xfffffffe msp: 0xfffffffc
(gdb) c
Continuing.
at91samd21j18.cpu -- clearing lockup after double fault

Program received signal SIGINT, Interrupt.
0xfffffffe in ?? ()

 

I can flash a new bootloader via EDBG, and then flashing a program via the bootloader with BOSSA works! (not with the problem firmware but a test that toggles pins).

 

If I flash the problem firmware with a fresh bootloader, I get the same issues detailed above.

 

After flashing the problem firmware, the bootloader seems to get wiped out, as it never creates a serial TTY again. Failing with this error in Linux dmesg:

 

May 29 17:18:05 matt-pc kernel: [344622.913622] cdc_acm 1-1.1:1.1: urb 0 failed submission with -19

 

-19 is ENODEV.

 

If anyone has any ideas of what to look for I'd be very grateful. In particular:

 

  1. Can anyone think why openocd has problems reading at fffffffe? Is this a dead end or am I right to consider this a clue.
  2. Could this be related to the reset handler failing while initialising the data section?
  3. How is it possible for the problem firmware to destroy the bootloader?
  4. Why would uploading via bossa work, but not edbg?

 

Thanks,

Matt

Last Edited: Wed. May 30, 2018 - 03:25 PM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

mattvenn wrote:
But I failed to find exactly where the loop failed as the values for pDest and pSrc were optimized out.
When that happens "Goto disasembly" and use the mixed C + Asm view. It should be fairly obvious which ARM registers are used for the src/dest pointers.

 

Not sure how a for() loop can hang when the continuation condition was pDest < &__data_end__ as I guess pDest must surely exceed that eventually?

 

As this appears to be the ".data" setup loop it would perhaps hint of some linker script error if this could ever run out of RAM or something. Do you have a particularly large amount of data in .data ?

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Thanks for the tip with gdb.

 

I was wondering if it was hanging for the same reason that openocd is failing to read at 0xfffffffe, but that address is way above the maximum flash size.

It's annoying I can't run gdb anymore to check that loop out.

 

RE data size, here's the result of the sizer:

 

matt-pc:2494 [master]: arm-none-eabi-size  output/Raise-Firmware.ino.elf
   text    data     bss     dec     hex filename
  36656     528    5468   42652    a69c output/Raise-Firmware.ino.elf

 

And I'm using the chip with 256kb flash.

 

I've also just checked the bootloader is still intact after flashing the problem firmware. my version of openocd doesn't support the flash verify_bank subcommand, but I was able to use

 

edbg -b -c 48000 -t atmel_cm0p --verify --file /home/matt/Arduino/hardware/dygma/samd/bootloaders/zero/samd21_sam_ba.bin

 

to validate bootloader before flashing the bad firmware. then after flashing the bad firmware and the chip is hung the bootloader is still valid.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

OK, a few answers today.

Sometimes I see the fuse settings getting wiped, like in this thread: https://www.avrfreaks.net/forum/...

I don't know why, but setting them back to defaults at least allows me to get the debugger to work again.

 

During debugging, the process still hangs in the reset handler, trying to initialise the text and bss sections. However if I set a conditional breakpoint on the last cycle of the loop I can then step over it and it starts the next loop.

Again, no idea why this is necessary.

 

Finally, after getting to where control is passed to main() I see that the compilation process has linked a test program I wrote a couple of days ago in the same directory! Which is why the real program never started and USB isn't initialised.

I guess the URB error I was seeing was the bootloader setting up the USB  serial, and then this immediately failing as the test program doesn't do what's necessary to keep it working.

 

I still don't understand how I can get into the situation where the bootloader no longer works, but at least my debugging tools work again.

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

bootloader does run briefly at reset, but hands over straightaway to the uploaded firmware. If this firmware doesn't look after the USB connection then Linux shows the URB error I've seen.

But the bootloader provides a 'double tap' option which holds it in bootloader mode. So that it can be unbricked if the new firmware is not working properly.