First posting. I hope this is the correct subsection!
Quick summary. I am helping a friend port working code from an Arduino based Teensy(3.5) to real hardware which is ATmega128 based. The code (so far) runs fine on the Teensy but crashes and reboots on the real hardware.
I have bought an ATmega128 dev board. It has two 10 pin connectors: one for SPI programming; and the other is labelled JTAG. All pins are brought out to headers. It was really good value for around £10 off Ebay. I am programming directly using the usbTiny programmer, and I am not using a boot loader.
I have lots of experience with microprocessors and microcontrollers of various types, however much less when it comes to the AVR family. I am comfortable with assembly code, low level monitor debuggers, logic analysers, etc.
We are using the MegaCore Arduino plug-in (from GitHub) for the Arduino GUI, which supports a number of non standard AVR cores such as the ATmega128. It is gcc based.
We have narrowed down the crash to the first call to lcd.print("message") in code run as part of setup. Could be a memory leak or stack corruption, I have no idea. The serial port sign on message keeps being printed, suggesting a crash and reboot.
The Teensy breadboard setup uses a I2C based colour OLED. The OLED uses the Adafruit library which itself uses hardware I2C. It appears to work OK.
I modified this library for this project to use bit banged I2C, since on the real hardware the hardware I2C pins are connected elsewhere and cannot be freed. There are spare pins brought to a header, so these have been used. This version of the code on the ATmega128 does not run, there is no sign on message and the serial based GUI never appears. I was concerned about memory usage, since the ATmega128 only has 4k bytes of RAM, so I tried to optimised away all fixed strings which were needlessly consuming RAM. The available free RAM went up, but we couldn't get the code to run.
Let's use the 20 x 4 I2C LCD module, I thought. So I modified yet another Adafruit library to make this bit banged too. The example sketch prints to all 4 lines and runs correctly (both my dev board and my friends real hardware). So we integrated this code and replaced the OLED code. This is where we at at now. No sign of anything being printed on the LCD, just the crash and reboot. If the LCD code is #ifdef'ed out, the code runs and we can operate the serial based (VT100) simple text GUI. The "lcd" (bit banged LCD library) setup is called from setup and so occurs early in the code flow execution. The crash is the same on both our hardware.
So I could have easily done something wrong when I modified the Adafruit library, so here was try #3.
On my dev board (only) connect up the I2C LCD to the hardware I2C pins and use the Adafruit LCD library. I was expecting this to solve all the problems, but no, it still crashes and reboots.
So now I am very puzzled, and unsure how to debug this.
One method I have used for other processors is to run a command line based simulator which prints out lots of trace information, such as code executed, writes and reads. The log file can be post processed using Linux tools to extract all reads and writes to look for anything unusual. I did download some code called "avr_sim" which runs under Linux. I gave it the Arduino hex file, it uses CPU time, but generates no output. I am doing something wrong, and really need to find a support group to progress this angle of attack.
I note that the dev board has a JTAG connector, however the only JTAG units I have are for Altera and Xilinx FPGAs. It would be great to hear that some clever hackers have managed to get these working with the AVR, but I suspect it is unlikely.
Since this looks like some kind of memory corruption, I think I need a low level debug solution.
I would appreciate any suggestions, however I would like to minimise cost ($$$), so I don't really want to buy an AVR JTAG pod.