AVR Freaks

Does the system know if the SD card is installed, (are you watching for any feedback from the card)?

The point being, can you run one unit on the bench with the SD card, as per usual, and one unit on the bench without the SD card inserted?

I think you have mentioned before that your code pretty much fills the memory available.

Are you re-using the same variables in different parts of the program?

One of the hardest bugs I ever remember tracking down was on a GPS and "map" display project.

It would run for hours, or days, or occassionallly a week or two, and then crash.

After years of debugging with an LED, and often an LCD, and an O'scope, I finally broke down and purchased a logic analyzer to help find the problem.

It turns out if one (of several) ISRs fired during one particular line of code, (of many), the ISR trashed a variable and subsequently crashed the program.

Obviously you need to make a list of possible problems, (such as stack overflow, etc.), and then devise your testing strategy to help rule in or rule out each potential problem one at a time.

Projects beget more projects.

There are chips that can do the high side current sense resistor monitoring for you, and you can read that signal into a Nano and then beep a piezo when the current increases 8X.

You will at least then have an easy way of knowning when the project has entered its faulty state.

Also, make sure to read the SD card data and make sure it didn't miss a data entry or two when the faulty state was entered, (if possible).

This is to help make sure that the SD card wasn't stuck doing some internal bad sector remapping, etc. as a trigger for the problem.

Any tropical lightning storms triggering the failure?

Deoes it fail on a very clean power supply on the bench?

I assume you have good reason to believe that the devices are failing while stuck up in a tree, acting alone, and that the failures are not triggered by the user messing with the device to try to download the data, (static electricity, not powering the device off before messing with it, etc.).

Gotta ask, how many versions of the software are out in the wild?

Is there any correlation between the failures and the SW version?

Does the chip family allow you to assemble one testbed on a similar uC with more memory?

And if so then assign more stack spaces, (whatever your compiler uses), as a poor man's test for a stack overflow?

i.e. It overflows on the production model, but not on the 'almost' identical model with more memory and stack spaces.

Does every unit eventually fail, when whatever the trigger is hits it, or are ther some units that fail, and will fail again, and others that seem to never fail?

It is very frustrating trying to find an intermittent bug that you can't reproduce on command.

But it is very rewarding when you definitively find the bug, (and then you kick yourself for making the error...).

That said, do you have a Plan B?

Is it time to design and build Version N+1, with a larger memory micro, and using one of the DB micros with a 2 level priority interrupt controller, or an Xmega with a 3 level priority interrupt controller, and with spare memtory to build in some of the debugging code so that its addition or removal doesn't alter the operation of the device?

Recall that the DB series runs at 24 MHz, while the Xmega will give you 32 MHz, so you have more processing power to work with, both for the native application as well as for the addition of the debug / self monitoring data integrity code.

I feel your pain.

JC

My small input.

I would use a logic analyzer, today it would be a USB one, to all the needed io's, (probably with a 10k resistor to avoid interference, so they have common ground but not power(but same voltage)).

this way you can place it close to correct conditions.

About the SD card, are some extra garbage collection when it's about full, or same file gets segmented ?

(I remember a C64 there was a way to build strings so it stopped for up to 1/2 hour doing garbage collection, the solution was to run the command that ask for free memory once in a while to force a cleanup.)

I just fear that sleep and debugger is a bad combo.

But perhaps write at lot of extra data on the SD card like :

1. stackpointer
2. reset counter (bod etc.)
3. sleepcounter
4. time in SD write routine
5. batt voltage 
6. temperature ! thunder 
7. time since last sleep

I have just had so many error I can't repeat in lab.

Once we had to put a camera up (There was a lot of status led. and relays and then a real meter, it was a automatic concrete mixer and some times it was to wet). But it never happened when I was there.

It turned out that it only happened when I added zero water and this way we could prove it :)   (they had denied that there was a problem with rain water in the sand!!! )