Storing and retrieving variable length data in RAM and NV memory

Go To Last Post
27 posts / 0 new
Author
Message
#1
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I'm planning a data logging system that needs to log various values, and store various configuration parameters. The logged data and config parameters can be all sorts of things, ranging from single bits to 64 bit doubles to strings and other arrays of data.

 

I can set some limits, like no data can be more than say 128 bytes. Config data will likely be added to later with new parameters. Configurations need to be stored in EEPROM (internal/external). Logged data will be time-stamped (32 bit int) and have a channel number associated with it. Logged data will be up to tens of megabytes per day. I was thinking of having an external SRAM cache, say 128k or even a few megabytes, and secondary flash memory for longer term storage. Minimum required is 1 day storage, say 64MB. Thinking 8 bit MCU, but could go 32 bit. Ideally running at 1.8V, battery powered, low-ish cost.

 

How can I store this data in a way that is energy efficient, allows fairly random access (e.g. get data between two times or that hasn't been transmitted yet) and if possible will allow for some compression during transmission? I'm fairly flexible in the design at this stage.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

My current thinking is to pile data into flash memory with a bit of metadata (earliest and latest time-stamps) and scan through it when I need to retrieve stuff. The clock might change so I can't guarantee that time stamps will always be ascending, for example, so this could be an expensive operation. Maybe have some kind of bitmap of untransmitted blocks. Firmware would have to ensure that the first byte in every block is the start of a variable length record, although blocks could contain multiple records.

 

I'd rather avoid filesystems, too much overhead and possibility of corruption. However, if anyone has any good ideas for robust ones I'll listen.

 

For the config side maybe some kind of MODBUS style thing where parameters all have an address, but modified to accept a length parameter too so that long strings don't just take up 100s of "registers". Throw in some min/max limits or maybe handler functions called by reference. I want to keep the code as clean and generic as possible, rather than say writing a command line parser and dozens of config commands.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

mojo-chan wrote:
and secondary flash memory for longer term storage.

Do you mean SD/MMC? Once you have that you can apply a file system and then the world is your (data processing) oyster. All the usual techniques for indexing data can now be applied from simple things like naming files with dates and times to full on solutions like SqlLite/MySQL databases etc

mojo-chan wrote:
I'd rather avoid filesystems, too much overhead and possibility of corruption
Well yes and no - but there is a reason why almost every desktop computer on planet earth has some kind of filing system on some kind of secondary storage (and your phone and your tablet etc etc). Do you trust your desktop to still have the .c files you write in a week/month/year/decade?

 

True may filing systems (and FAT is one of them) are not powerfail safe. You could switch to some kind of journalling filing system or use one of the other data processing solutions from antiquity - like RAID arrays and so on (yes, multiple storage devices) - depends how robust you really want it.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

clawson wrote:

Do you mean SD/MMC?

 

The product I'm replacing uses SD cards, but they have proven unreliable. As well as variable quality, they tend to corrupt when the battery starts to die, and we run into problems with the larger cards not working without upgrading the Microchip library code... The product will likely have a 15+ year lifecycle, and SD cards are a somewhat fast moving target. All solvable, but I'd rather avoid these issues.

 

I was thinking either a large serial flash, 512Mb perhaps, or some cheap NOR flash in the gigabit range. NOR makes you do your own bad block management though.

 

Quote:
Well yes and no - but there is a reason why almost every desktop computer on planet earth has some kind of filing system on some kind of secondary storage (and your phone and your tablet etc etc). Do you trust your desktop to still have the .c files you write in a week/month/year/decade?

 

Desktop PCs have robust filesystems though, with atomic writes and fairly solid power supplies. We want to run the new product at 1.8V mostly, at the very least 3.6V from a lithium cell, and when the cell is dying and the modem is on it's easy for the supply voltage to the SD card to fall too far and corrupt it.

 

Quote:
You could switch to some kind of journalling filing system

 

Have you tried using one on an MCU? Any recommendations?

 

Perhaps we could use cache SRAM and then just offload large chunks of data to SD card, that would solve a lot of the issues. SD cards are essentially infinite in size so even just assigning 64MB/channel blocks would be okay.

 

And any thoughts on data structures and access methods, especially for the configuration stuff?

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Well from about 1986 until about 2000 when XP with NTFS took off (NT3. 5 before that was a bit niche) the main desktop PC filing system for HDD was FAT though I guess the PSUs were more stable than micros. On the whole there wasn't that much in the way of catastrophic file system failure. 

 

In the past I've used EXT3 which is the journalling version of EXT2 on systems that had unexpected daily power interruptions. 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Data corruption from low voltage is more a hardware problem than a software problem.

You basically have two choices:

Ensure that you do not have low voltage or

find a way to live with low voltage.

 

For the former, find a way to keep the voltage up while changing a battery.

For the latter, make sure the processor never selects external memory while either's voltage is too low.

That still involves getting the hardware right.

Tying chip-select to active would probably be a bad idea.

Brown out detection is probaly required.

Iluvatar is the better part of Valar.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Concur with skeeve. Far better to avoid trying to fix a hardware problem with software. Once you do that, you have lost control over your system. Even more so, when the problem is predictable and likely to happen. 

 

In my case, my logger writes to a sequence of files, each representing a day of data. Thus, if something were to go wrong (eg, the battery dies), the user has lost only 1 day worth of data (out of possibly several hundred days). Yes, this is software fix to a hardware problem, but I cannot protect against the user who leaves the logger out longer than the expected battery life. No amount of backup will solve this as the user can leave it operating long enough to also use up the backup. I COULD have it close the file system (and terminate logging) when it switches over to backup.n So, these are cases where it appears to me that the software "fix" IS justified and is the only possible solution (because SOMEONE will use up the backup). 

 

As for variable format data storage, I would impose the requirement that the data structure remain constant though the data logging process. Then, store some sort of index to that structure in the metadata at the head of the file(s). For config data, use some sort of CSV format using ASCII text instead of numeric values. The commas then nicely delimit your fields and makes recovery MUCH simpler. Yes, it doubles the write times and the media size, but for me, it is worth it. Further, in my case, the user HAS to be able to read and process data on his/her own, and proprietary formats would be a killer. CSV text allows the user to read the data as a text file and anyone can do that using MatLab or what-ever.

 

Jim

Jim Wagner Oregon Research Electronics, Consulting Div. Tangent, OR, USA http://www.orelectronics.net

Last Edited: Mon. Aug 22, 2016 - 05:02 PM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Okay, let's look at hardware fixes. SD cards need 3.3V and about 50mA while reading. We have seen cards corrupt themselves while only being read, by the way, not just when being written. They were supposedly genuine industrial grade cards too.

 

We need to read while the modem is active. Modem can pull 2A for short periods. Already got masses of capacitance in there to help, but we still see 400+mA from the battery which is nominally 3.6V. It's hard to detect end of life state because it's hard to put a consistent load on it, and it's lithium so no nice curve. Hard to see how this can be resolved in hardware, and it doesn't help us if we want to run at below 3.3V either like say from 3x alkaline cells that will hit 0.9V before EOL.

 

Okay, the actual range is 2.7-3.3V, so there is a little more headroom, but it still isn't enough.

 

That's why we prefer flash memory. No controller to corrupt things, we have direct access to the memory at 1.8V so masses of headroom. No filesystem means we can just checksum blocks and discard them if they look bad, rather than trying to handle a broken FAT or whatever.

 

The old device uses a binary format encoded with a modified base64 so that it can flow over email and SMS originally, later FTP. CSV is okay for disk but needs compression for transmission. It's also super slow to skip through because you have to read every character to find the newline and next record.

 

EXT2/3 looks quite heavy for an MCU. In the 90s PCs used to run chkdsk quite a lot. Not sure but I don't think many MCU FAT implementations support chkdsk, even if it did seem like a good idea.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

What filing system were you using when it corrupted during read alone? Remember that Linux (until you tell it not to) will rewrite the latest "access time" on a file even if you only read the file. There are some core system files being read all the time and their access time updates can be very costly in terms of wear and tear and makes it much more likely the file system will be in mid-write at the time of a power outage. 

 

In Linux you mount the file system with the "noatime" option to prevent this. 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Using FAT. The problem seems to be that some SD card controllers write metadata, bad block lists and stuff like that I guess, even when reading only. I think some might just corrupt their metadata when there is an extended brown-out.

Sd cards are cheap. Even the "industrial" ones don't get a 100% test. Five year lifespan is apparently a lot to ask.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

The problem is that most forms of mass storage is going to have a sector size. For the bigger serial flash devices this is 4k. So to append data to the end of a file that hadn't filled the sector you neef to read into ram, append data, erase the sector and write. Interrupt this from the erase to write and you've lost data. So you need a method where you copy the data to another free sector and changing the allocation table to reflect the change. Sounds like a journalling file system.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

You can avoid error amplification by the file system by not using a file system.

A data logger is just append at the end.  Not terribly complicated.

 

Recognizing good data among bad:

Require that every chunk begin on a 16-byte boundary.

Require that every chunk begin with magic number 0X701523b38e2f347ea9ebb08f88405c55.

Require fixed length metadata appear after the magic number.

The data should be sufficient to infer the size of the chunk.

Require that every chunk end with the md5sum of the rest of the chunk.

The definition of chunk is left as an exercise for the reader.

 

Battery EOL:

Treat batteries like some businesses treat light bulbs:

Change them after a fixed amount of use.

My guess is that there is a way to measure the number of ampere-hours

generated by a battery and feed the information to the AVR and other devices.

When nearing EOL, the AVR can switch to another NV storage device,

possibly powering off the first one altogether.

Iluvatar is the better part of Valar.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

SD cards are a CPU + NAND Flash. Obviously, the CPU is running some software, of unknown quality. I think it will be pot luck how it behaves under sudden power loss. I don't think that is even in the SD spec?

 

FAT and SD cards were never designed for embedded use, where sudden power loss is normal. They are more designed for PC use, where an orderly shutdown is the normal case. Of course, PCs lose power suddenly, and users have been getting corrupted files systems for decades. Reformat and reinstall is the user recovery method, which is not available for embedded systems.

 

I think if I really needed a robust FAT for my commercial embedded projects, I would definitely consider an off the shelf solution, otherwise it's too much yak shaving. Suppliers like http://www.phaedsys.com/principa... (others are available).

 

Going direct to NAND gives you more control, but introduces another level of yak shaving. Normally I would favour the "keep it simple" approach, but making FAT and/or NAND reliable doesn't seem to have a simple solution.

Bob.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I found this thread quite interesting as I am busy using Arduino sd libraries myself. My files may not be that time critical, but my file gets opened and closed constantly with every write. I simply pull the Li-Po out of the breadboard before I read the SD in my PC. So far no corruption. (The Li-Po of 3.7V and the Uno chip at 16Mhz on breadboard was still stable down to 2.7V brownout.) I am using very old 256M SD'cards dating back to the first smart phones with external SD's. (Nokia's I believe.)

 

As I am not a boffin on this, the SD library open a file, and append the latest line to it. Perhaps that is why I had no corruption. The app I am using should run about an hour, and read four analog channels at from 1 sec intervals to about every minute. I simply use a csv format, so I can write basically anything I want. I have no idea how many bits a block can be, but as it's written line by line, my lines are quite long. In my case, I guess it can be a text string as long as there is free RAM.

 

There is course for thought for me in these posts as I may just have SD corrupt coming up.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

donotdespisethesnake wrote:

SD cards are a CPU + NAND Flash. Obviously, the CPU is running some software, of unknown quality. I think it will be pot luck how it behaves under sudden power loss.

 

This. We have found that there is a great variation, and unless you are willing to pay really silly money you can't even get consistent 100% tested cards with the same flash and controller firmware.

 

Bergie5737 wrote:

I found this thread quite interesting as I am busy using Arduino sd libraries myself. My files may not be that time critical, but my file gets opened and closed constantly with every write. I simply pull the Li-Po out of the breadboard before I read the SD in my PC. So far no corruption.

 

Note that at least from Windows 8 onwards it just silently fixes a lot of FAT issues without even bothering to tell you, such is the prevalence of corruption. Back in the XP days it was rare to plug a flash drive or SD card in without seeing the scandisk message.

 

Kartman wrote:
The problem is that most forms of mass storage is going to have a sector size. For the bigger serial flash devices this is 4k.

 

Indeed, the gigabit range NOR flash chips are usually 8 or 16k, hence we were considering the SRAM cache.

 

The major advantage of the smaller serial flash memories is that they guarantee all the blocks are good for the lifetime of the IC. A small bad block list is the most you will require, where as with NOR they come from the factory with bad blocks and you could potentially need to map a lot of them. In other words you become the SD card controller, with all the hassles that involves. Maybe we could use multiple serial flash memories and simply not fit them on some models.

 

Has anyone got experience with large (say 2Gb+) NOR flash and 8 bit MCUs, particularly with bad block mapping? How long does the bad block list need to be, so you search it linearly etc?

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

mojo-chan wrote:
Has anyone got experience with large (say 2Gb+) NOR flash and 8 bit MCUs, particularly with bad block mapping? How long does the bad block list need to be, so you search it linearly etc?

Surely you mean NAND? You shouldn't get bad blocks in NOR.

 

Decades agoi we have previously done ARM based projects that boot from (reliable) NOR but use NAND for their main data storage. The latter is subject to bad blocks. So we ECC every sector. When damage is found the sector is added to a bad block list and the contents are mapped to a new sector from a replacement pool. This does mean that there is not direct access to sector N but, instead, a request for N is made. It's first looked up in the bad block list and if found there its contents are delivered from the remap sector. If it's not in th ebad block list then a direct access to N can be made. Things become fun when remap sectors need remapping. Also you have to worry about the integrity of the bad block list itself!

 

All this stuff is going on inside SD/MMC cards - it's why people use them so THEY don't have to worry about wear levelling and bad block remapping and all that stuff!

 

PS forgot to say that when NAND comes from the factory the manufacturer has already labelled bad blocks so before you do anything else (ANY write!) you need to scan the device for the markers and create your initial bad block list from that (and find a place to store it that doesn't itself involve bad blocks).

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Yeah, sorry, NAND.

 

Your ARM system sounds like what we are considering. That's some useful information, as the datasheets for these parts suggest you need to use ECC to get the rated reliability, on top of bad block mapping.

 

Can I ask, how big was your bad block list? I'm just trying to get an idea of the overhead vs. reliability.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

As I say this was literally decades ago so my memory is pretty hazy. I'm guessing in that time frame it was probably only a 1MB device or perhaps a little larger and, honestly, I cannot remember the details of things like how many sectors were reserved for a remap pool.

 

We wrote the FFS (Flash filing system) ourselves from scratch because back then there wasn't really any alternative but over the years a lot of companies have got very rich selling their own take on Nand flash filing systems with wear levelling and bad block avoidance and the reason they got rich is that it's a fairly honed art - lots of "gotchas" and conditions you may not think of initially - so you are paying for their experience.

 

As such I'd probably google for Nand flash filing solutions with wear levelling and bad block avoidance and see what's available. At this stage I imagine some may have made their way in to the open domain but, as always, you'd have to be wary about the "quality" of any such solutions. If you pay someone money you probably have some kind of guarantee on quality and maintenance.

 

At the end of the day you will just be buying what is sitting inside an SD/MMC card though ;-)

 

PS I tried that Google. Interesting hit near the top...

 

http://www.keil.com/pack/doc/mw/...

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Hmm, good point Cliff, I shouldn't be so cheap :-)

 

Maybe 512Mb serial flash is the way to go. More reliable, easier to use, easy to stack a few up if more than 64MB is required.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I'm a bit confused.  In particular, about the SD card.

 

First, on a side note, how much storage do you need?  (for example does FRAM or similar fit your criteria?)

 

Anyway, if you run your AVR or ARM or SD card or whatever at a lower voltage than the specified minimum -- yes, weird things are going to start to happen.  Won't that happen file system or no file system?  

 

Is there a storage technology for very low supply V?  Dunno.

 

I have SD cards in a series of apps and haven't seen corruption.  I've outlined my approach in other threads.  But then again, the app uses brownout on the 5V AVR8.

 

If you care to, you can to some extent have your cake and eat it too with respect to file system.  Use FAT or whatever.  Prepare an empty LOG.DAT as big as you care to make it.  Now your PC can still manipulate LOG.DAT.  Your app needs to know it's "logical" EOF but any changes to the file system itself should be minimal during run time.

 

 

You can put lipstick on a pig, but it is still a pig.

I've never met a pig I didn't like, as long as you have some salt and pepper.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Hi theusch.

 

Storage wise I'd say 64MB+ would be idea. That does limit our choice a bit because serial flash tends to max out at about 512Mb, with only a few models at that or the 1Gb size. I'm actually thinking about using multiple 64/128Mb chips, as some of our customers won't need more than that.

 

We are aiming for 1.8V. Basically the idea is to support 3.6V lithium and alkaline (1V/cell minimum, 2x or 3x in a pack). The problem is that when you have cellular modems demanding 2A it tends to make your supply voltage fall, and we have seen problems with it in the past corrupting SD cards and the like. Actually even without the modem, the SD card drawing 50mA can cause problems with lithium cells near the end of life, and then you get into adding load resistors to try to test the battery under load before writes etc... Maybe there is a better way. Obviously we try to avoid accessing the SD card when the supply voltage is low, especially writes, but it's not a perfect fix and it seems to depend on what random controller they fitted to the card this week. So just run everything at 1.8V, loads of head-room on lithium/3x alkaline and even enough to do a non-modem version from 2x alkaline.

 

You can get other types of flash memory that are designed for 1.8V operation. It's as low as most MCUs will run at, including the XMEGA range, PIC24, and various 32 bit/ARM models. It's popular with FPGAs and the like too, so there is a fair bit of support from things like high resolution ADCs. And modems themselves use 1.8V I/O the Telit models we prefer.

 

I think the SD corruption isn't just due to low voltage, it's due to fluctuating voltage. They use an internal step-up to generate the programming voltage, and if the supply bounces around they won't be able to fit enough capacitance to it to avoid problems. It seems to make the controllers crash occasionally too.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

It seems to me the real problem is separating the modem power from the rest of the board.

The ideal would probably be to give the modem a separate battery pack

and maybe connect it to the rest of the board through opto-isolators.

Failing that, perhaps you could play games with inductors and diodes to

ensure that the current fed to the modem circuit does not change too fast.

Iluvatar is the better part of Valar.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Yeah, separate battery pack. We already have a lot of capacitance and inductance. The problem is, all this stuff costs money and space.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

mojo-chan wrote:
The problem is, all this stuff costs money and space.

How much cost of money and space is consumed by ensuring that the writes are done only when not transmitting and when the supplyV is adequate and stable?

You can put lipstick on a pig, but it is still a pig.

I've never met a pig I didn't like, as long as you have some salt and pepper.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

We need to read during modem comms. Some controllers corrupt during reads.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

mojo-chan wrote:
We need to read during modem comms. Some controllers corrupt during reads.
Why?

If you really must, can you arrange for the SD card to be disabled during modem startup?

 

From what's been written so far, the modem will be the biggest drain on the batteries.

Would another battery for the SD card be all that expensive?

What about a switching power supply running off the battery?

How much is it going to cost to solve this problem with software?

Iluvatar is the better part of Valar.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

We already use a step-up switching power supply to provide power to the modem when the supply is below the minimum (IIRC 3.2V). An extra better is an option, but because of different usage patterns it's hard to size it vs the main system battery to get good life. Also, sometimes we run from mains or some other fixed power supply indefinitely without battery backup.

 

I suppose if we were going that route a supercap might be a better option. On another product we had these weird Tadiran cells that were a kind of battery/capacitor hybrid, I forget the details now.