I'm planning a data logging system that needs to log various values, and store various configuration parameters. The logged data and config parameters can be all sorts of things, ranging from single bits to 64 bit doubles to strings and other arrays of data.
I can set some limits, like no data can be more than say 128 bytes. Config data will likely be added to later with new parameters. Configurations need to be stored in EEPROM (internal/external). Logged data will be time-stamped (32 bit int) and have a channel number associated with it. Logged data will be up to tens of megabytes per day. I was thinking of having an external SRAM cache, say 128k or even a few megabytes, and secondary flash memory for longer term storage. Minimum required is 1 day storage, say 64MB. Thinking 8 bit MCU, but could go 32 bit. Ideally running at 1.8V, battery powered, low-ish cost.
How can I store this data in a way that is energy efficient, allows fairly random access (e.g. get data between two times or that hasn't been transmitted yet) and if possible will allow for some compression during transmission? I'm fairly flexible in the design at this stage.