malloc in external flash?

Go To Last Post
14 posts / 0 new
Author
Message
#1
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I understand that when I malloc, I am allocating a memory block in RAM.
Is it possible to do this when I am using external flash?

I have made this and I realize that (1) I need more space and (2) I need to remember the stored data even after powerdown.

int main(void)
{
	list_t *logs;
	datestamp_t date;
	timestamp_t time;
	uint32_t count[ROW_MAX][COL_MAX] = {{},{}};
	uint32_t *p_count;
	
	date.year = 14;
	date.month = 2;
	date.day = 13;
	
	time.hour = 18;
	time.minute = 10;
	time.second = 27;
	
	logs = NULL;
	
	p_count = &count[0][0];
	
	add_log(&logs, date, time, p_count);
	count[0][0]++;
	add_log(&logs, date, time, p_count);
	count[0][0]++;
	add_log(&logs, date, time, p_count);
	//read_log(logs, 0);
	print_logs(logs);

	return 0;
}

where add_log is

void add_log(list_t **arg_log, datestamp_t arg_date, timestamp_t arg_time, uint32_t *arg_p_count)
{
	list_t *new, *end, *ret;
	uint32_t temp_count[ROW_MAX][COL_MAX] = {{},{}};
	uint8_t a, b, c;
	uint8_t array_size;
	
	new = malloc(sizeof(list_t));
	
	if (*arg_log == NULL) {
		ret = new;
	} else {
		ret = *arg_log;
		end = *arg_log;
		while (end->next != NULL) {
			end = end->next;
		}
		end->next = new;
	}
	
	a = 0;
	b = 0;
	c = 0;
	array_size = ROW_MAX * COL_MAX;
	while (array_size--) {
		new->count[a][b] = *(arg_p_count + c++);
		b++;
		if (b >= 5) {
			a++;
			b = 0;
		}
	}
	
	new->date.year = arg_date.year;
	new->date.month = arg_date.month;
	new->date.day = arg_date.day;
	new->time.hour = arg_time.hour;
	new->time.minute = arg_time.minute;
	new->time.second = arg_time.second;
	new->next = NULL;
	
	*arg_log = ret;
}
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Personally in EEPROM I would just use an array. Yes I know that this requires a maximum size to be defined up front..... but the EEPROM is anyway fixed in size.

see http://www.nongnu.org/avr-libc/user-manual/group__avr__eeprom.html if you haven't already had a chance to have a look.

regards
Greg

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

additional info:
I first thought of implementing this in pointers since there is a possibility of deleting logs.
e.g.
log1 = 100
log2 = 200
log3 = 300
log4 = 400
and if log 2 is deleted,
log1 = 100
log2 = 300
log3 = 400

And then when the memory becomes full (e.g. I reach the end of the memory address) I could re-use those locations that I have deleted

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Flash storage is usually organised into 'pages' of bytes (eg 128 bytes per page).
That level of granularity complicates things because you must (re)write the whole page., therefore you either keep a copy of the page in RAM (or you read the flash-page into a buffer), then update it and then save it.
Flash storage also has a limited number of write cycles.

How many events do you need to store ?., as gregd99 says, your requirements might fit into the internal EEPROM.
(EEPROM on the AVR8s has a 1-byte granularity but it is not in the same address-space as the RAM and requires a different access methodology).

If you need to store a lot of data then data-compression/minimisation or external storage (SDCard, EEPROM) will be required.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

> 128 bytes per page
this might be lucky! each log contains 126 bytes, I can insert a couple of dummy bytes and do a page write for each log.

I expect to have to log every second and to keep a week or two worth of logs, about 1.2M logs. So about 148kB, lets call it 256kB

If I use serial flash, I think I will have a problem of keeping track of each log number.

As in my post above, log numbers are sequential but it is possible to delete a log entry. And I would need to "shift up" the remaining logs: log3 becomes log2, log4 becomes log3 etc...

And then there is the issue of reusing those empty spaces.

Very tedious to do if I have a million log entries.

I imagine I could keep the address of the "head" of each log. Setup a pointer with value equal to the head and point its tail to the next log address?

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Can't you just add an SD/MMC card to the design and write everything to a file? Do any post processing such as removing entries and such on a fast PC. With a proper filing system you can easily open a file and fseek() to and existing record to modify it to just set some kind of "deleted" flag. Then have the PC tidy up the data afterwards.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

If your hardware allows it I too would go down the SDCard path.

I am puzzled as to why a datalogger needs to delete arbitrary records., recycling the entire storage I can understand, but deleting individual records ?

If you really need to delete records on the recorder I would record everything with an old-school ring-buffer and ;
- have a bit-flag with each record that indicates if that record has been 'deleted' and is to be ignored later,
or
- have a seperate index that identifies which records are to be ignored. (but now you need to save that potentially large index somewhere)

Shuffling records to reuse the deleted record(s) is going to take a lot of elapsed time, reduce your write-endurance and to my way of thinking needlessly complicate matters.
Leave the data-cleaning to the post-processing phase.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Update.
Apparently, the added cost of about 1USD for an external flash is "too much". And customer would just rather limit the number of logs as to what can be accomodated without external components i.e. using internal flash memory. I already told him that would be about 300 logs.
I estimated to consume about 48KB of program memory so I would have about 80KB left. BTW, I plan to use atxmega128a3u.
I have not done this yet but I am not sure if I really could use those extra 80KB of flash memory?
Can I write to them at runtime? I assume that I would have to use SPM? I am now just reading up on AVR1316. Am I on the right track?

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Quote:

I estimated to consume about 48KB of program memory so I would have about 80KB left. BTW, I plan to use atxmega128a3u.

Stop right there, before you go any further... (to quote Meatloaf).

So you are paying for 128K flash micro when you only plan 48K code that could easily fit in 64K with some "headroom". Might I ask what the difference in price is between the 64K model and the 128K model? Would it be $1 or more by any chance?...

Quote:

Apparently, the added cost of about 1USD for an external flash is "too much"

If you totally insist you can use the 80KB flash using SPM routines in the bootloader section of the micro but note these restrictions.

1) you cannot write individual bytes but must erase/rewrite a whole "page" at a time. I'm guessing without looking it up but I suspect the pages in a 128K Xmega are 128 or 256 bytes long.

2) any particular page can only be written (erased) 10,000 times over the entire life of the micro. After that there is no guarantee that it will retain the bit patterns written to it.

Both of those are fairly major considerations and the general reason why folks do not use AVR flash for data storage. If they do it's generally only for very rare data updates and low amounts of data. If you are writing a lot, often then AVR flash is NOT the place to put it.

Anyway get the prices for the 64K and 128K micros and see if you can get management to trade micro size for some proper storage.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

malloc() isn't intended for persistent storage like flash.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

xmega128a3u is about 2.7USD
xmega64a3u is about 2.5USD
Not much difference there.

But after some reading, I am now thinking that this may be more trouble than its worth. Its not straightforward work as I initially thought. I am now just looking for cheaper flash memories, hopefully to meet customer's target.
Any suggestions? Although I do not have a memory size in mind as I will be discussing this first with him.

BTW, I chose xmega because I will be using USB CDC to transfer the logs to PC.

So I guess I am now back to my original question, which (I think?) has already been answered: I >cannot< malloc in external flash right?

Also, due to cost considerations, I do not think an SD will be an option. Although I will look up prices first and float that idea too.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Quote:

I >cannot< malloc in external flash right?

How could malloc() (run time allocation of storage which involves writing "hosuekeeping" data in that storage) ever have worked in flash which it's close to impossible to write at run time? Even if you managed a "char * p = (char *) malloc(40);" with that 40 bytes in flash then what? You obviously cannot just do p[23]= 'X' because that, too, would involve a write to flash.

It sounds to me like you maybe don't know how malloc() actually works if you ever thought it was possible to do it in anything but easily writable memory.

The two often suggest storage options for non-volatile memory in AVR are either SD/MMC (because you can buy GB's for a couple of $) or something like Atmel's own AT45 Dataflash (actually they sold it to someone else - but same meat, different gravy!). SPI flash (as the AT45 was/is) is easily interfaced to an AVR (also some have I2C interface) and you can buy a few MB for a ibt under a $.

Back in the day when Atmel were selling AT45 then for volume quantities (perhaps 50,000-100,000 units or more) you could have approached Atmel directly and got a "deal" on buying both the MCU and the memory form them (we did) but now that the AT45 division has been sold I don't know if they still have an arrangement with the owner where they work with Atmel to still offer deals like this.

How much data are you talking about? Are there any constraints on the storage like speed to write it and so on?

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

clawson wrote:
1) you cannot write individual bytes but must erase/rewrite a whole "page" at a time. I'm guessing without looking it up but I suspect the pages in a 128K Xmega are 128 or 256 bytes long.
From a previous post, OP would not mind 128 bytes at all.
IIRC page can be written half at a time.
To write the first half:
Erase the page.
Fill the first half of the page buffer with data and the other half with 0xFFs.
Write the page.
To write the second half:
Fill the second half of the page buffer with data and the other half with 0xFFs.
Write the page.
IIRC writing a page only causes 1->0 transitions.
The bytes with corresponding 0xFFs in the page buffer will not change.
Quote:
2) any particular page can only be written (erased) 10,000 times over the entire life of the micro. After that there is no guarantee that it will retain the bit patterns written to it.

Iluvatar is the better part of Valar.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Yep. I do not know how malloc() works underneath. Which is actually the point of my OP. Thank you for clarifying that for me :D
*sigh hope we could start actual work with this already, too many specs changes :roll: