View topic - umm... malloc fail :(

Search

umm... malloc fail :(

AVR Freaks Forum Index » AVR (8-bit) Technical Forums » AVR forum

Author

Message

luvocean1

Posted: Feb 03, 2012 - 10:35 AM

Joined: Dec 26, 2006
Posts: 1407
Location: Sydney, Australia

Ok I am aware of the memory fragmentation after using malloc multiple times and freeing multiple times.

Whats the way around it? I am allocating and deallocating 2K ish memory size for each of my tcp sockets tx and rx buffers. So all up 4k memory for each sockets. After sockets are closed I free the memory. But it seems after about doing a few of these my malloc fails and returns with NULL Sad

Is this normal?

clawson

Posted: Feb 03, 2012 - 10:40 AM

Joined: Jul 18, 2005
Posts: 62230
Location: (using avr-gcc in) Finchingfield, Essex, England

Why not pre allocate an array of packets and just use an array entry when you need one?

If the packets are variable size make the array entries as large as the largest one you will get and just waste the final few bytes for smaller packets.

You will know at compile time how many such array elements the system can cater for and if you need more you know you need to trade up to a larger device or add RAM (if technically possible).

As you've found fragmentation is, otherwise, quite a problem on micros with only a few K of memory unless all malloc()s and free()s are the same size (in which case it'll just keep reallocating the same blocks). In fact this would be another solution - make all the malloc's as big as the biggest one.

Another idea would be to "clear the heap" from time to time. When you get a lull in activity simply free() everything that's been allocated and the heap will return to its original "clean" condition.

_________________

luvocean1

Posted: Feb 03, 2012 - 10:43 AM

Joined: Dec 26, 2006
Posts: 1407
Location: Sydney, Australia

Ah seems I was not freeing the actual tx rx buffers in the sockets! I was freeing other data structures except for the tx rx buffers! grrr... anyways so I can now seem to be able to create sockets indefinitely...

I wonder if I could still come across the fragmentation issue... ?

clawson

Posted: Feb 03, 2012 - 10:49 AM

Joined: Jul 18, 2005
Posts: 62230
Location: (using avr-gcc in) Finchingfield, Essex, England

Are the allocations of different size or not?

BTW when we develop we often do complete system simulation in a PC app with I/O "faked" and you can then implement a malloc()/free() that not only makes the allocation but draws a picture of the heap as dots on a window rectangle (maybe one pixel = 16 bytes or something?). As bytes are allocated paint the equivalent pixels red. When freed return to background colour. You then get a visual representation of how the heap is being used and you can see when it starts to fragment and so on.

_________________

david.prentice

Posted: Feb 03, 2012 - 11:33 AM

Joined: Feb 12, 2005
Posts: 16271
Location: Wormshill, England

Quote:

I wonder if I could still come across the fragmentation issue... ?

You can often create a worst case on paper. e.g. a combination of some big and many small blocks.

You can devise a worst case that could be allocated in the worst way.

If you can run out of memory, then one day this worst case will actually happen in real life.

Cliff's suggestion of a static array is often a very good solution.

David.

luvocean1

Posted: Feb 03, 2012 - 11:49 AM

Joined: Dec 26, 2006
Posts: 1407
Location: Sydney, Australia

Thanks guys.... but the point is as long as i free ALL allocations then fragmentation is not a problem right?

Lets say I create sockets of size 2K then some with 4K and some with 1K buffers. As long as all sockets get processed and thus removed and freed all buffers then I have no issues with fragmentation right?

JohanEkdahl

Posted: Feb 03, 2012 - 12:00 PM

Joined: Mar 27, 2002
Posts: 18531
Location: Lund, Sweden

Quote:

but the point is as long as i free ALL allocations then fragmentation is not a problem right?

Right. You can even push that a bit further: As long as you free in the reverse order of the allocation order you will have no fragmentation.

sternst

Posted: Feb 03, 2012 - 12:04 PM

Joined: Jul 23, 2001
Posts: 2437
Location: Osnabrueck, Germany

JohanEkdahl wrote:

Quote:

but the point is as long as i free ALL allocations then fragmentation is not a problem right?

Right. You can even push that a bit further: As long as you free in the reverse order of the allocation order you will have no fragmentation.

Why should the order matter? Consecutive free blocks are merged, so at the end there is only one big free block no matter of the freeing order.

_________________
Stefan Ernst

bobgardner

Posted: Feb 03, 2012 - 12:06 PM

Joined: Sep 04, 2002
Posts: 21253
Location: Orlando Florida

What model of avr you using (just curious?)

_________________
Imagecraft compiler user

clawson

Posted: Feb 03, 2012 - 12:50 PM

Joined: Jul 18, 2005
Posts: 62230
Location: (using avr-gcc in) Finchingfield, Essex, England

Quote:

What model of avr you using (just curious?)

Peering into my crystal ball I actually see a UC3

(luvocean1 likes to ask his "generic" questions here rather than the AVR32 fora as apparently he gets a better response Wink

The problems of malloc() are likely to be the same everywhere though it obviously gets worse the less RAM you have))

_________________

JohanEkdahl

Posted: Feb 03, 2012 - 03:19 PM

Joined: Mar 27, 2002
Posts: 18531
Location: Lund, Sweden

Quote:

Why should the order matter? Consecutive free blocks are merged, so at the end there is only one big free block no matter of the freeing order.

OK, let's straighten this out: At any point in the execution of the program, if dynamic memory is freed in the reverse order from how it was malloc'ed there will never be any fragmentation problems.

Better now?

clawson

Posted: Feb 03, 2012 - 03:30 PM

Joined: Jul 18, 2005
Posts: 62230
Location: (using avr-gcc in) Finchingfield, Essex, England

That's true but the implication is that it's the only way to clear fragmentation. The point (I think) was that if you free() everything then it doesn't matter what order you do it in as everything will be merged back into one amorphous great lump.

_________________

luvocean1

Posted: Feb 03, 2012 - 10:42 PM

Joined: Dec 26, 2006
Posts: 1407
Location: Sydney, Australia

clawson wrote:

Quote:

What model of avr you using (just curious?)

Peering into my crystal ball I actually see a UC3

(luvocean1 likes to ask his "generic" questions here rather than the AVR32 fora as apparently he gets a better response Wink

The problems of malloc() are likely to be the same everywhere though it obviously gets worse the less RAM you have))

Clawson's witchcraft prediction is correct Razz

So as long as ALL the sockets eventually gets closed I shouldnt have any memory issue.

Another question...what about for data structures I define not as malloced pointers but rather global variables? Where do they get stored? not heap?

mnehpets

Posted: Feb 03, 2012 - 11:28 PM

Joined: Nov 09, 2011
Posts: 396

Quote:

Another question...what about for data structures I define not as malloced pointers but rather global variables? Where do they get stored? not heap?

The static data sections (.data and .bss). Variables get allocated there by the linker. There's discussion (for the 8 bit AVRs, but the same happens in almost all microcontrollers running C code) here:

http://www.nongnu.org/avr-libc/user-manual/malloc.html

- S

luvocean1

Posted: Feb 04, 2012 - 01:37 AM

Joined: Dec 26, 2006
Posts: 1407
Location: Sydney, Australia

hhhm..thanks for that.
So I should try to define data structures as global variables where possible...

clawson

Posted: Feb 04, 2012 - 12:37 PM

Joined: Jul 18, 2005
Posts: 62230
Location: (using avr-gcc in) Finchingfield, Essex, England

Quote:

So I should try to define data structures as global variables where possible...

Not necessarily but in a limited RAM environment it does mean you'll have a clearer picture of what your RAM use will be when you build/link as the .map file will show the allocation of globals in .data and ,bss (and the avr(32)-size tool will report totals in those sections).

_________________

luvocean1

Posted: Feb 04, 2012 - 11:43 PM

Joined: Dec 26, 2006
Posts: 1407
Location: Sydney, Australia

OK.

Whats a good way to store strings you can print using Usart debug to hyperterminal? I have lots of them...

does having them as global static const keep them only in code and not in RAM? Quite often I just do UsartSendString("hello worl") in my functions or subfunctions....where do these strings get stored?

clawson

Posted: Feb 05, 2012 - 01:55 PM

Joined: Jul 18, 2005
Posts: 62230
Location: (using avr-gcc in) Finchingfield, Essex, England

Quote:

Whats a good way to store strings you can print using Usart debug to hyperterminal? I have lots of them...

This is where you might be better posting in the UC3 forum. I haven't a clue what architecture it is? I'm assuming it's Harvard like the AVR8? I further assume it's got an equivalent of __attribute__((progmem)) ? Obviously you don't want to waste precious RAM for strings that are copied from flash to .data

If you do use:

Code:

UsartSendString("hello world")

then by default that "hellow world" will be held in flash then as C starts up it will be copied to RAM then a pointer to its location there will be passed into the above call to UsartSendString(). In AVR8 the avr/pgmspace.h support offers

Code:

UsartSendString(PSTR("hello world"))

And the UsartSendString() would be written to dereference tha passed pointer as a code, not RAM address. Again I haven't a clue how this is handled in UC3 land.

_________________

digitalDan

Posted: Feb 05, 2012 - 07:23 PM

Joined: Nov 01, 2008
Posts: 191

Under the hood the UC3 is a modified Harvard Architecture. The flash and RAM live in the same address space although at different and non overlapping address ranges obviously. From a programmer's perspective you can't really distinguish it from von Neumann except that the flash is read only. You can put executable code in the RAM and the UC3 will happily execute it from there.
Declaring an initialized variable as const will automatically put it in flash (with the Atmel supplied tools).

Code:

const char *hw = "hello world";

const int intArray[] = {1,2,3,4,5};

I'm not sure how string constants are handled in the case you cite above. I'll have to spend a little time with an assembly listing. Wink

edit: after looking through some disassembled code, something like this:

Code:

usart_write_line (&AVR32_USART1, "hello world");

causes the string constant to be stored in flash and the address of that string in flash is passed as a parameter to the function.

_________________
Letting the smoke out since 1978

luvocean1

Posted: Feb 08, 2012 - 01:17 AM

Joined: Dec 26, 2006
Posts: 1407
Location: Sydney, Australia

Thanks Clawson and digitaldan...so if I pass the string inside the UsartSendString(), the string is placed in the flash section. So it shouldnt be affecting my RAM size.

Coming back to the original topic here... I did end up finding a section in the code where I was dereferencing a NULL unallocated memory area...this may have cause follow on issues I guess and then eventually corrupting the heap and causing another malloc to freeze the micro Smile

clawson

Posted: Feb 08, 2012 - 09:24 AM

Joined: Jul 18, 2005
Posts: 62230
Location: (using avr-gcc in) Finchingfield, Essex, England

Quote:

Thanks Clawson and digitaldan...so if I pass the string inside the UsartSendString(), the string is placed in the flash section. So it shouldnt be affecting my RAM size

That's what Dan's edit suggests.

If you want to know what IS using your RAM then study the map file (though it will only show global/static allocations and not automatics or the heap).

_________________

luvocean1

Posted: Feb 14, 2012 - 04:45 AM

Joined: Dec 26, 2006
Posts: 1407
Location: Sydney, Australia

Another question... is it possible to find out how much free memory is available in the heap? Any built in functions in C available for checking free memory?

clawson

Posted: Feb 14, 2012 - 10:05 AM

Joined: Jul 18, 2005
Posts: 62230
Location: (using avr-gcc in) Finchingfield, Essex, England

Well one way is to use a for() loop starting with a stupidly high value and keep trying to malloc(). That is:

Code:

char * p;

int i;

for (i=32767; i>0; i--) {

  p = (char *)malloc(i);

  if (p) break;

}

if (p) {

  free(p);

}

At the end of this i says how big the largest malloc() possible is.

_________________

luvocean1

Posted: Feb 14, 2012 - 11:19 AM

Joined: Dec 26, 2006
Posts: 1407
Location: Sydney, Australia

lol that is really scary!
Surely there has to be some low level keeping track of the heap? When calling malloc it must check whether the size of free available memory is bigger than the space asked for in malloc....

So free memory checking is inside malloc function... we just need to know what variable it checks may be?

clawson

Posted: Feb 14, 2012 - 11:32 AM

Joined: Jul 18, 2005
Posts: 62230
Location: (using avr-gcc in) Finchingfield, Essex, England

Well for AVR-LibC the source of malloc() is open:

http://svn.savannah.nongnu.org/viewvc/t ... iew=markup

BTW while the C standard calls for malloc() and free() in stdlib it does not specify a freemem() type call or a specific form of implementation.

From a user perspective the repeated calls to malloc() I showed above really is the way to do it. My method will tell you the largest single block you can allocate. This may not be what you want. Perhaps you plan to make 100's of small malloc()s and there could still be some small blocks available after the large one is allocated so an alternative strategy is to just keep making "small" mallocs of N bytes until you get 0. Then the total is M * N where M is the number you made. Because you don't know how many you are likely to make and you need to free them all after then you probably want to build a linked list of allocations and then walk the list freeing them. This shows one way to do it:

http://www.java-samples.com/showtutoria ... rialid=587

_________________

JohanEkdahl

Posted: Feb 14, 2012 - 12:10 PM

Joined: Mar 27, 2002
Posts: 18531
Location: Lund, Sweden

Quote:

Surely there has to be some low level keeping track of the heap? When calling malloc it must check whether the size of free available memory is bigger than the space asked for in malloc....

The implementations I've seen keep a linked list of free blocks, and I've seen no function that reports statistics on that. There is no total free size maintained.

The source of avr-gcc malloc, free and the heap data structure is public and browsable. Have you had a look?

clawson

Posted: Feb 14, 2012 - 12:17 PM

Joined: Jul 18, 2005
Posts: 62230
Location: (using avr-gcc in) Finchingfield, Essex, England

Quote:

The source of avr-gcc malloc, free and the heap data structure is public and browsable. Have you had a look?

I'm kind of hoping he might have followed the link I gave but note this is the AVR8 implementation, not AVR32 though it's possible it might use similar code. If it does then note that MALLOC_TEST build switch in the source file which can build it into a complete, standalone test program. The stuff they have their that walks the list looks kind of useful and may be adaptable. (but again this is AVR8 not for UC3).

_________________

luvocean1

Posted: Feb 15, 2012 - 07:22 AM

Joined: Dec 26, 2006
Posts: 1407
Location: Sydney, Australia

hhmmm.. interesting code! Malloc if not very efficient, the code for allocation seems expensive!

clawson... i was reluctant to use that for loop malloc method cos I was hoping to have a periodic task function that monitored the memory usage (free mem) on the cpu every now and then and made sure there arnt any memory leak etc....seems I cant do that then if I have to use those for looped malloc way...

clawson

Posted: Feb 15, 2012 - 11:13 AM

Joined: Jul 18, 2005
Posts: 62230
Location: (using avr-gcc in) Finchingfield, Essex, England

Quote:

a periodic task function

If it's a "task" then set it to a low priority though I suppose this is going to affect the high priority tasks that were hoping to find some heap space?

_________________

Jump to:

All times are GMT + 1 Hour

AVR Freaks Forum Index » AVR (8-bit) Technical Forums » AVR forum

AVRFreaks.net

Search