View topic - umm... malloc fail :(

Search

umm... malloc fail :(
1, 2 Next All

AVR Freaks Forum Index » AVR (8-bit) Technical Forums » AVR forum

Author

Message

luvocean1

Posted: Feb 03, 2012 - 10:35 AM

Joined: Dec 26, 2006
Posts: 1407
Location: Sydney, Australia

Ok I am aware of the memory fragmentation after using malloc multiple times and freeing multiple times.

Whats the way around it? I am allocating and deallocating 2K ish memory size for each of my tcp sockets tx and rx buffers. So all up 4k memory for each sockets. After sockets are closed I free the memory. But it seems after about doing a few of these my malloc fails and returns with NULL Sad

Is this normal?

clawson

Posted: Feb 03, 2012 - 10:40 AM

Joined: Jul 18, 2005
Posts: 62354
Location: (using avr-gcc in) Finchingfield, Essex, England

Why not pre allocate an array of packets and just use an array entry when you need one?

If the packets are variable size make the array entries as large as the largest one you will get and just waste the final few bytes for smaller packets.

You will know at compile time how many such array elements the system can cater for and if you need more you know you need to trade up to a larger device or add RAM (if technically possible).

As you've found fragmentation is, otherwise, quite a problem on micros with only a few K of memory unless all malloc()s and free()s are the same size (in which case it'll just keep reallocating the same blocks). In fact this would be another solution - make all the malloc's as big as the biggest one.

Another idea would be to "clear the heap" from time to time. When you get a lull in activity simply free() everything that's been allocated and the heap will return to its original "clean" condition.

_________________

luvocean1

Posted: Feb 03, 2012 - 10:43 AM

Joined: Dec 26, 2006
Posts: 1407
Location: Sydney, Australia

Ah seems I was not freeing the actual tx rx buffers in the sockets! I was freeing other data structures except for the tx rx buffers! grrr... anyways so I can now seem to be able to create sockets indefinitely...

I wonder if I could still come across the fragmentation issue... ?

clawson

Posted: Feb 03, 2012 - 10:49 AM

Joined: Jul 18, 2005
Posts: 62354
Location: (using avr-gcc in) Finchingfield, Essex, England

Are the allocations of different size or not?

BTW when we develop we often do complete system simulation in a PC app with I/O "faked" and you can then implement a malloc()/free() that not only makes the allocation but draws a picture of the heap as dots on a window rectangle (maybe one pixel = 16 bytes or something?). As bytes are allocated paint the equivalent pixels red. When freed return to background colour. You then get a visual representation of how the heap is being used and you can see when it starts to fragment and so on.

_________________

david.prentice

Posted: Feb 03, 2012 - 11:33 AM

Joined: Feb 12, 2005
Posts: 16324
Location: Wormshill, England

Quote:

I wonder if I could still come across the fragmentation issue... ?

You can often create a worst case on paper. e.g. a combination of some big and many small blocks.

You can devise a worst case that could be allocated in the worst way.

If you can run out of memory, then one day this worst case will actually happen in real life.

Cliff's suggestion of a static array is often a very good solution.

David.

luvocean1

Posted: Feb 03, 2012 - 11:49 AM

Joined: Dec 26, 2006
Posts: 1407
Location: Sydney, Australia

Thanks guys.... but the point is as long as i free ALL allocations then fragmentation is not a problem right?

Lets say I create sockets of size 2K then some with 4K and some with 1K buffers. As long as all sockets get processed and thus removed and freed all buffers then I have no issues with fragmentation right?

JohanEkdahl

Posted: Feb 03, 2012 - 12:00 PM

Joined: Mar 27, 2002
Posts: 18581
Location: Lund, Sweden

Quote:

but the point is as long as i free ALL allocations then fragmentation is not a problem right?

Right. You can even push that a bit further: As long as you free in the reverse order of the allocation order you will have no fragmentation.

sternst

Posted: Feb 03, 2012 - 12:04 PM

Joined: Jul 23, 2001
Posts: 2439
Location: Osnabrueck, Germany

JohanEkdahl wrote:

Quote:

but the point is as long as i free ALL allocations then fragmentation is not a problem right?

Right. You can even push that a bit further: As long as you free in the reverse order of the allocation order you will have no fragmentation.

Why should the order matter? Consecutive free blocks are merged, so at the end there is only one big free block no matter of the freeing order.

_________________
Stefan Ernst

bobgardner

Posted: Feb 03, 2012 - 12:06 PM

Joined: Sep 04, 2002
Posts: 21272
Location: Orlando Florida

What model of avr you using (just curious?)

_________________
Imagecraft compiler user

clawson

Posted: Feb 03, 2012 - 12:50 PM

Joined: Jul 18, 2005
Posts: 62354
Location: (using avr-gcc in) Finchingfield, Essex, England

Quote:

What model of avr you using (just curious?)

Peering into my crystal ball I actually see a UC3

(luvocean1 likes to ask his "generic" questions here rather than the AVR32 fora as apparently he gets a better response Wink

The problems of malloc() are likely to be the same everywhere though it obviously gets worse the less RAM you have))

_________________

JohanEkdahl

Posted: Feb 03, 2012 - 03:19 PM

Joined: Mar 27, 2002
Posts: 18581
Location: Lund, Sweden

Quote:

Why should the order matter? Consecutive free blocks are merged, so at the end there is only one big free block no matter of the freeing order.

OK, let's straighten this out: At any point in the execution of the program, if dynamic memory is freed in the reverse order from how it was malloc'ed there will never be any fragmentation problems.

Better now?

clawson

Posted: Feb 03, 2012 - 03:30 PM

Joined: Jul 18, 2005
Posts: 62354
Location: (using avr-gcc in) Finchingfield, Essex, England

That's true but the implication is that it's the only way to clear fragmentation. The point (I think) was that if you free() everything then it doesn't matter what order you do it in as everything will be merged back into one amorphous great lump.

_________________

luvocean1

Posted: Feb 03, 2012 - 10:42 PM

Joined: Dec 26, 2006
Posts: 1407
Location: Sydney, Australia

clawson wrote:

Quote:

What model of avr you using (just curious?)

Peering into my crystal ball I actually see a UC3

(luvocean1 likes to ask his "generic" questions here rather than the AVR32 fora as apparently he gets a better response Wink

The problems of malloc() are likely to be the same everywhere though it obviously gets worse the less RAM you have))

Clawson's witchcraft prediction is correct Razz

So as long as ALL the sockets eventually gets closed I shouldnt have any memory issue.

Another question...what about for data structures I define not as malloced pointers but rather global variables? Where do they get stored? not heap?

mnehpets

Posted: Feb 03, 2012 - 11:28 PM

Joined: Nov 09, 2011
Posts: 402

Quote:

Another question...what about for data structures I define not as malloced pointers but rather global variables? Where do they get stored? not heap?

The static data sections (.data and .bss). Variables get allocated there by the linker. There's discussion (for the 8 bit AVRs, but the same happens in almost all microcontrollers running C code) here:

http://www.nongnu.org/avr-libc/user-manual/malloc.html

- S

luvocean1

Posted: Feb 04, 2012 - 01:37 AM

Joined: Dec 26, 2006
Posts: 1407
Location: Sydney, Australia

hhhm..thanks for that.
So I should try to define data structures as global variables where possible...

clawson

Posted: Feb 04, 2012 - 12:37 PM

Joined: Jul 18, 2005
Posts: 62354
Location: (using avr-gcc in) Finchingfield, Essex, England

Quote:

So I should try to define data structures as global variables where possible...

Not necessarily but in a limited RAM environment it does mean you'll have a clearer picture of what your RAM use will be when you build/link as the .map file will show the allocation of globals in .data and ,bss (and the avr(32)-size tool will report totals in those sections).

_________________

luvocean1

Posted: Feb 04, 2012 - 11:43 PM

Joined: Dec 26, 2006
Posts: 1407
Location: Sydney, Australia

OK.

Whats a good way to store strings you can print using Usart debug to hyperterminal? I have lots of them...

does having them as global static const keep them only in code and not in RAM? Quite often I just do UsartSendString("hello worl") in my functions or subfunctions....where do these strings get stored?

clawson

Posted: Feb 05, 2012 - 01:55 PM

Joined: Jul 18, 2005
Posts: 62354
Location: (using avr-gcc in) Finchingfield, Essex, England

Quote:

Whats a good way to store strings you can print using Usart debug to hyperterminal? I have lots of them...

This is where you might be better posting in the UC3 forum. I haven't a clue what architecture it is? I'm assuming it's Harvard like the AVR8? I further assume it's got an equivalent of __attribute__((progmem)) ? Obviously you don't want to waste precious RAM for strings that are copied from flash to .data

If you do use:

Code:

UsartSendString("hello world")

then by default that "hellow world" will be held in flash then as C starts up it will be copied to RAM then a pointer to its location there will be passed into the above call to UsartSendString(). In AVR8 the avr/pgmspace.h support offers

Code:

UsartSendString(PSTR("hello world"))

And the UsartSendString() would be written to dereference tha passed pointer as a code, not RAM address. Again I haven't a clue how this is handled in UC3 land.

_________________

digitalDan

Posted: Feb 05, 2012 - 07:23 PM

Joined: Nov 01, 2008
Posts: 191

Under the hood the UC3 is a modified Harvard Architecture. The flash and RAM live in the same address space although at different and non overlapping address ranges obviously. From a programmer's perspective you can't really distinguish it from von Neumann except that the flash is read only. You can put executable code in the RAM and the UC3 will happily execute it from there.
Declaring an initialized variable as const will automatically put it in flash (with the Atmel supplied tools).

Code:

const char *hw = "hello world";

const int intArray[] = {1,2,3,4,5};

I'm not sure how string constants are handled in the case you cite above. I'll have to spend a little time with an assembly listing. Wink

edit: after looking through some disassembled code, something like this:

Code:

usart_write_line (&AVR32_USART1, "hello world");

causes the string constant to be stored in flash and the address of that string in flash is passed as a parameter to the function.

_________________
Letting the smoke out since 1978

luvocean1

Posted: Feb 08, 2012 - 01:17 AM

Joined: Dec 26, 2006
Posts: 1407
Location: Sydney, Australia

Thanks Clawson and digitaldan...so if I pass the string inside the UsartSendString(), the string is placed in the flash section. So it shouldnt be affecting my RAM size.

Coming back to the original topic here... I did end up finding a section in the code where I was dereferencing a NULL unallocated memory area...this may have cause follow on issues I guess and then eventually corrupting the heap and causing another malloc to freeze the micro Smile

Jump to:

All times are GMT + 1 Hour

1, 2 Next All

AVR Freaks Forum Index » AVR (8-bit) Technical Forums » AVR forum

AVRFreaks.net

Search