umm... malloc fail :(

Go To Last Post
29 posts / 0 new
Author
Message
#1
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Ok I am aware of the memory fragmentation after using malloc multiple times and freeing multiple times.

Whats the way around it? I am allocating and deallocating 2K ish memory size for each of my tcp sockets tx and rx buffers. So all up 4k memory for each sockets. After sockets are closed I free the memory. But it seems after about doing a few of these my malloc fails and returns with NULL :(

Is this normal?

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Why not pre allocate an array of packets and just use an array entry when you need one?

If the packets are variable size make the array entries as large as the largest one you will get and just waste the final few bytes for smaller packets.

You will know at compile time how many such array elements the system can cater for and if you need more you know you need to trade up to a larger device or add RAM (if technically possible).

As you've found fragmentation is, otherwise, quite a problem on micros with only a few K of memory unless all malloc()s and free()s are the same size (in which case it'll just keep reallocating the same blocks). In fact this would be another solution - make all the malloc's as big as the biggest one.

Another idea would be to "clear the heap" from time to time. When you get a lull in activity simply free() everything that's been allocated and the heap will return to its original "clean" condition.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Ah seems I was not freeing the actual tx rx buffers in the sockets! I was freeing other data structures except for the tx rx buffers! grrr... anyways so I can now seem to be able to create sockets indefinitely...

I wonder if I could still come across the fragmentation issue... ?

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Are the allocations of different size or not?

BTW when we develop we often do complete system simulation in a PC app with I/O "faked" and you can then implement a malloc()/free() that not only makes the allocation but draws a picture of the heap as dots on a window rectangle (maybe one pixel = 16 bytes or something?). As bytes are allocated paint the equivalent pixels red. When freed return to background colour. You then get a visual representation of how the heap is being used and you can see when it starts to fragment and so on.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Quote:
I wonder if I could still come across the fragmentation issue... ?

You can often create a worst case on paper. e.g. a combination of some big and many small blocks.

You can devise a worst case that could be allocated in the worst way.

If you can run out of memory, then one day this worst case will actually happen in real life.

Cliff's suggestion of a static array is often a very good solution.

David.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Thanks guys.... but the point is as long as i free ALL allocations then fragmentation is not a problem right?

Lets say I create sockets of size 2K then some with 4K and some with 1K buffers. As long as all sockets get processed and thus removed and freed all buffers then I have no issues with fragmentation right?

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Quote:

but the point is as long as i free ALL allocations then fragmentation is not a problem right?

Right. You can even push that a bit further: As long as you free in the reverse order of the allocation order you will have no fragmentation.

As of January 15, 2018, Site fix-up work has begun! Now do your part and report any bugs or deficiencies here

No guarantees, but if we don't report problems they won't get much of  a chance to be fixed! Details/discussions at link given just above.

 

"Some questions have no answers."[C Baird] "There comes a point where the spoon-feeding has to stop and the independent thinking has to start." [C Lawson] "There are always ways to disagree, without being disagreeable."[E Weddington] "Words represent concepts. Use the wrong words, communicate the wrong concept." [J Morin] "Persistence only goes so far if you set yourself up for failure." [Kartman]

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

JohanEkdahl wrote:
Quote:

but the point is as long as i free ALL allocations then fragmentation is not a problem right?

Right. You can even push that a bit further: As long as you free in the reverse order of the allocation order you will have no fragmentation.
Why should the order matter? Consecutive free blocks are merged, so at the end there is only one big free block no matter of the freeing order.

Stefan Ernst

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

What model of avr you using (just curious?)

Imagecraft compiler user

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Quote:

What model of avr you using (just curious?)

Peering into my crystal ball I actually see a UC3

(luvocean1 likes to ask his "generic" questions here rather than the AVR32 fora as apparently he gets a better response ;-) The problems of malloc() are likely to be the same everywhere though it obviously gets worse the less RAM you have))

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Quote:

Why should the order matter? Consecutive free blocks are merged, so at the end there is only one big free block no matter of the freeing order.

OK, let's straighten this out: At any point in the execution of the program, if dynamic memory is freed in the reverse order from how it was malloc'ed there will never be any fragmentation problems.

Better now?

As of January 15, 2018, Site fix-up work has begun! Now do your part and report any bugs or deficiencies here

No guarantees, but if we don't report problems they won't get much of  a chance to be fixed! Details/discussions at link given just above.

 

"Some questions have no answers."[C Baird] "There comes a point where the spoon-feeding has to stop and the independent thinking has to start." [C Lawson] "There are always ways to disagree, without being disagreeable."[E Weddington] "Words represent concepts. Use the wrong words, communicate the wrong concept." [J Morin] "Persistence only goes so far if you set yourself up for failure." [Kartman]

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

That's true but the implication is that it's the only way to clear fragmentation. The point (I think) was that if you free() everything then it doesn't matter what order you do it in as everything will be merged back into one amorphous great lump.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

clawson wrote:
Quote:

What model of avr you using (just curious?)

Peering into my crystal ball I actually see a UC3

(luvocean1 likes to ask his "generic" questions here rather than the AVR32 fora as apparently he gets a better response ;-) The problems of malloc() are likely to be the same everywhere though it obviously gets worse the less RAM you have))

Clawson's witchcraft prediction is correct :P
So as long as ALL the sockets eventually gets closed I shouldnt have any memory issue.

Another question...what about for data structures I define not as malloced pointers but rather global variables? Where do they get stored? not heap?

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Quote:

Another question...what about for data structures I define not as malloced pointers but rather global variables? Where do they get stored? not heap?

The static data sections (.data and .bss). Variables get allocated there by the linker. There's discussion (for the 8 bit AVRs, but the same happens in almost all microcontrollers running C code) here:

http://www.nongnu.org/avr-libc/u...

- S

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

hhhm..thanks for that.
So I should try to define data structures as global variables where possible...

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Quote:

So I should try to define data structures as global variables where possible...

Not necessarily but in a limited RAM environment it does mean you'll have a clearer picture of what your RAM use will be when you build/link as the .map file will show the allocation of globals in .data and ,bss (and the avr(32)-size tool will report totals in those sections).

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

OK.

Whats a good way to store strings you can print using Usart debug to hyperterminal? I have lots of them...

does having them as global static const keep them only in code and not in RAM? Quite often I just do UsartSendString("hello worl") in my functions or subfunctions....where do these strings get stored?

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Quote:

Whats a good way to store strings you can print using Usart debug to hyperterminal? I have lots of them...

This is where you might be better posting in the UC3 forum. I haven't a clue what architecture it is? I'm assuming it's Harvard like the AVR8? I further assume it's got an equivalent of __attribute__((progmem)) ? Obviously you don't want to waste precious RAM for strings that are copied from flash to .data

If you do use:

UsartSendString("hello world")

then by default that "hellow world" will be held in flash then as C starts up it will be copied to RAM then a pointer to its location there will be passed into the above call to UsartSendString(). In AVR8 the avr/pgmspace.h support offers

UsartSendString(PSTR("hello world"))

And the UsartSendString() would be written to dereference tha passed pointer as a code, not RAM address. Again I haven't a clue how this is handled in UC3 land.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Under the hood the UC3 is a modified Harvard Architecture. The flash and RAM live in the same address space although at different and non overlapping address ranges obviously. From a programmer's perspective you can't really distinguish it from von Neumann except that the flash is read only. You can put executable code in the RAM and the UC3 will happily execute it from there.
Declaring an initialized variable as const will automatically put it in flash (with the Atmel supplied tools).

const char *hw = "hello world";
const int intArray[] = {1,2,3,4,5};

I'm not sure how string constants are handled in the case you cite above. I'll have to spend a little time with an assembly listing. ;)

edit: after looking through some disassembled code, something like this:

usart_write_line (&AVR32_USART1, "hello world");

causes the string constant to be stored in flash and the address of that string in flash is passed as a parameter to the function.

Letting the smoke out since 1978

 

 

 

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Thanks Clawson and digitaldan...so if I pass the string inside the UsartSendString(), the string is placed in the flash section. So it shouldnt be affecting my RAM size.

Coming back to the original topic here... I did end up finding a section in the code where I was dereferencing a NULL unallocated memory area...this may have cause follow on issues I guess and then eventually corrupting the heap and causing another malloc to freeze the micro :)

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Quote:

Thanks Clawson and digitaldan...so if I pass the string inside the UsartSendString(), the string is placed in the flash section. So it shouldnt be affecting my RAM size

That's what Dan's edit suggests.

If you want to know what IS using your RAM then study the map file (though it will only show global/static allocations and not automatics or the heap).

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Another question... is it possible to find out how much free memory is available in the heap? Any built in functions in C available for checking free memory?

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Well one way is to use a for() loop starting with a stupidly high value and keep trying to malloc(). That is:

char * p;
int i;
for (i=32767; i>0; i--) {
  p = (char *)malloc(i);
  if (p) break;
}
if (p) {
  free(p);
}

At the end of this i says how big the largest malloc() possible is.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

lol that is really scary!
Surely there has to be some low level keeping track of the heap? When calling malloc it must check whether the size of free available memory is bigger than the space asked for in malloc....

So free memory checking is inside malloc function... we just need to know what variable it checks may be?

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Well for AVR-LibC the source of malloc() is open:

http://svn.savannah.nongnu.org/v...

BTW while the C standard calls for malloc() and free() in stdlib it does not specify a freemem() type call or a specific form of implementation.

From a user perspective the repeated calls to malloc() I showed above really is the way to do it. My method will tell you the largest single block you can allocate. This may not be what you want. Perhaps you plan to make 100's of small malloc()s and there could still be some small blocks available after the large one is allocated so an alternative strategy is to just keep making "small" mallocs of N bytes until you get 0. Then the total is M * N where M is the number you made. Because you don't know how many you are likely to make and you need to free them all after then you probably want to build a linked list of allocations and then walk the list freeing them. This shows one way to do it:

http://www.java-samples.com/show...

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Quote:

Surely there has to be some low level keeping track of the heap? When calling malloc it must check whether the size of free available memory is bigger than the space asked for in malloc....

The implementations I've seen keep a linked list of free blocks, and I've seen no function that reports statistics on that. There is no total free size maintained.

The source of avr-gcc malloc, free and the heap data structure is public and browsable. Have you had a look?

As of January 15, 2018, Site fix-up work has begun! Now do your part and report any bugs or deficiencies here

No guarantees, but if we don't report problems they won't get much of  a chance to be fixed! Details/discussions at link given just above.

 

"Some questions have no answers."[C Baird] "There comes a point where the spoon-feeding has to stop and the independent thinking has to start." [C Lawson] "There are always ways to disagree, without being disagreeable."[E Weddington] "Words represent concepts. Use the wrong words, communicate the wrong concept." [J Morin] "Persistence only goes so far if you set yourself up for failure." [Kartman]

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Quote:

The source of avr-gcc malloc, free and the heap data structure is public and browsable. Have you had a look?

I'm kind of hoping he might have followed the link I gave but note this is the AVR8 implementation, not AVR32 though it's possible it might use similar code. If it does then note that MALLOC_TEST build switch in the source file which can build it into a complete, standalone test program. The stuff they have their that walks the list looks kind of useful and may be adaptable. (but again this is AVR8 not for UC3).

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

hhmmm.. interesting code! Malloc if not very efficient, the code for allocation seems expensive!

clawson... i was reluctant to use that for loop malloc method cos I was hoping to have a periodic task function that monitored the memory usage (free mem) on the cpu every now and then and made sure there arnt any memory leak etc....seems I cant do that then if I have to use those for looped malloc way...

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Quote:

a periodic task function

If it's a "task" then set it to a low priority though I suppose this is going to affect the high priority tasks that were hoping to find some heap space?