unreferenced PROGMEM causing "region text is full"

Go To Last Post
15 posts / 0 new
Author
Message
#1
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Greetings list,

I've got a source file that contains a large number of PROGMEM lookup tables, on the order of 50 or so. Together they amount to almost 12 KB. The file is #included into another source file, and preprocessor directives select exactly one table for referencing (based on F_CPU and some other information).

I'm compiling for a target that has 8K of flash. When linking, I get "region text is full". It seems that even though only one of the lookup tables is referenced by the compiled code, they are all being linked.

I'm compiling and linking with avr-gcc (4.3.2, also tried 4.3.4). The source files are in C. The #included file is a .h, the file that #includes it is a .c.

I've tried compiling with avr-g++. The error goes away, and the linked .elf contains only the referenced lookup table, but the compile stage reports about 50 warnings (one for each lookup table) "only initialised variables can be placed into program memory area."

This seems similar to:

andbut without the size issue.

Admittedly, I know only a little about the build process. I imagine there is a way to tell the compiler to omit unreferenced objects, but I don't know it.

Thoughts?

Many thanks,

Cheers,
JJ

"Experience is what enables you to recognise a mistake the second time you make it."

"Good judgement comes from experience.  Experience comes from bad judgement."

"Wisdom is always wont to arrive late, and to be a little approximate on first possession."

"When you hear hoofbeats, think horses, not unicorns."

"Fast.  Cheap.  Good.  Pick two."

"We see a lot of arses on handlebars around here." - [J Ekdahl]

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Quote:

the compiler to omit unreferenced objects

-fdata-sections then -gc-sections but I have a feeling this mechanism may post-date the ancient compiler you are trying to use.

However this should only be required if all the data is being linked. If it is protected in macros so that only one variant should be exposed to the compiler I'd suspect your macro mechanism and would -save-temps then study the .i file to see exactly what is being exposed to the C compiler.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

What do you mean by "referencing"? Show us a short example.
JW

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

clawson wrote:
-fdata-sections then -gc-sections but I have a feeling this mechanism may post-date the ancient compiler you are trying to use.
I'm already using those options. My compile command:
avr-gcc -Os -c -g -Wall -fno-exceptions -ffunction-sections -fdata-sections -mmcu=attiny85 -DF_CPU=16000000UL ${file} -o ${file}.o 

My link command:

avr-gcc -Os -Wl,--gc-sections -mmcu=attiny85 -o ${file}.elf ${file}.o -lm

Quote:
However this should only be required if all the data is being linked. If it is protected in macros so that only one variant should be exposed to the compiler I'd suspect your macro mechanism and would -save-temps then study the .i file to see exactly what is being exposed to the C compiler.
I suspected that as well, until testing proved otherwise. I removed the macros that select the one true lookup table, and all references to those macros. There are no references of any kind to any of the lookup tables. The problem persists.

What remains is a reference to different PROGMEM object, #included from a different file. Once I removed that, the problem disappeared. It would seem as though all of the PROGMEM objects, no matter their source, are linked even if only one of them is referenced. Is that not what -fdata-sections is supposed to prevent?

JJ

"Experience is what enables you to recognise a mistake the second time you make it."

"Good judgement comes from experience.  Experience comes from bad judgement."

"Wisdom is always wont to arrive late, and to be a little approximate on first possession."

"When you hear hoofbeats, think horses, not unicorns."

"Fast.  Cheap.  Good.  Pick two."

"We see a lot of arses on handlebars around here." - [J Ekdahl]

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

wek wrote:
What do you mean by "referencing"? Show us a short example.
JW
#included file contains things of this nature:
struct foo_t {
  uint8_t thing1;
  uint8_t thing2;
};

struct foo_t const bar_16000000[] PROGMEM =
{ { 0x00, 0x01 }, { 0x02, 0x03 }, };

struct foo_t const bar_8000000[] PROGMEM =
{ { 0x00, 0x02 }, { 0x04, 0x06 }, };

...etc...

#define concatenate(a, b) __concatenate(a, b)
#define __concatenate(a,b) a ## b

#define bar concatenate(bar_, F_CPU)

The #including file then contains references to the macro:

int i;
foo_t ding;
memcpy_P(&ding, &(bar), sizeof(foo_t));

However, as mentioned, even when removing the macro from the #included file and all references to it from the #including file, the problem persists. So long as [i]any PROGMEM object from any #included file gets referenced, then all PROGMEM objects are getting linked. Why?

JJ

"Experience is what enables you to recognise a mistake the second time you make it."

"Good judgement comes from experience.  Experience comes from bad judgement."

"Wisdom is always wont to arrive late, and to be a little approximate on first possession."

"When you hear hoofbeats, think horses, not unicorns."

"Fast.  Cheap.  Good.  Pick two."

"We see a lot of arses on handlebars around here." - [J Ekdahl]

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

But that's not building out the unrequired parts? Both bar_16000000 and bar_8000000 data will be created. So this presumably does depend on gc-sections in fact. Why not

#if F_CPU == 16000000
struct foo_t const bar_16000000[] PROGMEM = 
{ { 0x00, 0x01 }, { 0x02, 0x03 }, }; 
#else
struct foo_t const bar_8000000[] PROGMEM = 
{ { 0x00, 0x02 }, { 0x04, 0x06 }, };
#endif

Then the data you don't want simply wouldn't be seen by the C compiler. In fact I had assumed this is what you were describing above.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

clawson wrote:
But that's not building out the unrequired parts? Both bar_16000000 and bar_8000000 data will be created. So this presumably does depend on gc-sections in fact.
I am linking with gc-sections. Should that not be sufficient?
Quote:
Why not
#if F_CPU == 16000000
struct foo_t const bar_16000000[] PROGMEM = 
{ { 0x00, 0x01 }, { 0x02, 0x03 }, }; 
#else
struct foo_t const bar_8000000[] PROGMEM = 
{ { 0x00, 0x02 }, { 0x04, 0x06 }, };
#endif

I assume you mean:
#if F_CPU == 16000000
struct foo_t const bar[] PROGMEM = 
{ { 0x00, 0x01 }, { 0x02, 0x03 }, }; 
#else
struct foo_t const bar[] PROGMEM = 
{ { 0x00, 0x02 }, { 0x04, 0x06 }, };
#endif

Quote:
Then the data you don't want simply wouldn't be seen by the C compiler. In fact I had assumed this is what you were describing above.
Indeed, this approach would work for the case where I wish to build in only one table. However, another project that would #include the same file might require more than one of the tables, based on runtime changes to CLKPR. That's why I want to create all of the tables. The reference via macro is for this application only, and seems irrelevant to the problem.

The question remains: why does avr-gcc (with the options mentioned) fail to expunge the unreferenced tables, while avr-g++ with the exact same options succeeds? In particular, why does a reference to a PROGMEM object of a different type, #included from a different file, result in that failure?

JJ

"Experience is what enables you to recognise a mistake the second time you make it."

"Good judgement comes from experience.  Experience comes from bad judgement."

"Wisdom is always wont to arrive late, and to be a little approximate on first possession."

"When you hear hoofbeats, think horses, not unicorns."

"Fast.  Cheap.  Good.  Pick two."

"We see a lot of arses on handlebars around here." - [J Ekdahl]

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

But if you build all the tables on the basis that any may be required at runtime based on CLKPR then they are always going to occupy all the space aren't they?

Any what did the discard section in the .map file say when you thought some should be discarded?

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

clawson wrote:
But if you build all the tables on the basis that any may be required at runtime based on CLKPR then they are always going to occupy all the space aren't they?
I was simply proposing that a future project might deliberately refer to 2 or 3 of them, but not all 50. They should only occupy space if some other entity references them, even if they are built. That's what happens when I compile with avr-g++. But not when I compile with avr-gcc. Why?
Quote:
Any what did the discard section in the .map file say when you thought some should be discarded?
None of them appear under the discard section, they all appear under the "Linker script and memory map" section, under .text and .progmem.data.

Interestingly, compiling with --combine and -fwhole-program elicits the correct behaviour. Compiling with avr-g++ does not require these options to elicit the same correct behaviour. I'm still hoping someone can tell me why, and (other than using --combine and -fwhole-program) how to get avr-gcc to behave. Is this by design? What is the underlying difference between avr-gcc and avr-g++ that causes this?

Cheers,
JJ

"Experience is what enables you to recognise a mistake the second time you make it."

"Good judgement comes from experience.  Experience comes from bad judgement."

"Wisdom is always wont to arrive late, and to be a little approximate on first possession."

"When you hear hoofbeats, think horses, not unicorns."

"Fast.  Cheap.  Good.  Pick two."

"We see a lot of arses on handlebars around here." - [J Ekdahl]

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Quote:

None of them appear under the discard section, they all appear under the "Linker script and memory map" section, under .text and .progmem.data.

Add the cref option to the map generation. What is shown as cross referencing them?

Also what happens if you try a 4.6 or even 4.7 compiler. There have been many bug fixes in the last 3-4 years.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Your basic failure is in the assumption that PROGMEM-tagged data *are* "data" and that the compiler treats them as "data"; more specifically, that the -fdata-sections switch applies to them. It does not.

You should have inspected either the sections of the compiled object .o (using objdump with an appropriate set of switches), or maybe even better, the assembler output from the compiler (obtained either using -S or the keep-intermediates switch I don't remember but Cliff will be so kind to supply it :-) ) to see the sections assigned to individual PROGMEM variables. You'd see they are none and they are all assigned the same section (.progmem.data).

That this works in g++ is in fact a bug.

The workaround is to explicitly place the variables into separate sections, e.g.

#include 
#include 

volatile int a;

char __attribute__((section(".progmem.data.p1"))) p1[10]={'a'};
char __attribute__((section(".progmem.data.p2"))) p2[10]={'a'};
char  m1[10]={'a'};
char  m2[10]={'a'};

int main(void) {
  a = strlen_P(p1);
  a = strlen(m1);
}
c:\tmp>avr-gcc -Os -DF_CPU=14745600UL mmcu=atmega2561 -Wa,-adhlns=aw.lst -Wl,-Map=aw.map,--cref -o aw.elf -fdata-sections "-Wl,-gc-sections,--print-gc-sections"   aw.c
c:/program files/atmel/avrtools/winavr/bin/../lib/gcc/avr/4.2.2/../../../../avr/bin/ld.exe: Removing unused section '.progmem.data.p2' in file 'C:\Users\OM7ZZ\AppData\Local\Temp/cciOhHFh.o'
c:/program files/atmel/avrtools/winavr/bin/../lib/gcc/avr/4.2.2/../../../../avr/bin/ld.exe: Removing unused section '.data.m2' in file 'C:\Users\OM7ZZ\AppData\Local\Temp/cciOhHFh.o'
.text           0x00000000      0x15c
 *(.vectors)
 .vectors       0x00000000       0xcc c:/program files/atmel/avrtools/winavr/bin/../lib/gcc/avr/4.2.2/../../../../avr/lib/avr6/crtm2561.o
                0x00000000                __vectors
                0x00000000                __vector_default
 *(.vectors)
 *(.progmem.gcc*)
 *(.progmem*)
 .progmem.data.p1
                0x000000cc        0xa C:\Users\OM7ZZ\AppData\Local\Temp/cciOhHFh.o
                0x000000cc                p1
                0x000000d6                . = ALIGN (0x2)
                0x000000d6                __trampolines_start = .
 *(.trampolines)
 .trampolines   0x000000d6        0x0 linker stubs
 *(.trampolines*)
                0x000000d6                __trampolines_end = .

I'd guess in 4.7+ using the native _flash named address space -fdata-sections would work, but to try this is something left for the OP as homework.

JW

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

wek wrote:
Your basic failure is in the assumption that PROGMEM-tagged data *are* "data" and that the compiler treats them as "data"; more specifically, that the -fdata-sections switch applies to them. It does not.
Failure is such a harsh word :) ... I prefer to think of myself as ignorant. I made no such assumption (In truth, as mentioned, I don't know very much about the build process). Neither are functions 'data'. However:
-fdata-sections
    Place each function or data item into its own section in the output file if the target supports arbitrary sections.  The name of the function or the name of the data item determines the section’s name in the output file.

My real failure was in not realizing that -fdata-sections allows the linker to omit individual items by placing only one item per section, and then dropping the section. I imagined that sections could be split up by the linker.

Quote:
You should have inspected either the sections of the compiled object .o (using objdump with an appropriate set of switches), or maybe even better, the assembler output from the compiler (obtained either using -S or the keep-intermediates switch I don't remember but Cliff will be so kind to supply it :-) ) to see the sections assigned to individual PROGMEM variables. You'd see they are none and they are all assigned the same section (.progmem.data).
Did. I just didn't know that the single section was the root of the problem, nor what to do about it.
Quote:
That this works in g++ is in fact a bug.
Really! A darned convernient one. Can you point me to a bug report?
Quote:
The workaround is to explicitly place the variables into separate sections...
This, I like. Works like a charm, and fairly elegant. Thanks, Wek!
Quote:
I'd guess in 4.7+ using the native _flash named address space -fdata-sections would work, but to try this is something left for the OP as homework.
First I'll have to install it :)

Cheers,
JJ

"Experience is what enables you to recognise a mistake the second time you make it."

"Good judgement comes from experience.  Experience comes from bad judgement."

"Wisdom is always wont to arrive late, and to be a little approximate on first possession."

"When you hear hoofbeats, think horses, not unicorns."

"Fast.  Cheap.  Good.  Pick two."

"We see a lot of arses on handlebars around here." - [J Ekdahl]

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

clawson wrote:
Add the cref option to the map generation. What is shown as cross referencing them?
They're all associated with the .o file generated from the source file into which they were #included. The map file isn't more specific than that.

"Experience is what enables you to recognise a mistake the second time you make it."

"Good judgement comes from experience.  Experience comes from bad judgement."

"Wisdom is always wont to arrive late, and to be a little approximate on first possession."

"When you hear hoofbeats, think horses, not unicorns."

"Fast.  Cheap.  Good.  Pick two."

"We see a lot of arses on handlebars around here." - [J Ekdahl]

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

As Jan says put them in named sections. If there are 50 then rather than 50 lots of -section-start it would then pay to take a copy of the linker script and add a sort of *.mydata.* entry or similar to pull them all in. Then the gc-sections should throw out any that are unreferenced.

The way -fdata-sections and -gc-sections effectively works is that if you have:

int foo;
int bar;

it will place foo into .data.foo an bar into .data.bar. When the code is linked and gc-sections is passed the linker will count references on .data.foo and .data.bar and if either remains 0 at the end of the link then the gc-sections (gc=garbage collect) will discard them.

On the other hand if you write:

int foo = 23 PROGMEM;
int bar = 39 PROGMEM;

then BOTH will be placed into .progmem. If something accesess foo but not bar then the count on .progmem references will still be 1 so the linker will not discard the whole of .progmem

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

clawson wrote:
As Jan says put them in named sections. If there are 50 then rather than 50 lots of -section-start it would then pay to take a copy of the linker script and add a sort of *.mydata.* entry or similar to pull them all in. Then the gc-sections should throw out any that are unreferenced.
Is not -section-start used for placing a section at an absolute address? I have no need to control the placement of these lookup tables.

While 50 tables would be a lot to maintain manually, I generate them automatically on a host with another C program. It was easy to get it to output unique section names for each table.

Quote:
The way -fdata-sections and -gc-sections effectively works is that if you have:

int foo;
int bar;

it will place foo into .data.foo an bar into .data.bar. When the code is linked and gc-sections is passed the linker will count references on .data.foo and .data.bar and if either remains 0 at the end of the link then the gc-sections (gc=garbage collect) will discard them.

On the other hand if you write:

int foo = 23 PROGMEM;
int bar = 39 PROGMEM;

then BOTH will be placed into .progmem. If something accesess foo but not bar then the count on .progmem references will still be 1 so the linker will not discard the whole of .progmem

I'll have to do some reading on avr-ld and the build process in general. Thanks for the clear explanation.

And once again, thank you all for your speedy help.

Cheers,
JJ

"Experience is what enables you to recognise a mistake the second time you make it."

"Good judgement comes from experience.  Experience comes from bad judgement."

"Wisdom is always wont to arrive late, and to be a little approximate on first possession."

"When you hear hoofbeats, think horses, not unicorns."

"Fast.  Cheap.  Good.  Pick two."

"We see a lot of arses on handlebars around here." - [J Ekdahl]