For at exiting project the question came up, are there a boot loader, that can receive compressed data?
It's for a GCC project, where the program take about 60% of the space, and it would be nice if we could have two versions in the flash at the same time.
The code the compiler generate use the same instructions over and over again, so it should not be that hard to make, but I don't want to reinvent the wheel.
Where were you going to unpack TO?
If you have two compressed images in flash, each taking 30% of the space, you clearly can't execute the compressed code. You can only execute uncompressed code, that also must be in flash. And there certainly isn't room.
There are bootloaders that boot from SD or Serial Flash, which usually make compression unnecessary, since external flash tends to be very large compared to internal flash.
cisco routers used to store compressed images in EPROM, but they decompressed to, and ran from, RAM. It was quite a life-saver in the days of 4x512kbit EPROMs (256kByte) and 1MB RAM. Later hardware had more and bigger EPROM sockets (and eventually flash), but I imagine the some platforms still support compressed images.
Lets say bootloader 5%
app 60%
compressed app 30%
Then when you want to change app you use the bootloader to get it into flash, without harming the old running app.
When everything seems ok and ready (crc ok etc so we know it will work) then the botaloader, unpack the 30% and place them in the app area.
Or some will keep a fail safe app. (first version), in the 30% and if an update fail the bootloader should be able to make a safe recovery.
I once took a long journey by ship, and after the boatloader had finished its job, the chore I most despised was unpacking.
Then I will ask are there some interest in one?
If yes, where are there some hex files (or binary) files generated with the gcc with "normal" code, so I can find out what that is and how much it can be compressed with 100, 200 1000 ... byte of code.
What form of compression / decompression were you thinking of using? Simple RLE? Huffman? Dictionary (LZW etc)? The usual trade off with compression / decompression is the better it is the more RAM usage it costs. AVRs don't have a lot of RAM.
You would probably have to make one. This could be complicated as you don't have enough ram to hold the whole compressed code while you uncompress it into flash. Unless, of course, your uncompressed code is very small. Calls for careful consideration.
If you don't know my whole story, keep your mouth shut.
If you know my whole story, you're an accomplice. Keep your mouth shut. 
As I say there are a lot of repeats (system) in the GCC output. But because of the AVR's instruction set is 16 bit, with an odd bit use, it will be something special for the AVR.
but I think like make some repeated patterns for int (16 bit). thinking of code like
sts r24,0x300 sts r25,0x301
there are only 2 bits that are different in the two 32 bit combinations. (and it there are a system in the order), so most int operations will have a known 2. instruction (but not always the same ADC will follow ADD).
I don't think a huffman will be very good, but perhaps shuffle bit's round, and make huffman on 8 of the bits of each instruction I don't know.
But before I would spend time on it, I will like to get some "normal" programs , and run some tests on a PC.
Are there some good suggestions of some projects ? (HEX or binary files).
Traditionally, binary code compresses "poorly." While you might think there is a lot of repetition, it doesn't compare to (say) english text.
An 8k arduino sketch I had lying around will compress to about 75% of original size using unix Compress. And that's pretty typical, IIRC.
But look at #9 that code can't be compressed by a normal algorithm, because of the way the bit changes are placed. And I can make to about 1/2 the size.
And the change that the same variable are loaded with LDS a normal algorithm don't see a repeated pattern, and there are only 2 out off 64 bit that have changed!
First I have the order wrong in #9 it store r25 first, and then r24 (LDS is done in the order as #9).
A side question to GCC, it looks like for an int r24 always are stored in a even addr. and r25 in a odd. is that always correct ? (for a 8 bit micro it doesn't need to be that way )
I just zip'ed (normal deflate) the file and it went from 19.3K downto 8.6K.
And the best 7zip seems to be able to do is about 7.5k with LZMA
So I think that there are hope for an about 1/2 size with something like 500 words code :)
A side question to GCC, it looks like for an int r24 always are stored in a even addr. and r25 in a odd. is that always correct ? (for a 8 bit micro it doesn't need to be that way )
I guess what you are asking is do (u)int16_t always use ".align 2". No I don't believe that to be the case. Consider this example:
struct {
int n1;
char c;
int n2;
} foo;It doesn't matter whether foo starts on an even or odd byte one of those int's will be even/odd and the other will be odd/even (because of the byte packing).
I just zip'ed (normal deflate) the file and it went from 19.3K downto 8.6K.
And the best 7zip seems to be able to do is about 7.5k with LZMA
So I think that there are hope for an about 1/2 size with something like 500 words code :)
Did you zip the .hex file where each binary byte is expanded into two hex characters and only about 18 different characters are used?
If you don't know my whole story, keep your mouth shut.
If you know my whole story, you're an accomplice. Keep your mouth shut.
ok thanks.
But it seems to be true for normal global int's.
Are there some there could try to see the size of a "typical" binary file , just with a normal zip deflate.
I just went to a LUFA tree where there are about 50+ .hex files after you have used "make" on the complete tree to build everything. The biggest one of the lot seems to be a Webserver example:
uid23021@lxl0131u:~/windows/LUFA-130901/Projects/Webserver$ ls Web* -ld -rwxr-xr-x 1 uid23021 domain_users 33532 Apr 13 13:24 Webserver.bin -rw------- 1 uid23021 domain_users 2276 Jun 12 2013 Webserver.c -rw-r--r-- 1 uid23021 domain_users 19732 Apr 13 13:24 Webserver.d -rw-r--r-- 1 uid23021 domain_users 13 Apr 13 13:24 Webserver.eep -rwxr-xr-x 1 uid23021 domain_users 424433 Apr 13 13:24 Webserver.elf -rw------- 1 uid23021 domain_users 2651 May 24 2013 Webserver.h -rw-r--r-- 1 uid23021 domain_users 94338 Apr 13 13:24 Webserver.hex -rw-r--r-- 1 uid23021 domain_users 1015518 Apr 13 13:24 Webserver.lss -rw-r--r-- 1 uid23021 domain_users 219024 Apr 13 13:24 Webserver.map -rw-r--r-- 1 uid23021 domain_users 6264 Apr 13 13:24 Webserver.o -rw-r--r-- 1 uid23021 domain_users 15667 Apr 13 13:24 Webserver.sym -rw------- 1 uid23021 domain_users 5532 Jun 16 2013 Webserver.txt
Rather usefully Dean builds BOTH a .hex and a .bin so I don't even need to do an "avr-objcopy -I intel -O binary foo.hex foo.bin" to convert.
For the 33,532 byte input file I get:
/Webserver$ ls demo* -ld -rw-r--r-- 1 uid23021 domain_users 18255 Apr 20 13:10 demo.7z -rwxr-xr-x 1 uid23021 domain_users 21135 Apr 13 13:24 demo.bz2 -rw-r--r-- 1 uid23021 domain_users 20210 Apr 20 13:14 demo.zip
So around about 40% saving each time.
If it were me I think I'd amass as many AVR .bin files as I could then perform a static Huffman analysis over them all then use that tree built into the code. This means the code tree does not need to be built in RAM if it is fixed.
yes but I don't think that will be good :( because of the placement of the bits in the instruction set (ldi r24,10 , and ldi r24,9 don't share a byte for ldi!),
so for that to work you would need to shuffle the bits round so they do.
are those files somewhere on the net so I don't need to compile them?
The LUFA ones? Nope I don't think he version manages the build output.
But (assuming avr-gcc in installed) it really is as simple as pulling and unpacking the source then just typing "make" at the top level - It then builds everything:
/LUFA-130901$ find . -name \*.bin -print ./Bootloaders/Printer/BootloaderPrinter.bin ./Bootloaders/HID/BootloaderHID.bin ./Bootloaders/CDC/BootloaderCDC.bin ./Bootloaders/MassStorage/BootloaderMassStorage.bin ./Bootloaders/DFU/BootloaderDFU.bin ./Projects/RelayBoard/RelayBoard.bin ./Projects/Benito/Benito.bin ./Projects/TempDataLogger/TempDataLogger.bin ./Projects/MIDIToneGenerator/MIDIToneGenerator.bin ./Projects/MissileLauncher/MissileLauncher.bin ./Projects/LEDNotifier/LEDNotifier.bin ./Projects/AVRISP-MKII/AVRISP-MKII.bin ./Projects/XPLAINBridge/XPLAINBridge.bin ./Projects/SerialToLCD/SerialToLCD.bin ./Projects/USBtoSerial/USBtoSerial.bin ./Projects/Magstripe/Magstripe.bin ./Projects/Webserver/Webserver.bin ./Projects/HIDReportViewer/HIDReportViewer.bin ./Projects/MediaController/MediaController.bin ./Demos/Host/ClassDriver/PrinterHost/PrinterHost.bin ./Demos/Host/ClassDriver/MouseHostWithParser/MouseHostWithParser.bin ./Demos/Host/ClassDriver/AudioOutputHost/AudioOutputHost.bin ./Demos/Host/ClassDriver/AndroidAccessoryHost/AndroidAccessoryHost.bin ./Demos/Host/ClassDriver/StillImageHost/StillImageHost.bin ./Demos/Host/ClassDriver/MassStorageHost/MassStorageHost.bin ./Demos/Host/ClassDriver/KeyboardHostWithParser/KeyboardHostWithParser.bin ./Demos/Host/ClassDriver/RNDISEthernetHost/RNDISEthernetHost.bin ./Demos/Host/ClassDriver/AudioInputHost/AudioInputHost.bin ./Demos/Host/ClassDriver/KeyboardHost/KeyboardHost.bin ./Demos/Host/ClassDriver/MouseHost/MouseHost.bin ./Demos/Host/ClassDriver/VirtualSerialHost/VirtualSerialHost.bin ./Demos/Host/ClassDriver/JoystickHostWithParser/JoystickHostWithParser.bin ./Demos/Host/ClassDriver/MIDIHost/MIDIHost.bin ./Demos/Host/LowLevel/PrinterHost/PrinterHost.bin ./Demos/Host/LowLevel/MouseHostWithParser/MouseHostWithParser.bin ./Demos/Host/LowLevel/AudioOutputHost/AudioOutputHost.bin ./Demos/Host/LowLevel/AndroidAccessoryHost/AndroidAccessoryHost.bin ./Demos/Host/LowLevel/StillImageHost/StillImageHost.bin ./Demos/Host/LowLevel/MassStorageHost/MassStorageHost.bin ./Demos/Host/LowLevel/KeyboardHostWithParser/KeyboardHostWithParser.bin ./Demos/Host/LowLevel/RNDISEthernetHost/RNDISEthernetHost.bin ./Demos/Host/LowLevel/AudioInputHost/AudioInputHost.bin ./Demos/Host/LowLevel/GenericHIDHost/GenericHIDHost.bin ./Demos/Host/LowLevel/KeyboardHost/KeyboardHost.bin ./Demos/Host/LowLevel/MouseHost/MouseHost.bin ./Demos/Host/LowLevel/VirtualSerialHost/VirtualSerialHost.bin ./Demos/Host/LowLevel/JoystickHostWithParser/JoystickHostWithParser.bin ./Demos/Host/LowLevel/MIDIHost/MIDIHost.bin ./Demos/Device/ClassDriver/DualVirtualSerial/DualVirtualSerial.bin ./Demos/Device/ClassDriver/RNDISEthernet/RNDISEthernet.bin ./Demos/Device/ClassDriver/Mouse/Mouse.bin ./Demos/Device/ClassDriver/VirtualSerialMouse/VirtualSerialMouse.bin ./Demos/Device/ClassDriver/Joystick/Joystick.bin ./Demos/Device/ClassDriver/MIDI/MIDI.bin ./Demos/Device/ClassDriver/AudioOutput/AudioOutput.bin ./Demos/Device/ClassDriver/AudioInput/AudioInput.bin ./Demos/Device/ClassDriver/GenericHID/GenericHID.bin ./Demos/Device/ClassDriver/MassStorage/MassStorage.bin ./Demos/Device/ClassDriver/VirtualSerial/VirtualSerial.bin ./Demos/Device/ClassDriver/KeyboardMouseMultiReport/KeyboardMouseMultiReport.bin ./Demos/Device/ClassDriver/VirtualSerialMassStorage/VirtualSerialMassStorage.bin ./Demos/Device/ClassDriver/MassStorageKeyboard/MassStorageKeyboard.bin ./Demos/Device/ClassDriver/KeyboardMouse/KeyboardMouse.bin ./Demos/Device/ClassDriver/Keyboard/Keyboard.bin ./Demos/Device/LowLevel/DualVirtualSerial/DualVirtualSerial.bin ./Demos/Device/LowLevel/RNDISEthernet/RNDISEthernet.bin ./Demos/Device/LowLevel/Mouse/Mouse.bin ./Demos/Device/LowLevel/Joystick/Joystick.bin ./Demos/Device/LowLevel/MIDI/MIDI.bin ./Demos/Device/LowLevel/AudioOutput/AudioOutput.bin ./Demos/Device/LowLevel/AudioInput/AudioInput.bin ./Demos/Device/LowLevel/GenericHID/GenericHID.bin ./Demos/Device/LowLevel/MassStorage/MassStorage.bin ./Demos/Device/LowLevel/VirtualSerial/VirtualSerial.bin ./Demos/Device/LowLevel/KeyboardMouse/KeyboardMouse.bin ./Demos/Device/LowLevel/Keyboard/Keyboard.bin ./Demos/DualRole/ClassDriver/MouseHostDevice/MouseHostDevice.bin
ok thanks, but I should not do that on this computer :)
other good .bin (or Hex files)?