AVR Freaks

To simply build your code I dummied some things that weren't shown:

#define safe_malloc malloc

typedef struct size24 {
    uint8_t r;
    uint8_t g;
    uint8_t b;
} size24_t;

void set_value(struct size24 * buf, uint32_t val) {
    buf->r = val >> 16;
    buf->g = val >> 8;
    buf->b = val;

}

Then used some test data:

char input[] = { 1, 3, 3, 4, 3, 7, 7, 7, 2, 2, 9, 9, 9, 9, 9 };

int main()
{
    encode_rle(input, sizeof(input));
}

and get the result:

loop 67
loop 68
loop 69
loop 70
loop 71
loop 72
loop 73
loop 74
loop 75
loop 76
loop 77
loop 78
loop 79
loop 80
loop 81
loop 82
loop 83
loop 84
loop 85
loop 86
loop 87
File Size before compression 15
File Size after compression  24

So, yes, you have serious problems! I would liberally sprinkle the code with printf()s to see which code paths it is taking and how the pointers are incrementing. Having done that:

    while ((inBuffer - inStart) < inSize) {

        // detected a string of three matching pixels, begin to traverse the pixels if they match
        if (memcmp(&pixel1, &pixel2, 3) == 0) {
            if (memcmp(&pixel2, &pixel3, 3) == 0) {
                printf("pix1 == pix2 and pix2 == pix3, in=%08X, out=%08X\n", inBuffer, outBuffer);

                // set the MARKER pixel
                set_value(outBuffer++, 0x101010);

                // three bytes match thus far
                count = 3;

                // next pixel to compare
                memcpy(&pixel, inBuffer++, 3);

                // while string of likewise pixels begin counting
                while (memcmp(&pixel3, &pixel, 3) == 0) {
                    count++; memcpy(&pixel, inBuffer++, 3); printf("\n\rloop %i", count);
                }

                // set the repetition count
                set_value(outBuffer++, count);

                // set the repetition pixel color
                memcpy(outBuffer++, &pixel3, 3);

                // load next palette to compare
                memcpy(&pixel1, &pixel, 3);
                memcpy(&pixel2, inBuffer++, 3);
                memcpy(&pixel3, inBuffer++, 3);
            }
            else {
                printf("pix1 == pix2,  in=%08X, out=%08X\n", inBuffer, outBuffer);

                if (memcmp(&pixel1, &pixel_marker, 3) == 0) {
                    printf("pix1 == pix2 and pix1 == marker, in=%08X, out=%08X\n", inBuffer, outBuffer);

                    memcpy(outBuffer++, &pixel1, 3);
                }

                printf("3 bytes to output\n");
                memcpy(outBuffer++, &pixel1, 3);

                memcpy(&pixel1, &pixel2, 3);
                memcpy(&pixel2, &pixel3, 3);
                memcpy(&pixel3, inBuffer++, 3);
            }
        }
        else {
            printf("pix1 != pix2, in=%08X, out=%08X\n", inBuffer, outBuffer);
            if (memcmp(&pixel1, &pixel_marker, 3) == 0)
                memcpy(outBuffer++, &pixel1, 3);

            printf("3 bytes to output\n");
                memcpy(outBuffer++, &pixel1, 3);

            memcpy(&pixel1, &pixel2, 3);
            memcpy(&pixel2, &pixel3, 3);
            memcpy(&pixel3, inBuffer++, 3);
        }
    }

I get this output...

pix1 != pix2, in=0087A03D, out=00C22040
3 bytes to output
pix1 != pix2, in=0087A040, out=00C22043
3 bytes to output
pix1 != pix2, in=0087A043, out=00C22046
3 bytes to output
pix1 != pix2, in=0087A046, out=00C22049
3 bytes to output
pix1 != pix2, in=0087A049, out=00C2204C
3 bytes to output
pix1 == pix2 and pix2 == pix3, in=0087A04C, out=00C2204F

loop 4
loop 5
loop 6
loop 7
loop 8
loop 9
loop 10
loop 11
etc...

So what's actually going on here:

is that all the pixel1/2/3 are all {0,0,0}. In other words it has "run off the end of the input data". In effect it means your stopping condition:

    while ((inBuffer - inStart) < inSize) {

Is wrong. It went too far.

 

I'm too lazy to investigate further but I do wonder about the build warning:

d:\c\testfp\testfp\testfp.cpp(56): warning C4018: '<': signed/unsigned mismatch

which relates to that very line.

Yeah I did that too but sometimes it's simpler just to sprinkle printf()s just to get a "broad brushstrokes" view of what the path of execution is.

Most image formats have some kind of header on the front typically giving width, height and bit depth so you can calculate how much data you'll end up with before you start.

You should really pursue making as much code as possible portable which would allow you to test on your development computer. This gives you the following benefits (and more):

• Segfault and core dump when dereferencing null pointers
• Access to dynamic analyzers (such as valgrind)
• Access to compiler supplied sanitizers (address, memory, thread, and undefined behavior)
• Access to easy to use automated testing frameworks
• Easy integration of build and automated test verification into your workflow (Git hooks and/or CI/CD)

These tools can make your a whole lot easier when running into these kinds of problems. Combining them with test driven development makes things even easier.

If you're dealing with images, it's much more likely that adjacent pixels in a single colour plane are going to match than (say) RGB or YUV pixels across a full image. The larger your bit size, the lower the chance of adjacent bits being amenable to RLE coding. There are many better ways to encode pictures, particularly photographic images.

 

RLE really doesn't work well with images once you get past four or five bits colour depth. Trying it with a 32 bit depth is a serious waste of time; odds are it'll never find two matching pixels adjacent (except maybe in black edges).

 

Neil