Writing pages to flash in real time, too much for Arduino?

Go To Last Post
11 posts / 0 new
Author
Message
#1
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Hello!

 

I have a project with a custom board running a 328PB. I'm logging a thermocouple, barometric pressure, a pulse frequency, 4 adcs, and a 19200baud serial data stream. I'm then broadcasting these sensors to a ECU using a MCP2515. So far I've done everything in arduino using minicore hardware package. I've made my own "libraries" for the max31856, bmp280, using timer/counter for frequency measurement, and sync'ing the serial data stream. It works, actually pretty well!

 

My ECU allows can passthru, so I should be able to update some values on this device without going through a whole flash write. For most of these values eeprom is more than enough. However, I'd like to store 4 2048 byte lookup tables to flash for 10 bit thermistor or MAF lookup tables. The ECU allows a easy way to program and output a table over canbus from a laptop, I'd rather not fight that format. I could just pass the calibration parameters into eeprom and run the calculation every time for thermistors, but non linear sensors like MAFs won't work that way, sending a lookup table would work a lot better.

 

I've poked about in the datasheet and it looks like I should be able to do this, but I can only execute code that resides in a smaller chunk of flash. If I try and access the lower 2/3ds of flash for code to update the flash, CPU crash?

 

I am still using a library for the MCP2515, and I have no way of telling where anything else goes in flash. I think I could just about fit all my code in the NRWW section, I've not been flash limited so I've been rather inefficient in my coding...

 

Is this just asking too much from the arduino environment? I find very little about writing flash pages from a program and even less about arranging flash memory. Does anyone know of any open source projects that might use a similar function I can examine? Once I know where everything is at in flash, or how I can forcibly arrange it, I think I can figure out actual erase/write code from the datasheet.

 

Thank you in advance for any advice!

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Nothing to do with the Arduino environment. When you erase/write a flash sector the cpu stops for around 4ms. Not something you’d do on a live system. The firsr 2/3 of flash is where your code resides, so a crash is not unexpected if you start erasing the code you’re executing. If you want flash based tables, then you’ll need to place them above your code and below the bootloader.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I understand that, I just don't understand how to make sure the code I have in the arduino environment ends up where I want it.

 

I've since found Optiboot bootloader, which seems to be the standard any more. I can carry a 128 byte buffer in ram, and it seems calling progmem on an array while compiling is enough to properly allocate pages of memory properly aligned to write boundaries. It seems so long as I have my page ready in ram and I use optiboot I can call functions stored off in the safe to run bootloader section of the memory which is safe to run while erasing/writing. It looks like this might actually be a function of minicore, I just didn't RTFM enough. There are examples of exactly what I'm looking to do installed with the core, derp!

 

https://github.com/MCUdude/MiniC...

 

My ECU allows a page write size, along with a configurable delay. It seems as long as I can write 128 bytes to a page before I miss the next canbus packet I should be OK. Writing these tables won't be done on a running engine, so loosing interrupts and all that isn't a big deal.

 

From there pulling bytes from a progmem array is easy as it ever was. I think I might actually be able to make this work!

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0


When you have reflashable data the usual placement strategy is to put it at the very END of the application flash section. If it's 2K then find out where BOOTSZ is pointing and use the 2K immediately below that. This allows the app itself to grow as large as it can before it starts to clash with the data tables (at which point you have simply run out of flash and need a bigger micro).

moonie223 wrote:
If I try and access the lower 2/3ds of flash for code to update the flash, CPU crash?
So why does that result surprise you? It would seem to suggest that the lower 2/3rds is where the app is currently residing and you can't just dump on top of it.

 

What you are aiming for is something like:

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 1

moonie223 wrote:

Writing pages to flash in real time

 

moonie223 wrote:

...logging a thermocouple, barometric pressure, a pulse frequency, 4 adcs, and a 19200baud serial data stream.

 

Define what you mean by 'real time' as there is no such thing. Everything in quantised.

 

A thermocouple is hardly real-time. A pressure sensor can only respond up to a maximum rate. Your pulse only occurs at a certain rate.....

#1 This forum helps those that help themselves

#2 All grounds are not created equal

#3 How have you proved that your chip is running at xxMHz?

#4 "If you think you need floating point to solve the problem then you don't understand the problem. If you really do need floating point then you have a problem you do not understand." - Heater's ex-boss

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

clawson wrote:

 

 

When you have reflashable data the usual placement strategy is to put it at the very END of the application flash section. If it's 2K then find out where BOOTSZ is pointing and use the 2K immediately below that. This allows the app itself to grow as large as it can before it starts to clash with the data tables (at which point you have simply run out of flash and need a bigger micro).

moonie223 wrote:

If I try and access the lower 2/3ds of flash for code to update the flash, CPU crash?

 

So why does that result surprise you? It would seem to suggest that the lower 2/3rds is where the app is currently residing and you can't just dump on top of it.

 

What you are aiming for is something like:

 

 

 

 

I guess the fact that you can't run code in page erased flash doesn't really surprise me, I just still have no way of knowing how to optimally place things in flash to take advantage of that. As far as I know, in the arduino environment the bootloader is stored in the core files as a hex already aligned to the 2kb fuseable boot section. If I export a hex of my code, it starts at page zero. Is there an easy way to tell the compiler to work backwards, high addresses to low?

 

Brian Fairchild wrote:

moonie223 wrote:

 

Writing pages to flash in real time

 

moonie223 wrote:

 

...logging a thermocouple, barometric pressure, a pulse frequency, 4 adcs, and a 19200baud serial data stream.

 

Define what you mean by 'real time' as there is no such thing. Everything in quantised.

 

A thermocouple is hardly real-time. A pressure sensor can only respond up to a maximum rate. Your pulse only occurs at a certain rate.....

 

In real time is probably not the best description, I guess I should say I need ~8K bytes of flash to update on occasion. These chunks of flash will not be updated regularly, only occasionally as I update MAF transfer curves or swap to different thermistors. The ~8k of bytes stores 1024 16bit words. Each corresponds with a ADC value and is used as a progmem lookup table with every ADC read. "Live" sensor values are all stored in ram, not to many of them as it's only around 16 bytes or so total.

 

Stuff like the thermocouple, baro sensor and others all run on timed intervals. Well, the thermocouple has about a 125ms read time as configured and I use a data ready pin as a trigger to update. Same on the baro sensor, it's set with a lot of filtering and outputs at ~10hz, using i2c reads to tell when data is ready. Once again, all my code works and is configured by and large from eeprom values on every startup. For the vast majority of configurable values (egt low map value, egt high map value, accelerometer output alignment, etc,) can all be stored in bytes in eeprom and read once on boot. I could and would do the same with these lookup tables, except they won't fit! I guess it's important to note that any "flash" to this board has to be followed by a reboot, doesn't matter if it's eeprom or flash memory. This is OK, still a lot easier than a full reflash every time I need to recalibrate sensors.

 

Either way I've figured out how to do what I want, and it looks a little like this.
 

// The temporary data (data that's read or is about to get written) is stored here
uint8_t ramBuffer[SPM_PAGESIZE];

// This array allocates the space you'll be able to write to
const uint8_t flashSpace[SPM_PAGESIZE * NUMBER_OF_PAGES] __attribute__ (( aligned(SPM_PAGESIZE) )) PROGMEM = {

The other relevant part is making sure to use a proper optiboot bootloader and making sure I have a full 128 byte page residing in ram before I attempt to write to this aligned array as my CAN code is probably in a bad spot and won't run as soon as I erase a page. Since the code which writes flash is part of the bootloader it's in a place where the SPM command can actually run, It's pre-compiled hex already stored where I need it. Another advantage is I've got a brand new data stream to work with every time I write, I don't ever need to consider or save flash contents before erasing. I'll always be sequentially writing so I don't need to worry about the single address update per write limit.

 

Thank you all for the help! I think I can make this work from here!

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

moonie223 wrote:
As far as I know, in the arduino environment the bootloader is stored in the core files as a hex already aligned to the 2kb fuseable boot section.
The situation is exactly as I showed in the diagram. 

 

Assuming a 328 with Optiboot then the 328 has flash from 0x0000 to 0x7FFF. Optiboot is usually built to locate at 0x7E00 and that is where BOOTSZ/BOOTRST point. So the application can occupy 0x0000..0x7DFF. If you have a 2K data table you would SPM it to 0x7600 so it occupies 0x7600..0x7DFF. The rest of the flash is available to the application so it could grow up to 30,308 (0x7600) bytes in size before it would start to "bump into" the flash data.

 

If you actually need 8K then you would use from 0x5E00 to 0x7E00 for the flash data and the app could not grow bigger than 24,064 bytes.

 

Do you always deliver the new data at run time? Or do you need an initial copy of the 8K built in to start. In "normal" C/C++ the way you'd do that is:

__attribute__((section(".flash"))) const __flash unt8_t flahsData[] = {
    1,2,3,4,5...
}

and then later you would pass -Wl,-section-start=.flash=0x5E00 to the linker - but I'm not sure how you achieve something like that ("special" link options) when building with Arduino?

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

clawson wrote:

moonie223 wrote:

As far as I know, in the arduino environment the bootloader is stored in the core files as a hex already aligned to the 2kb fuseable boot section.

 

The situation is exactly as I showed in the diagram. 

 

Assuming a 328 with Optiboot then the 328 has flash from 0x0000 to 0x7FFF. Optiboot is usually built to locate at 0x7E00 and that is where BOOTSZ/BOOTRST point. So the application can occupy 0x0000..0x7DFF. If you have a 2K data table you would SPM it to 0x7600 so it occupies 0x7600..0x7DFF. The rest of the flash is available to the application so it could grow up to 30,308 (0x7600) bytes in size before it would start to "bump into" the flash data.

 

If you actually need 8K then you would use from 0x5E00 to 0x7E00 for the flash data and the app could not grow bigger than 24,064 bytes.

 

Do you always deliver the new data at run time? Or do you need an initial copy of the 8K built in to start. In "normal" C/C++ the way you'd do that is:

__attribute__((section(".flash"))) const __flash unt8_t flahsData[] = {
    1,2,3,4,5...
}

and then later you would pass -Wl,-section-start=.flash=0x5E00 to the linker - but I'm not sure how you achieve something like that ("special" link options) when building with Arduino?

 

I think I understand now, I can run code pretty much anywhere and use it to load the flash page buffer. In my application I am completely filling the table with new values I need not worry about the RWW and NRWW sections, I only need to make sure that the code that writes to flash starts from the BLS, since that's the only way SPM instruction will work. Since a page is small and easily fits in ram I can just buffer pages using code running anywhere. It seems to me almost anyone need not worry about this catch, just buffer all the data in ram and modify as necessary before calling a SPM, then write all new data sequentially. I honestly have no idea what I'm doing though, I mean optiboot does exactly what I want and was right in front of my face when I made this thread!

 

I can then compile code to hex and verify where the last used page of actual program is and then use all pages between it and optiboot. I can make sure I am page aligned and even use the addresses for my lookup tables in the code. It would probably be wise to make sure SPM can't access the BLS, but since SPM will always be able to overwrite my program I'll have to be super duper careful with my write calls. If I miss, I'm pulling the hardware and ISP programming to fix it. Because of this I'll probably use the PROGMEM array/compiler to handle addresses for me.  My ECU will stream 8 byte can packets in 128 byte intervals, pausing for a configurable period inbetween. It also sends table offsets for the corresponding data with every 8 bytes, so I shouldn't get anything out of order. I should be able to divide the requested write address (if greater than 128) to tell which page I am on. Then the difference between page*128 and the requested write offset should give the current ramBuffer offset. From there writing a page will look something like this, except I will fill the ramBuffer with CAN data.

 

    // Get all characters from the serial monitor and store it to the ramBuffer
    memset(ramBuffer, 0, sizeof(ramBuffer));
    uint16_t counter = 0;
    while (counter < SPM_PAGESIZE && charBuffer != terminationChar) 
    {
      if(Serial.available() > 0) 
      {
        charBuffer = Serial.read(); // read character from serial
        if(charBuffer != terminationChar)
        {
          Serial.write(charBuffer); // echo character back
          ramBuffer[counter] = charBuffer;
          counter++;
        }      
      }
    }
    charBuffer = 0;
    Serial.println(F("\n\nAll chars received \nWriting to flash..."));

    // WRITE RECEIVED DATA TO THE CURRENT FLASH PAGE
    // flash_buffer is where the data is stored (contains the memory addresses)
    // ramBuffer contains the data that's going to be stored in the flash
    // pageNumber is the page the data is written to
    optiboot_writePage(flashSpace, ramBuffer, pageNumber);

 

I do not necessarily need a initial copy in flash, part of the configurable code stored in eeprom controls weather to use raw ADC values or use the lookup table. The default will be raw, but I might program a rather useless default raw lookup table for fun. I would use something a little like your example there, I think Arduino makes it look a little more like this. But, this supposedly aligns pages and handles addressing for me.

 

const uint8_t flashSpace[SPM_PAGESIZE * NUMBER_OF_PAGES] __attribute__ (( aligned(SPM_PAGESIZE) )) PROGMEM = {
0x02,0xBC,
0x02,0xBC,
0x02,0xBC,
0x02,0xBC,
0x02,0xBC,
0x02,0xBC,
0x02,0xBC,...
}

 

To read data from the flash in a lookup table fashion I just use a standard char array lookup. Here's one that prints all allocated space in hex. Instead I will multiply the ADC value by two, read that (and the next) array address and construct a signed 16bit value which is later broadcast over can.

 

      for(uint16_t page = 1; page < (NUMBER_OF_PAGES*SPM_PAGESIZE)+1; page++)
      {
        Serial.println(flashSpace[page], HEX);
      }

 

I won't be uploading a new table at run time, more often than not the thing will power on and read eeprom for the config file and then run while only outputting data over can, no updates to persistent memory. Every once in a while I may move a thermocouple, pressure sensor, or try and rescale my MAF table. I've also thought about a way of logging suspension position, and a linear potentiometer logging off a control arm seems easiest, I'd need a non-linear table to make sense of that but wouldn't change it once it's flashed, at least till I removed it. 

 

Of course I could just log the raw ADC values and use my logging software to correct the values later, what I do now, but where's the fun in that!

 

I also made a display gauge that has all it's settings configured through a rotary encoder menu. And I suck at coding, it's a mess but it works. Changing it after the fact sucks! Instead of using an encoder menu I am thinking I'll use this can passthru feature. If I can make it work with this stuff on a simpler 8 bit I'll stand a much better chance on the SAMD51 I am using now. ...

 

https://www.youtube.com/watch?v=...

 

 

Anyway, TL;DR to answer the thread title. Optiboot lets you write to flash and even provides an example. You do not need to know where your code resides in memory, a PROGMEM array will handle all addressing for you. You just need to have optiboot flashed to the default location even if you aren't planning on using uart programming. If you can (I think you must) manage a page sized ram buffer (128 bytes, 64 words on 328s) then just copy the current data and modify ram copy before using the example to write the updates to flash.

 

Thank you all for the advise!

 

 

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

you might want to look at interpolation tables. This is a common technique used on car ecus. having a 1024 word table for lookup is just overkill.

 

find a disassembly of the Delco 808 and borrow the techniques from there. The Delco is basically the 'gold standard' on how to do an ecu.

 

we seem to have run of posters confusing 'advise' and 'advice' as well as 'insure' and 'ensure'/ Picky maybe, but we deal with compilers that are much pickier!

Last Edited: Sat. Feb 15, 2020 - 03:03 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

moonie223 wrote:
I need not worry about the RWW and NRWW sections
why do you say that? Every time an SPM operation starts it's not just the flash page you are writing that becomes inaccessible but the entire lower 28K of the 32K chip. So your entire program is held in stasis while flash writing occurs. Nothing like interrupts can occur or anything.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

For exactly that reason, "Writing pages to flash in real time, too much for Arduino?" is the wrong question. The real question is "how long does a write to flash take, and what happens while the write is going on".

 

IMHO, you would be FAR better off using an external nonvolatile memory, such as FRAM. They are FAST and nonvolatile. They are easy to access and occupies the CPU only as long as an SPI transfer takes. For the size you are talking about, there are a modest set of choices. Not cheeeeep certainly, but low power and easy to use. Even a small serial flash chip would be better than what you are proposing.

 

Jim

Jim Wagner Oregon Research Electronics, Consulting Div. Tangent, OR, USA http://www.orelectronics.net

Last Edited: Sat. Feb 15, 2020 - 05:50 PM