Porting to DataFlash from SD. Pros and Cons.

Go To Last Post
55 posts / 0 new

Pages

Author
Message
#1
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Hi,

I currently have a project, in which I'm using a 2 GB SD card with elm-chan's FAT FS.  But as Flash ICs like AT45DB and SST26VF are cheaper I thought I would change my hardware design. But I have a few questions.

1. Will it be worthwhile to port to Data Flash from an SD card? In terms of cost and development time.

2. If porting to Data Flash, how should I implement the erase block logic with the FileSystem. For example, the SST26VF has an erase block size of 4KB (for SD card I didn't have to bother about this), so should I always work with 4KB when I'm writing and reading too? If I have smaller data to read and write operations, won't an erase operation erase the whole data?  

3. Wear leveling and bad block management. I know I will have to implement them and this will cost a lot of development time. Do you know of any open-source implementation of these for data flash? Or any FileSystem that takes care of this.

 

In short: How to implement erase logic, is porting to flash worth it (speed, cost) as I already have a working SD logic, will it need a huge development time.

 

Thank you.

 (I'm a student and can't afford any industry grade driver. So a free open source implementation would be helpful)

This topic has a solution.

Clament John | Still in school 

Last Edited: Mon. May 29, 2017 - 09:16 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Have you considered eMMC - which is basically an SD-Card on a chip?

Top Tips:

  1. How to properly post source code - see: https://www.avrfreaks.net/comment... - also how to properly include images/pictures
  2. "Garbage" characters on a serial terminal are (almost?) invariably due to wrong baud rate - see: https://learn.sparkfun.com/tutorials/serial-communication
  3. Wrong baud rate is usually due to not running at the speed you thought; check by blinking a LED to see if you get the speed you expected
  4. Difference between a crystal, and a crystal oscillatorhttps://www.avrfreaks.net/comment...
  5. When your question is resolved, mark the solution: https://www.avrfreaks.net/comment...
  6. Beginner's "Getting Started" tips: https://www.avrfreaks.net/comment...
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

No, I haven't. I have a few AT45DB and SST26VF064b lying around. So I thought I would use them. But eMMC looks interesting. Thank you, but I was hoping to use a flash IC.

Clament John | Still in school 

This reply has been marked as the solution. 
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I'd be thinking about overall system cost. Sure the AT45 may be cheaper but how long is it going to take you to develop a wear leveling, bad block remapping filing system? A SD/MMC card has a micro within it that is handling all this kind of stuff. So if some sectors of the NAND array either start out with bad blocks or develop bit faults over time there is a system in the card "intelligence" to detect this and to remap sectors from a spare "pool". So you may make a write to sector 12345 but if the controller writes the data and finds that some bits dropped it will go to its remap table. Find another available sector to use and remap that for any future requests to 12345. It will also be using ECC codes to detect and correct such bad blocks as they develop over time. It may also maintain write counts on sectors and if it finds any that are being heavily written it may choose to remap those too.

 

AT45 on the other hand is a pretty "raw" NAND array. In fact the reason that the 256 byte sector devices have (optionally) 264 bytes and the 512 sectors have 528 bytes (that is +8 or +16) is so you can implement your own ECC on the stored data to monitor bit faults over time. You may then need to start implementing a remap strategy and so on.

 

Most FAT systems are 512bytes/sector based so the 256/264 and 512/528 byte pages in AT45 should easily lend themselves to implementing some kind of 512 byte sector accessing scheme.

 

Linux does have various FFS available that may be used to add robustness to raw NAND arrays so you may be able to leverage something from there with the usual proviso about GPL tainting your code.

To be honest for the difference between $2 for an AT45 and $4 for a small SD/MMC I know what I would be choosing in volumes any lower than about 10,000 units! It's just not worth the hassle unless you are cost engineering something for mass market volumes.

 

Anyway how mush is 2GB of AT45 going to cost anyway? You can't really make an apples-apples comparison between AT45 and SD/MMC anyway. AT45 capacities are down in the 100KB's and SD up in the 1,000KB's. So $2 might get you a 128KB AT45 but to fit out 2GB is going to cost a whole load more than the alternative SD/MMC.

How much and what kind of data is it anyway? Audio? Video? Something else?

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Clement.J wrote:
will it need a huge development time.

To summarise what clawson said,

 

YES !

 

Top Tips:

  1. How to properly post source code - see: https://www.avrfreaks.net/comment... - also how to properly include images/pictures
  2. "Garbage" characters on a serial terminal are (almost?) invariably due to wrong baud rate - see: https://learn.sparkfun.com/tutorials/serial-communication
  3. Wrong baud rate is usually due to not running at the speed you thought; check by blinking a LED to see if you get the speed you expected
  4. Difference between a crystal, and a crystal oscillatorhttps://www.avrfreaks.net/comment...
  5. When your question is resolved, mark the solution: https://www.avrfreaks.net/comment...
  6. Beginner's "Getting Started" tips: https://www.avrfreaks.net/comment...
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I'm not using the 2GB memory available in the SD card.

The device is an OBD-II scanner with GPS tracking and connectivity to a server using SIM800 and I'm using the SD card to store data that might have failed to post to the server so I can repost it later on.

So the data won't go above a few MBs. And I feel the 2GB is over kill.

Clament John | Still in school 

Last Edited: Wed. Apr 19, 2017 - 06:15 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Haha yes. It looks so. Thank you *awneil*. I should start researching on an eMMC then.

Clament John | Still in school 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Surely, if you've got a GPS module in there, and a GSM module in there, the cost saving on the SD card is immaterial ... ?

Top Tips:

  1. How to properly post source code - see: https://www.avrfreaks.net/comment... - also how to properly include images/pictures
  2. "Garbage" characters on a serial terminal are (almost?) invariably due to wrong baud rate - see: https://learn.sparkfun.com/tutorials/serial-communication
  3. Wrong baud rate is usually due to not running at the speed you thought; check by blinking a LED to see if you get the speed you expected
  4. Difference between a crystal, and a crystal oscillatorhttps://www.avrfreaks.net/comment...
  5. When your question is resolved, mark the solution: https://www.avrfreaks.net/comment...
  6. Beginner's "Getting Started" tips: https://www.avrfreaks.net/comment...
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Clement.J wrote:
I should start researching on an eMMC then.

May well not be cheaper, but certainly more rugged - which must be more important in an automotive environment ... ?

Top Tips:

  1. How to properly post source code - see: https://www.avrfreaks.net/comment... - also how to properly include images/pictures
  2. "Garbage" characters on a serial terminal are (almost?) invariably due to wrong baud rate - see: https://learn.sparkfun.com/tutorials/serial-communication
  3. Wrong baud rate is usually due to not running at the speed you thought; check by blinking a LED to see if you get the speed you expected
  4. Difference between a crystal, and a crystal oscillatorhttps://www.avrfreaks.net/comment...
  5. When your question is resolved, mark the solution: https://www.avrfreaks.net/comment...
  6. Beginner's "Getting Started" tips: https://www.avrfreaks.net/comment...
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Now when you can't move the card over in a PC etc, perhaps you don't need a real file system, (but just an array(s) of sectors or something like that).

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

google gave this (I don't know anything about it).

 

https://piconomix.com/fwlib/grou...

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

sparrow2 wrote:

 (but just an array(s) of sectors or something like that).

Even in this case won't the flash have bad blocks? So will a simple error management overcome bad blocks?

Then again wear leveling,  this is a major factor for the life span of a block. 

Do you know of any open-source (simple, Linux repo is overwhelming) solution I could use?

Clament John | Still in school 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Have you considered eMMC - which is basically an SD-Card on a chip?

I thought eMMC had no support for SPI interface.  And FFS has support only for SPI.  Is this an AVR project?

 

They're also awfully expensive.  Digikey has them for $20 qty1 for 8GB, but nothing in stock.  12 week lead time.

 

16GB Qty1000 for $5.31, 10 weeks.

"Experience is what enables you to recognise a mistake the second time you make it."

"Good judgement comes from experience.  Experience comes from bad judgement."

"Wisdom is always wont to arrive late, and to be a little approximate on first possession."

"When you hear hoofbeats, think horses, not unicorns."

"Fast.  Cheap.  Good.  Pick two."

"We see a lot of arses on handlebars around here." - [J Ekdahl]

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

sparrow2 wrote:

google gave this (I don't know anything about it).

I did see that too. Currently reading it. Thanks for the effort.

Clament John | Still in school 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

joeymorin wrote:

I thought eMMC had no support for SPI interface.  And FFS has support only for SPI.  Is this an AVR project?

 

They're also awfully expensive.  Digikey has them for $20 qty1 for 8GB, but nothing in stock.  12 week lead time.

 

16GB Qty1000 for $5.31, 10 weeks.

 

I'm using an ARM IC. 

Wow, that is pretty expensive. And I was hoping to use SPI. I won't need to change a lot in the hardware design if its SPI, just the SD pads need to be changed to the flash pads.

So, I guess I have to do better research for flash. 

Piconomix looks promising.

Clament John | Still in school 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

How many of these are you making that a couple bucks difference for storage makes a difference?

 

Oh, yeah -- if just making a few, a batch of NOS or pulls of low-density MMC/SD can be had on eBay, sometimes for next to nothing.

 

Clement.J wrote:
So the data won't go above a few KBs.

A few KB >>total<<?

 

-- Many AVR8 models have "a few KB" of EEPROM. [tell what processor you are working with]

-- There are many choices for "a few KB", from serial or parallel EEPROMs to FRAM to ...

You can put lipstick on a pig, but it is still a pig.

I've never met a pig I didn't like, as long as you have some salt and pepper.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

SD/MMC have so many advantages:

 

1) readily available - even pretty much every supermarket on planet Earth has them alongside the baked beans!

2) swappable - so if you want to "archive " data you just remove one car, put it in a draw then fit the next

3) data interchange with PC. If only for debug you can simply remove the card from the MCU, plug it into a PC and analyse the same data because they all "know" the same (FAT) filing system

4) simple - SPI is always going to be simple and there are SPI-MMC driver code for pretty much every micro on earth

5) complex - as noted above the controller on the card does all the hard graft of stuff like bad block avoidance and wear levelling so you don't have to worry

6) guaranteed supply - given that almost every phone, tablet and camera on earth uses SD/MMC then I don't see them going away any time in the next few decades

7) multiple suppliers - even if, God forbid, Sandisk went out of business there's still going be the other 99 companies making them

 

As I say the one possible "advantage" of AT45 (for small amounts of data) might be cost - that would be an issue if you plan to make 10,000+ of whatever this is - otherwise it's not worth the hassle

 

(all the above only In My Arrogant Opinion of course!)

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

theusch wrote:

How many of these are you making that a couple bucks difference for storage makes a difference?

3000

theusch wrote:
A few KB >>total<<?

We need 4-5MB to be on the safe side. Each GSM packet is 200 bytes. So if we miss (error in pushing to the server) quite a few, say the user is going through a long tunnel the data can be in MBs. So can't use the inbuilt flash storage.

Clament John | Still in school 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

clawson wrote:

As I say the one possible "advantage" of AT45 (for small amounts of data) might be cost - that would be an issue if you plan to make 10,000+ of whatever this is - otherwise, it's not worth the hassle

[/quote

Currently, we won't be manufacturing 10,000 units. But won't it be better to do this now (if we are) than later?

But yeah, using an SD card makes more sense.

Quick question - will an automotive environment (high noise and temperature) cause problems in the SD card? Can the SD card be corrupted due to any of these? We have faced physical errors in the file system (the ones where the file system can't do anything - like FR_DISK_ERR. Any thoughts on that?

Clament John | Still in school 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Clement.J wrote:
will an automotive environment (high noise and temperature) cause problems in the SD card?

Probably the biggest problem for an SD-Card - because of the socket - is vibration ... ?

 

 

Top Tips:

  1. How to properly post source code - see: https://www.avrfreaks.net/comment... - also how to properly include images/pictures
  2. "Garbage" characters on a serial terminal are (almost?) invariably due to wrong baud rate - see: https://learn.sparkfun.com/tutorials/serial-communication
  3. Wrong baud rate is usually due to not running at the speed you thought; check by blinking a LED to see if you get the speed you expected
  4. Difference between a crystal, and a crystal oscillatorhttps://www.avrfreaks.net/comment...
  5. When your question is resolved, mark the solution: https://www.avrfreaks.net/comment...
  6. Beginner's "Getting Started" tips: https://www.avrfreaks.net/comment...
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

FAT (and indeed most filing systems) are NOT robust and are easily damaged. For years FAT has generally maintained  TWO file allocation tables and yet nothing uses these for redundancy! If you want something more robust you might want to explore something like a journalling file system but it's something you are unlikely to find implemented for a low resource micro like AVR8. 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

The AT45 series are NOR flash devices, so what is the worry?

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Clement.J wrote:
The device is an OBD-II scanner with GPS tracking and connectivity to a server using SIM800 and I'm using the SD card to store data that might have failed to post to the server so I can repost it later on.

So the data won't go above a few KBs.

Clement.J wrote:
We need 4-5MB to be on the safe side.
MRAM might work but would need the ARM's EBI to access megabytes; the SPI and QSPI MRAM are limited to 0.5MB max.

MRAM has internal ECC and an automotive rating.

https://www.everspin.com/

 

"Dare to be naïve." - Buckminster Fuller

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Or FRAM ?

 

Top Tips:

  1. How to properly post source code - see: https://www.avrfreaks.net/comment... - also how to properly include images/pictures
  2. "Garbage" characters on a serial terminal are (almost?) invariably due to wrong baud rate - see: https://learn.sparkfun.com/tutorials/serial-communication
  3. Wrong baud rate is usually due to not running at the speed you thought; check by blinking a LED to see if you get the speed you expected
  4. Difference between a crystal, and a crystal oscillatorhttps://www.avrfreaks.net/comment...
  5. When your question is resolved, mark the solution: https://www.avrfreaks.net/comment...
  6. Beginner's "Getting Started" tips: https://www.avrfreaks.net/comment...
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Kartman wrote:
The AT45 series are NOR flash devices, so what is the worry?
I had to look that up but:

 

http://www.adestotech.com/produc...

 

confirms that to be the casesurprise.

 

In which case I'll take back everything I said about the need for bad block avoidance. I do kind of wonder though, what the +8 or +16 bytes per page are there for if it's not for implementing an ECC? And why then would you need that in a NOR flash?

 

There is still the question of the need for wear levelling though. Adesto have this:

 

http://www.adestotech.com/wp-con...

 

(interesting that they haven't bothered to re-brand that even after all these years!)

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I dare say these parts were bleeding edge in their day, so ecc was maybe a good idea. The serial flash devices I've worked with don't have ecc. 16MB seems to be the sweet spot at the moment but you can get larger parts - some even are multi-chip in the one package. The usual sector size seems to be 4k.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I thought NOR and NAND just differed in speed. Why won't a NOR flash require a bad block management?

Clament John | Still in school 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

gchapman wrote:
MRAM might work but would need the ARM's EBI to access megabytes; the SPI and QSPI MRAM are limited to 0.5MB max.

awneil wrote:
 Or FRAM ? 

Will look into these.

 

Going through piconomix. But how does he handle bad blocks here? He uses an AT45DB but what if I'm using an SST26VF which is a NAND flash.

Also, read through the adestotech's application note on flash translation layer. This would let me use the current FAT FS. But they haven't given any links to their source code. :/

Found this - haven't gone through the full thing Link

Clament John | Still in school 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Clement.J wrote:
I thought NOR and NAND just differed in speed. Why won't a NOR flash require a bad block management?
Nope. There is a clue in the name!...

 

https://en.wikipedia.org/wiki/Fl...

 

(in fact don't just read that section - read the entire page)

But how does he handle bad blocks here?

As noted above - AT45 is NOR so it won't have bad blocks. But it is still subject to wear. I cannot see the author doing anything to mitigate that in the code shown apart from his point that the memory array is used sequentially/circularly so that each part should get equal wear.

Last Edited: Wed. Apr 19, 2017 - 10:56 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

SST26VF is NOR flash. NAND flash is usually significantly higher in density. NOR write endurance is greater than NAND, so one tends to ignore wear levelling and bad block management. If you are doing lots of writes, then the device will eventually wear out.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Hi again,

I went forward with SST26VF as I already wrote a driver for that, and my AT45DB isn't working. And I linked the init, read, write functions to user_diskio.c. But I'm facing a few hurdles.

In f_mount Chan calls find_volume(&fs, &path, 0), inside this he calls check_fs(**) and inside this (sorry for the rabbit hole) he calls move_window (line2254). It is here the fs struc is updated. (It is updated in disk_read(**), line815). I have attached the ff.c file where all this happens.

And I have the driver->read_flash to read from the beginning address of SST26VF (0x010000 * sector * ERASE_BLOCK_SIZE). But I don't have any data in that memory space that FAT-FS is looking for, its just 0xFF as I just erased the flash. When I looked into an SD card, at this point in the program the buffer is filled with data like "boot record signature" (at position 510, check like 2257), "File System Type", etc. So the SD card had data in the memory block FAT-FS reads. The buffer has decimal '85' at position 510, always.

My question is:

Should I pre-load that sector with the required data so that my find_volume is successful? This "read" data is used in multiple places in find_volume.

Or is there a standard memory I should be reading. Basically, I'm clueless. I get an ERROR - "No FAT volume is found". So what should a memory device have so that it is a "FAT Volume"?

ERASE_BLOCK_SIZE = 4KB
I know even if this works I would need a wear leveling system between my driver and FAT-FS. But this is really fun just going deeper and deeper.

Attachment(s): 

Clament John | Still in school 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

You need to format the volume.
http://elm-chan.org/fsw/ff/en/mkfs.html

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I guess that there are plenty of AT45 code out there. The Org stk500 boards had one (at the beginning), and the butterfly had one (and I know that there are some demo code from Atmel)

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Kartman wrote:
You need to format the volume.
+10

 

Yet another advantage of SD/MMC. When you pop down to Tesco/Asda/Walmart/Costco and buy yourself an SD/MMC the nice folks at Sandisk have thoughtfully already put a FAT (usually FAT32) layout on the sectors so you just plug it into your PC or AVR or camera or tablet and off you go.

 

If you wire up some "blank" memory array it will not have any such format. I guess you could "fake" it by simply providing a faked MBR and boot/BPB when the filing system requests it but other than that you need some kind of mkfs operation to create a FAT layout in the memory sectors.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I got the SST26 working. But the FAT FS uses too much RAM ( can be seen when compiled/ build ) and so I'm unable to run my RTOS. Why is that? I've opened 4 files.

Is this because I have a 4KB sector size for my FS? Will the RAM used reduce if I shift to AT45DB where a sector is 512 bytes? 

Mr Chan's application note (Link) says that RAM usage depends on files and volumes opened. Doesn't mention about sector size. 

Clament John | Still in school 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Each file structure has a one sector buffer. On a micro why on earth would you design things to need 4 files active at once. On a PC or similar pre-process the data to interleave all 4 data streams into a single file. 

 

Also consider implementing a shim layer to present the storage as if it were really the normal 512 byte sectors (though this may need a 4K buffer anyway). 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

@clawson. I was going through your dataflash.c Link. In this, you're passing the address as LSB first? In line 29/32 you are transmitting zeros first ( I assume they are the start address for the page), then the page then zeros again. 

In the datasheet, it is specified a Continuous Array Read (0xE8H) with 264 bits will need the opcode code clocked in along with a 24-bit address sequence. Of with the "first 11 are the page number and the last 9 bits are starting byte address on the page", this is quite confusing. Where do the other 4 bits go? In the beginning or in the end? Is it little endian or big endian? Where should I stuff the extra 4 bits?

I'm making my own driver so that I can learn how it's done. Your module is great by the way.

Clament John | Still in school 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Well it's code from 10+ years ago and I'm not that blown away by it either but I do know that it worked. So when writing the flash it starts with a call to:

void
dflash_start_write_active(
	void
){
	// this send the command that prepares the DF to receive the next 264/528 bytes to the active buffer
	// assumes chip select and SPI_enable already set
	
	if (gl.active == 0) {
		write_SPI(0x84); // make buffer 1 active
	}
	else {
		write_SPI(0x87); // make buffer 2 active
	}
	write_SPI(0); // always start writing at beginning of the 264 / 528 bytes
	write_SPI(0);
	write_SPI(0);
	gl.buf_ptr = 0; //this will be used to count the bytes output to the buffer
}

Let's assume that the global "active" is 0 to start with so this will issue:

 

0x84

0x00

0x00

0x00

 

This is because of:

 

 

In this command the 24 bits of "address" are mostly "X" (don't care) apart from the last 9 bits which could pick an offset to starting writing in the RAM buffer. But when buffering I always want to write complete buffer fulls. So for the entire 24 bits of address I send:

 

0x00

0x00

0x00

 

so whether it is 0x84 (buffer 1) or 0x87 (buffer 2) I will start writing from BBBBBBBBB = 000000000 - the start of the buffer.

 

Subsequently for each bye written the code:

void
dflash_write_byte(
	unsigned char byte
){
	write_SPI(byte); // byte value out to active buffer
	gl.buf_ptr++;

#ifdef FLASH_528
	if (gl.buf_ptr == 528) { // the current buffer is full - force it to be written
#else
	if (gl.buf_ptr == 264) { // the current buffer is full - force it to be written
#endif
	    dflash_flush_buffer();

	    dflash_chip_select(); //and then get ready for the next command
	    if (gl.active == 0) { // toggle the active buffer for the receipt of the next 264/528 bytes
	        gl.active = 1;
	    }
	    else {
	        gl.active = 0;
	    }
	    dflash_start_write_active(); // make the "other" buffer the active receive buffer and reset the 264 byte counter
	}
}

is called. Normally that does nothing but write the data byte to the next (auto-increment) position in the RAM buffer but when the count reaches 264/528 it will execute:

	    dflash_flush_buffer();

Up to now the writes have been into the active (0x84 or 0x87) RAM buffer but now it's time to write that buffer full to the flash array using:

void
dflash_flush_buffer(
	void
){
	unsigned long addr = gl.df_page;

#ifdef FLASH_528
	addr = addr << 10; // turn page address into ppp+bbb address with bbb=0
#else
	addr = addr << 9; // turn page address into ppp+bbb address with bbb=0
#endif

	dflash_chip_unselect(); // stop current buffer write sequence
	dflash_chip_select(); //start new command write
	if (gl.active == 0) {
	    write_SPI(0x83); // write buffer 1 to main memory with erase
	}
	else {
	    write_SPI(0x86); // write buffer 2 to main memory with erase
	}
#ifdef FLASH_528
	write_SPI((addr >> 16) & 0x3F);
#else
	write_SPI((addr >> 16) & 0x07);
#endif
	write_SPI((addr >> 8) & 0xFF); // only the page bits here relevant - byte offset bits ignored (but sent to pad)
	write_SPI( addr & 0xFF);
	
	gl.df_page++; //step on for the next buffer write

	dflash_chip_unselect(); // kick off the commit of buffer to main memory array
}

Now you may not know this but I maintain a global called "df_page" which is simply the page number to be used next (before I ever start data recording I set df_page to be the starting page number for data storage). So let us assume that is set to page 137 on entry here so gl.df_page == 137

 

So the first process is:

addr = gl.df_page;

so now addr = 137. Then let's also assume a 264 byte/page device so:

#else
	addr = addr << 9; // turn page address into ppp+bbb address with bbb=0
#endif

If you read one of the datasheets for the 264 byte device you find:

 

 

So for both the 0x83 (buffer 1 to main array) and 0x86 (buffer 2 to main array) the addressing this time passes the page number to be written offset to the left by 9 bit places. Hence the reason for <<9 in the code. So if the page number is 137 (0x89 = 0b10001001) and the command being used is 0x83 it will be:

 

                                       0   0  0   0  0  0  0   1  0  0  0   1  0  0   1  0  0  0  0   0  0  0   0  0 

 

So after the 0x83 (in this case) has been written I do this:

#ifdef FLASH_528
	write_SPI((addr >> 16) & 0x3F);
#else
	write_SPI((addr >> 16) & 0x07);
#endif
	write_SPI((addr >> 8) & 0xFF); // only the page bits here relevant - byte offset bits ignored (but sent to pad)
	write_SPI( addr & 0xFF);

and because we're talking about 264 byte devices that reduces to:

	write_SPI((addr >> 16) & 0x07);
	write_SPI((addr >> 8) & 0xFF); // only the page bits here relevant - byte offset bits ignored (but sent to pad)
	write_SPI( addr & 0xFF);

so this is writing the three bytes of the 24 bit address (held in "addr" which is actually a uin32_t in fact). To be honest the shifting and masking here is all pretty pointless! I already constructed addr by simply taking the page number and doing a <<9 to move the bits to the right place so the bits "above" and "below" (the XXXX's in the datasheet) are all likely 0 anyway. So it's a bit futile to mask bits here but I suppose I might have picked a page number that was "too big" for the device so I maybe don't want to send the upper bits if it spills out the top - but the "X" mean "don't care" so the device would have ignored such bits anyway. Also, because of the <<9 I know there can't be anything but 0 in the lower 9 bits so while:

	write_SPI((addr >> 8) & 0xFF); // only the page bits here relevant - byte offset bits ignored (but sent to pad)

should really have been 0xF8 (a bug after all these years!!) it doesn't matter. And there's an argument to say that:

	write_SPI( addr & 0xFF);

should simply have been:

	write_SPI( 0 );

because, for definite, the lowest 8 bits will always be 0 in an 0x83 or 0x86 command. But it kind of does the job.

 

When it comes to reading back, as you say I use "continuous array read" - the 0xE8 command. Curiously the datasheet does not have a pretty X/P/A style diagram for that  but it does say:

 

Now again I only ever use the device at page boundaries - I don't do anything fancy like trying to read/write starting part way through a page so in my case the last 9 bits will always be 0 in an E8 command. So I do:

	dflash_chip_select();

	write_SPI(0xE8); //continuous array read command
#ifdef FLASH_528
	write_SPI((page >> 6) & 0x3F); //send page number then byte offset = 0
	write_SPI((page &0x3F) << 2);
#else
	write_SPI((page >> 7) & 0x07); //send page number then byte offset = 0
	write_SPI((page &0x7F) << 1);
#endif
	write_SPI(0);

as we're talking 264 that reduces to:

	dflash_chip_select();

	write_SPI(0xE8); //continuous array read command
	write_SPI((page >> 7) & 0x07); //send page number then byte offset = 0
	write_SPI((page &0x7F) << 1);
	write_SPI(0);

So this time the bottom 8 bits of the address are definitely sent as 0. The real address starts one bit into the next byte up. So

	write_SPI((page &0x7F) << 1);

gets the lowest 7 bits of the address and moves them to the bit 9 position. The top of the address is done with:

	write_SPI((page >> 7) & 0x07); //send page number then byte offset = 0

which is stripping off the bottom 7 bits that are already used and then the mask & 0x07 is presumably because I was using a very small AT45 (2Mbit perhaps?) in which the top bits of the address (beyond bit 7) could never be beyond bit 10?

 

As I say this code is old and showing it. I'm sure someone will have written a very slick driver for Arduino and in this day and age I'd just lift and use that rather than trying to battle to do this all over from scratch!

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Got it working with Chan's FAT FS. Thank you all.

Now have to start working on wear leveling. Sorry, but does anyone know of a middleware wear leveling block I can use right out of the box? I found Spiff, but is a filesystem which has wear leveling included. There seem to be research papers on this topic and looks fun, but after writing two drivers I just want to take a break, and grab a coffee. :)

Clament John | Still in school 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Each sector has 100,000 writes. In your system how long would it take to reach this? Then determine if you want wear levelling.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I have a persistent file I update every 5 seconds. It contains the current message ID (this is used so that the backend can differentiate between old packets and new ones, used for GPS plotting). So in around 5.5 hours the same block will be written 100,000 times. 
So I think I need wear leveling

Clament John | Still in school 

Last Edited: Wed. Apr 26, 2017 - 06:34 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

If you have well defined time in the file(data) why would you need a filesystem?

Just write data in a known order, and when full overwrite (or stop).

scanning thru the flash to find the oldest data, (only happens a reboot).  

 

read out (plot) then read all or the last nnn samples or minuts. 

 

or am I missing something?

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Yes, I could do that. I could use piconomix. I might as well use this logic, it would reduce a lot of development time. Thank you.
But could you help me with wear leveling anyway? I'm really interested in doing this.
 

Clament John | Still in school 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

No I can't really help you, take a step back look at the hole thing, how will it be used, and then make a plan from there.

 

I prefer loggers to be stupid!, and log readable data, so if something fail you have a change to get some kind of data, (nothing worse than something out in the field for a month and then there are nothing!).  

 

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Deleted post (spiffs)

"Dare to be naïve." - Buckminster Fuller

Last Edited: Wed. Apr 26, 2017 - 08:53 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Clement.J wrote:
But could you help me with wear leveling anyway?
The Contiki OS has a wear leveling file system :

Contiki: The Open Source OS for the Internet of Things

http://contiki-os.org/

...

Coffee flash file system

For devices that has an external flash memory chip, Contiki provides a lightweight flash file system, called Coffee. With Coffee, application programs can open, close, read from, write to, and append to files on the external flash, without having to worry about flash sectors needing to be erased before writing or flash wear-leveling. The performance of Coffee is within 95% of the raw throughput of the flash memory.

...

 


https://github.com/contiki-os/contiki/wiki/File-systems#Coffee

 

Edit : wiki

 

"Dare to be naïve." - Buckminster Fuller

Last Edited: Wed. Apr 26, 2017 - 09:24 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Hey, guys. Firstly, thank you for all your help. This has let me learn a lot about FAT, flash, etc.

I implemented a very basic wear leveling. @gchapman I did not use coffee, sorry. The reason I didn't is that I have the RTOS made by Real Time Engineers running and coffee depends on its OS

So what I have done is ----> A Flash Translation Layer which is a look-up table at the beginning of the flash memory, which will contain memory address that FAT can use to read/write. Please go through the attachment (flow chart). I have divided the AT45db041's 2048 pages into 20+1500+528, of which 1500 is the main memory, 528 is the spare memory I have kept for remapping, and 20 is for the FTL look-up table.  And Chan's FAT minimum sector size is 512, so my number of available pages for FAT is 750. So anyway here is the current issue. I'm not able to f_mount the driver, it gives FR_NO_FILESYSTEM. I get a F_OK for f_mkfs. It is similar to the scenario I had earlier #31. But in this case, I'm able to format the disk. It makes an FS for FAT12.  NOTE: It does not fail to say "too small volume".

And as before I went into the ff.c and saw that it was failing at check_fs() (Load a sector and check if it is a FAT boot sector).

* Why would it be failing even after a successful f_mkfs()?

* My write sequences are being executed well, as I'm verifying if data was written into the memory after every write (to monitor if a page has worn out).

Attachment(s): 

Clament John | Still in school 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Clement.J wrote:
Why would it be failing even after a successful f_mkfs()?
Read the sectors where it is supposed to have created the MBR (if used) and the boot/BPB and see what they actually contain.

 

To do this is may be easiest to simply extract a binary dump of some/all of the sectors and then use WinHex from X-ways (read-only = free) to study the FAT12 layout and verify that the info in the MBR/BPB is coherent.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

In ff.c at line 4364, we write FSINFO into the memory device. This is why an FAT32 formatted memory device worked perfectly fine, but my FAT12 or let alone FAT16 is not considered. 

I'm really sorry, this FS wasn't written by anyone is this forum, I know, so there is not way to understand why Chan left out on this, but if anyone is familiar with Microsoft's FAT FS could you explain why a FSINFO is not written into the memory for FAT12 and 16 formats.

For ff.c please refer to #31

Clament John | Still in school 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

FSINFO is an optional extension to FAT. It is not required to make FAT work. It's simply a "placeholder" to let the OS know the next place it is likely to be active. So that is a red herring.

 

The key parts for a FAT system are:

 

a) the Master Boot record (MBR) - this is optional but if it exists it's in the first sector and says where the partitions are located. Often (for micros) you only bother considering the first partition entry

b) the boot/BPB sector. This has the BIOS Parameter Block (BPB) and is the core of FAT because it gives all the info about where other parts of the FAT structures are located on the device. The BPB is either in sector 0 (device with no MBR) or in the first sector of the partition given in the MBR

c) numbers in the BPB tell how many FAT tables there are and how big they are. Together with fields like "reserved" they can be used to calculate where the data are is

d) the data area holds the root directory and then everything that stems from it. The AU entry in each directory entry gives the allocation Unit of the first cluster in a file chain. Using that and referring to the FAT tables the OS can then find the other clusters that make up the file storage.

 

As I say WinHex knows all this. So if you read a complete binary image of the storage area (sometimes just the bit at the front is enough) and feed that to WinHex it will locate MBR/BPB and from it the FATs and the data area/root dir and will allow you to explore the structure. It will highlight when some parts of the info are not valid.

 

If you want to post a .bin of your devicestorage after you have run f_mkfs() on it I will tell you whether it's valid or not and, if not, what is wrong.

 

(I spent about 5+ years of my working life writing analysis tools for FAT drives so I know a modicum about its operation!)

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I'm using Linux and don't own a hardware that could do a binary dump. But let me see, i could read the needed pages using my debugger and note it down.

Anyway, could you tell me what Boot Record Signature is? It is at an offset of 1FEh (at 510th byte). 

Clament John | Still in school 

Pages