FatFs file creation problem beyond 511 files

Go To Last Post
8 posts / 0 new
Author
Message
#1
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Hi,

I have been using the FatFs for a few months with fantastic reliability. Recently, I made a few mods to my application that requires creating 4096 individual files successively on my SD Card. each file is created one at a time, written to and closed before any other file is created. I always have one file open at any one time.

While performing load tests, I have noticed that I can create 510 files successfully but the creation of the 511th file fails. This is a very consistent and repeatable problem and I am positive that it has to do with a limitation (or a setting) on the FatFs.

Can anyone give some guidance to why this may be happening?

Regards...

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I guess you are using FAT16? ;-)

Switch to FAT32 and your problems will be solved (one of the improvements in FAT32 is that the root is just another file chain so is not limited to a single cluster).

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Hi clawson. Thanks for this suggestion. Yes, I am using FAT16.

However, I cannot see the logic here. If I am opening and closing 510 independent files, why should I hit a limit as such? I can understand that there may be limitations to the maximum number of bytes that can be written to a file (depending on FAT16 and FAT32) but the number of files I can open and close must be unlimited.

I must also note that I do f-sync per file to ensure stuff gets written physically to the disc. I am also debugging my application at the moment and the file object fs pointer seems to be zero for the 511th file when I try to create it. That is where the problem starts. However, I cannot see why the pointer is initialised to zero.

If you could elaborate on your response that will be very useful for my understanding.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Hello again clawson.

I had a rethink and a re-read after my last post. You mean to say I cannot save more than 512 files under any folder in FAT16. Is that right?

If the above is true, can I create a different subfolder for each group of 510 files? That way since I am dealing with a maximum of 4000 files, I will need 4 subfolders with ~500 files in them. Then I can stick to my good old FAT16 right?

In that case I will need to be careful with the total number of files (including all the subfolders) on my SD Card which cannot exceed 65536 in total. Is my understanding correct?

Regards...

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Quote:

You mean to say I cannot save more than 512 files under any folder in FAT16. Is that right?

You obviously have access to the web, and there you will find e.g. Wikipedia. Never under-estimate the possibility that someone has written down the answer to your question long before you actually pose it. :wink:

The article on FAT has this passage:

Quote:
The number of root directory entries available for FAT12 and FAT16 is determined when the volume is formatted, and is stored in a 16-bit field. For a given number RDE and sector size SS, the number RDS of root directory sectors is RDS=ceil((RDE×32)/SS), and RDE is normally chosen to fill these sectors, i.e., RDE*32=RDS*SS. FAT12 and FAT16 media typically use 512 root directory entries on non-floppy media.

This hints at it only being the root directory that has the limitation. Have you tried to create more than 512 files in one sub-directory.

Totally aside: It is my strong opinion that the term "folder" is something that should be used in the context of graphical UIs and not elsewhere. The generic term is "directory".

As of January 15, 2018, Site fix-up work has begun! Now do your part and report any bugs or deficiencies here

No guarantees, but if we don't report problems they won't get much of  a chance to be fixed! Details/discussions at link given just above.

 

"Some questions have no answers."[C Baird] "There comes a point where the spoon-feeding has to stop and the independent thinking has to start." [C Lawson] "There are always ways to disagree, without being disagreeable."[E Weddington] "Words represent concepts. Use the wrong words, communicate the wrong concept." [J Morin] "Persistence only goes so far if you set yourself up for failure." [Kartman]

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

The root directory in FAT12 and FAT16 is a fixed size.

There should be no limit on the number of files in a subdirectory.

If the mkfs() has created a Volume entry in the root directory, this will use up a slot. e.g. 512 slots = 1 vol + 511 files

Fatfs should give you an error if it runs out of root directory slots. Subdirectories simply allocate another cluster to contain a new set of directory slots.

Untested. I am sure Wikipedia explains it more accurately.

Incidentally, I am sure that you can ask for a bigger root directory when doing the mkfs().

David.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Don'tcha just love it when you type a long reply (and on a tablet keyboard!) and then it's lost when you [Submit]?!?! Grr!

@OP, Anyway what I was trying to say was do you understand about FAT cluster allocation? In FAT16 the root directory is a special case in that it is always the first cluster in the data area (has to be so you can find it!) that comes (usually) immediately after the (two) FAT tables but it is just one reserved cluster. In FAT16 the chances are the clusters are 16KB (32 sectors) and each sector can hold 16 file/directory entries so there are 16*32=512 files/directories available in the root. There is no mechanism for the root to extend beyond one cluster so this size is fixed.

Now the file and directory entries in it are all identical. The 32 bytes that describe a file includes 11 bytes for the name and 2 bytes for the "first cluster". When a file or directory pointed to by such an entry grows beyond 16KB (or whatever the cluster size is) the allocation system simply looks up the current cluster in the FAT, finds the next free one and puts its number in the current cluster's entry (and the next one is 0xFFFF as an end of chain mark). So a file or (sub)directory can grown limitlessly(*).

In FAT32 they had the good sense to make that first cluster, that holds the root, just the first entry in a FAT chain itself. So if you go beyond one cluster (16KB/512 files) then a FAT chain link is created to allocate another cluster to the root. So it, like files and sub-dirs can also grow limitlessly(*).

In summary FAT16: root limited, sub-dirs unlimited(*). FAT32: all dirs unlimited(*)

(*) now I say "unlimited" but that's a lie. If you read fatgen103.doc (Microsoft's seminal work on the FAT systems) you will discover that there is a limit on files in a directory because of the use elsewhere of a 16 bit counter. Now I can't remember whether it's treated as signed or unsigned so the limit is either 32768 or 65536 files. However it's probably just as well that such a limit exists as the search of directories tends to be linear and things could really bog down if you had hundred of thousands of files in a single directory. (The later NTFS and EXT4 file systems have mechanisms (B-tree in EXT4) to cope with the scenario and do fast indexing of many thousands of files)

If you suspect you need more than 32768/(65536?) files in a directory then split them over several directories (perhaps by month or something?).

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Thank you all for these clarifications. I did do some reading on this in parallel and confirmed that 512 files can be held under root directory and each subfolder under the root can have 65536 entries within it.

Note that the 512 includes the filed as well as the subfolder names. Also, longer filenames that have a
'~' suffix also shorten this 512 limit.

In my particular case, creating just one folder under the root under which I will need to save 4096 files will be more than enough.

As one last remark, the reason why I could only save 510 rather than 512 under the root was because there were 2 really old filed that were already sitting there! So it all adds up.

Thank you all once again for coming to the rescue.

Regards...