Micro$oft FAT16/Partitioning/Formatting Questions

Log in or register to post comments
Go To Last Post
21 posts / 0 new
Author
Message
zainka
zainka's picture
Level: Hangaround
Joined: Fri. Nov 16, 2001
Posts: 475 View posts
Location: Grimstad, NO(r)WAY!
#1
Posted by zainka: Wed. Oct 18, 2006 - 10:28 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Hello

I have currently spent my days trying to understand FAT16 and formatting of a disk, and lots of other stuff related to this. Thus, naturaly for me, there has arised some questions close to the end of my spine (back-bone). Hopefully some of you can share some light.
Here goes...

MBR - Master boot record
My first concern is related to the MBR when formating a device (yes I know, mmc and sd memory card is delivered pre-formatted, however in our case we need the format function for several purposes, including changing MBR sector, please do not ask why :) ). Aboute the boot code located in MBR, can this be copied from any "DOS/Windows" formatted memory card and be used in any mmc and sd memory cards which is formatted by our system to be used between our system and PC or Mac/Linux based systems? Does the Linux systems require different master boot code (cant beleive this to be true)?

The partition table will under any circumstances be built each time a format command is executed, and we do not suport GUID partition table for those which is familiar with this thingy, thus we are dealing with a "standard" MBR. Each partition (normaly only one) is to be FAT16, however, for FAT16 there is several partition types/ID's available like 0x06 for DOS 3.31 compatibility with CHS mapping or Win95 compatibility using LBA mapping. Question is, if setting partition ID to 0x0e, can I then drop setting the start and end CHS address in partition table and only use LBA mapping (0ffseth 0x08 and 0x0c (nr. of sectors in partition) in partition table)? Or would a win95 or higher version of Win OS say that partition is dead if CHS is missing.? what aboute Mac and Linux distros which supports LBA mapping (which I beleive should be any distroes available), does they read wind95 Fat16 formatted sd/mmc cards correctly?
(I ask these questions since there is some time before we can try it in real product).

Is it possible to format a memory card without any MBR on top at all (skip it) and use the whole memory for one partition where first physical sector in memory is the boot record for FAT16, and still atleast a windows enviroment will read the memory correctly?. Had to ask...

FAT16 - Boot record (BR)
The BR is not ment to contain any boot code (i.e. not a bootable partition) and, if using MBR above, the "bootable"-flag in the partition table is set to 0x00 = "partition have no boot code", thus question is, is it enough to set offset 0x00 (BR jump vector) to 0x000000, will OS ignore it?

What should the media descriptor (offset 0x15) be set to. For some memory cards (128MB MMC) it was set to 0xf8 giving "Single sided disk, 80 tracks per side, 9 sectors per track". The card was formatted using winXP sp4 format command from cmd prompt.Offset 0x18 (nr. of tracks) said nr. of tracks is 63 and as far as I know 63 is not like 80... Any comments?
What should the correct media descriptor and track/head values in BR be?

I guess i later will dig up more questions but I leave this for now, Hopes anyone can contribute into this.

Thanks in advance

Regards
Vidar (Z)

----------------------------------------------------------

"The fool wonders, the wise man asks"

Tags:

AVR Microcontrollers, megaAVR and tinyAVR
Top
Jepael
Jepael's picture
Level: Raving Lunatic
Joined: Mon. May 24, 2004
Posts: 6279 View posts
Location: Tampere, Finland
Posted by Jepael: Wed. Oct 18, 2006 - 10:53 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

You are dealing with MMC cards instead of hard disks right?

I believe MMC cards and such devices have no MBR like hard drives do, the MMC cards are like floppy disks, cannot be partitioned like hard drives and they only have the FAT16/whatever boot sector.

And about the Media Descriptor Byte, it is not widely used, and it is set to 0xf8 to look like a floppy disk or hard disk or something, so basically the sectors/cylinders/heads are not determined from there.

I don't know how the sectors/cylinders/head parameters should be set, but there are maximum limits for each of them, and propably the card uses none of these parameters, as the sectors are linearly addressed I think. They are just there to represent the file system, if DOS or other operating system wants to calculate the card size in sectors or something.

Read more info from Microsoft developer network (MSDN) and Wikipedia also has good articles about FAT16/FAT32

- Jani

EDIT: and about the bootability thing. A floppy disk for example is ALWAYS bootable. If the floppy cannot load an operating system like DOS, the floppy disk has enough code to print the "This floppy is not bootable" text on screen. So a FAT16 Boot Record always has code in it.

Top
zainka
zainka's picture
Level: Hangaround
Joined: Fri. Nov 16, 2001
Posts: 475 View posts
Location: Grimstad, NO(r)WAY!
Posted by zainka: Wed. Oct 18, 2006 - 11:26 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Thanks for your reply but I have to correct you a little. As mentioned in my opening I use winhex to read the physical content of a memory device like MMC and SD, wich is one of several memory technologies I intend to deal with but we leave this for now, and winhex shows that memory cards like MMC do begin with 32 hidden sectors (physical sector 0 to sector 31) where sector 0 was reserved for MBR with valid partition table but, ofcourse, with only one partition entry used.

I have gathered almost all information from wikipedia and other sources and I have becomed quite familiar with the FAT16/FAT32 structures, mostly my problems is related to such things like the BR boot code and

And, finaly, floppy disks is also always FAT12 which I do not suport. But, could i take the boot code from a non bootable disk, either it be a floppy, memory card or non bootable partition from a hdd, including the jump vector, and use it uncritical inside my boot record. Note, I am not familiar with Intel assembly so I have no possebility to extract what the boot code realy does.

Final, I have tried to use MSDN, but Micro$oft gives little information for developers working on my level, mostly information given is for sw application engineers. Details and exceptions for MBR, BR and bootcodes are deply burried in between uninteresting information.

Regards
Vidar (Z)

----------------------------------------------------------

"The fool wonders, the wise man asks"

Top
clawson
clawson's picture
Level: Moderator
Joined: Mon. Jul 18, 2005
Posts: 97332 View posts
Location: (using avr-gcc in) Finchingfield, Essex, England
Posted by clawson: Wed. Oct 18, 2006 - 11:48 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

For Windows to recognise a drive (but not boot from it) you don't need to have any actual code in either the MBR or the BPB sectors. Obviously the code that's normally there on a MS-DOS/Windows formatted drive is 8086 code but such drives are just as readable in Macs or other CPU based computers and they don't care about the existence of such code. The only things that MUST appear are:

1) LBA 0 on the device (the MBR) MUST contain at least one 16 byte partition table record at offset 0x1BE and the 55/AA signature at the end of the 512 byte sector. No other data but 0x00 has to appear in this sector. In fact within the partition record you probably don't even need CHS at offset 0x01 because most modern systems will only use start LBA at offset 0x08 and length at 0x0C anyway. There can optionally be a four byte disk signature at 0x1B8 but this doesn't HAVE to be there - so can be 0x00000000

2) The sector identified as the start of the partition at 0x1BE+0x08 in the MBR is the boot sector containing the BPB. It's traditional for the first byte to 0xE9 which is an 8086 JMP instruction but the offest can be 0x0000 following it with no further 8086 code present. At offset 0x03 there is an 8 character OEM name but this is optional. At offset 0x0B onwards is the Bios Parameter Block (BPB) - these figures must exist so the OS knows the structure of the drive. The values are as documented on:

http://en.wikipedia.org/wiki/Fil...

Under "Boot sector". Depending on which FAT system the BPB ends at either offset 0x3D or 0x59 and the rest of the space is for "boot code" but the fact is that apart from the very last 0x55/0xAA at the end of the sector all these bytes can remain 0x00

So the bottom line is that you do not need to include ANY code in either MBR or partition boot sector though the 0xE9 at the start of the boot sector is a good idea for compatability.

Cliff
(who works on satellite PVRs where the file system is effectively FAT32 but with the very sparse sectors described above - our disks are completely readable in PCs as "normal" FAT32 drives)

Top
zainka
zainka's picture
Level: Hangaround
Joined: Fri. Nov 16, 2001
Posts: 475 View posts
Location: Grimstad, NO(r)WAY!
Posted by zainka: Wed. Oct 18, 2006 - 01:58 PM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Cliff:
Thanks for valuable information, this is exactly as I hoped it would be.
----

According to offset 0x18 and 0x1a in boot record (Nr. of sectors in track/Nr. of heads respectively). For a memory card these values does not have any meaning. In the MMC card reffered to these values are set to 0x003f and 0x00ff respectively, it gives no usefull info except that this is the maximum values allowed in a CHS mapping. I guess this is the reason. (but then why 2 use bytes each for these entries???)
Never mind

Thanks

Regards
Vidar (Z)

----------------------------------------------------------

"The fool wonders, the wise man asks"

Top
clawson
clawson's picture
Level: Moderator
Joined: Mon. Jul 18, 2005
Posts: 97332 View posts
Location: (using avr-gcc in) Finchingfield, Essex, England
Posted by clawson: Wed. Oct 18, 2006 - 02:13 PM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

As modern operating systems use LBA to access drives a lot of the head/track/sector stuff is redundant anyway as the OS just "sees" a huge pool of sectors and doesn't care where they are physcially located on the drive as it no longer has to instruct the drive to position to track 81, Sector 14, head 2 and so on. Instead it just says read sector 43,287 or write sector 1,592,341 or whatever and the drive handles the actual location in it's ATA6/7 firmware

Cliff

Top
zainka
zainka's picture
Level: Hangaround
Joined: Fri. Nov 16, 2001
Posts: 475 View posts
Location: Grimstad, NO(r)WAY!
Posted by zainka: Wed. Oct 18, 2006 - 06:16 PM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Yes, I am aware of this, I think I have gathered enough information to be able to tune my system as wanetd.

There is only one thing inside a directory entry for deleted files which scares me a litle. There is two possible values which can be written to the firs letter of a deleted file, this is 0xe5 and 0x05.

I belive the system uses value 0x05 as long as it is bussy with freeing used clusters, i.e. when deleting a file then first the first letter in directory entry is set to 0x05, then system is starting to free clusters and when finished then first letter is set to 0xe5.

I also think that the clusters is deleted in the oposite direction as they was allocated. Doing it this way will prevent memory leakage which could happen if power goes before all clusters was freed. Thus, as long as the system finds a dir entry which is tagged 0x05, it knows that this is a deleted file but the FAT update is not finished yet.

If this is not the case, then I do not know what the diferenties between 0x05 and 0xe5 should be. Also, still if this is not the case, then sooner or later there will be a memory leakage if power is lost during deleting. Ofcourse Micro$oft has always said that "be careful not to turn off power in a non - controlled way", but I cant deal with the powerplants and their regular schedules for switching power off, neither with users of my product, or computers for that case

Or am I youst a bit to concerned now???

Regards
Vidar (Z)

----------------------------------------------------------

"The fool wonders, the wise man asks"

Top
Jepael
Jepael's picture
Level: Raving Lunatic
Joined: Mon. May 24, 2004
Posts: 6279 View posts
Location: Tampere, Finland
Posted by Jepael: Wed. Oct 18, 2006 - 07:40 PM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

From [url]http://en.wikipedia.org/wiki/Fat16 [/url]

Quote:

0x05 Initial character is actually 0xE5
0xE5 Entry has been previously erased and is not available.

So you must use 0xE5 to delete it.

- Jani

Top
laserman
laserman's picture
Level: Hangaround
Joined: Fri. Feb 7, 2003
Posts: 116 View posts
Posted by laserman: Wed. Oct 18, 2006 - 11:00 PM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

If DIR_Name[0] == 0x05, then the actual file name character for this byte is 0xE5. 0xE5 is
actually a valid KANJI lead byte value for the character set used in Japan. The special 0x05 value
is used so that this special file name case for Japan can be handled properly and not cause FAT file
system code to think that the entry is free.

Roger

Top
laserman
laserman's picture
Level: Hangaround
Joined: Fri. Feb 7, 2003
Posts: 116 View posts
Posted by laserman: Wed. Oct 18, 2006 - 11:15 PM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

As Cliff says, wikipedia and a few other links on the web tell most of it. There is one other good document *and it's from Microsoft*. Search for "Hardware White Paper" "FAT: General Overview of On-Disk Format". Read every word in infinite detail in that paper.

As you've already looked at the MMC with Winhex, you can see that CHS etc doesn't come into it. It's just a straight linear bunch of sectors - actually, just a linear bunch of bytes, but they generally get written in blocks of 512, so it looks like sectors. I simply set up a MMC as a FAT16 device, one partition, dead simple, and read/write standard DOS files via SPI. I've had too many incompatibilities (between a device and Windows systems running on different disk formats) with media formatted as FAT32.

Top
zainka
zainka's picture
Level: Hangaround
Joined: Fri. Nov 16, 2001
Posts: 475 View posts
Location: Grimstad, NO(r)WAY!
Posted by zainka: Thu. Oct 19, 2006 - 07:02 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

And as I have said, I have read all the stuff on wikipedia, which do not explain 0x05 vs 0xe5. But now ,reading the white paper, i have got the explanation as requested. Strangly enough does not this documentation come up as an resource when searching for FAT16 documentation at MSDN as, again, is because this fora is for sw application engineers only. In fact, when searching for "FAT white paper" using alltheweb/google, its mostly universities and other non MS sites which comes up.

However, then there realy is a possebility to leak memory if a delete prosess is interrupted because of power failure or because user removes card while file is beeing deleted. I can live with this since there is wery rare chances for this to happen but I think it is importante to be aware of that this actual can happen. Also, this has always been true for all FAT based systems (and others i beleive).

Regards
Vidar (Z)

----------------------------------------------------------

"The fool wonders, the wise man asks"

Top
clawson
clawson's picture
Level: Moderator
Joined: Mon. Jul 18, 2005
Posts: 97332 View posts
Location: (using avr-gcc in) Finchingfield, Essex, England
Posted by clawson: Thu. Oct 19, 2006 - 09:48 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

zainka wrote:
I also think that the clusters is deleted in the oposite direction as they was allocated.

Don't see how that would be possible - FATS are not doubly linked - they are only forward linked so for any kind of access you have to start with the "first cluster" field in the directory entry and work forward through the FAT chain. To work backwards you (a) wouldn't know what the last cluster of this file was (which of those many 0xFFFF's is it??) and (b) say you were at cluster 0x0123 and the previous cluster was 0x6244 then you'd have to scan the entire FAT for "0x0123" until you found it in the 0x6244 location. And you'd have to do this scan for every cluster.

Nope, I'm 99.9999% certain that deletion is forward through the file chain.

Cliff

Top
clawson
clawson's picture
Level: Moderator
Joined: Mon. Jul 18, 2005
Posts: 97332 View posts
Location: (using avr-gcc in) Finchingfield, Essex, England
Posted by clawson: Thu. Oct 19, 2006 - 10:43 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

laserman wrote:
There is one other good document *and it's from Microsoft*. Search for "Hardware White Paper" "FAT: General Overview of On-Disk Format". Read every word in infinite detail in that paper.

Google finds many copies. One is located here:

http://home.teleport.com/~brainy...

It looks like a really excellent description as it gives Microsoft's view on things and how they use/interpret the fields of the BPB etc. rather than simply documenting what they are.

Thanks for the pointer laserman!

Cliff

Top
zainka
zainka's picture
Level: Hangaround
Joined: Fri. Nov 16, 2001
Posts: 475 View posts
Location: Grimstad, NO(r)WAY!
Posted by zainka: Thu. Oct 19, 2006 - 11:03 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Cliff:

I agree,I am even 100% sure, now... It is correct that a FAT link is single linked, thus one would need to, if preventing leakage was a case, to

1) Go to end of chain and free last cluster used (0x0123) and remember which cluster nr. this was (for eksample 0x0120), then again, go from beginning of chain and free the cluster pointing to 0x0120 and now remembering which cluster nr. this was (for eksample 0x0111), then once again, go from beginning of chain and find cluster nr pointing to 0x0111 and now remember which cluster this was, and so on. Yes wery(!) time consuming, but still only way to be ensured there is no memory leakage. However, no needs for scanning entire FAT. (It might is also be possible that one does not need to remember previous cluster in chain since when freeing last cluster equals to setting it to 0x0000, thus previous cluster now will point to a free cluster, which is illegal, and can be released, leading to fact that prev. cluster for this cluster again will point to a free cluster ..... )

Alternatively (2) the complete chain (or a part of it) must be copied into memory before deleting so u know which clusters to delete, something which would be faster but which then requires unknown amount of memory and memory must be non volatile as well.

Nevertheless. If there was an interrupt in the deleting prosess, when power comes back, system needs to know that it was bussy with deleting a file and which one(s) this was and check that entire file cluster chain wsas released.

So, yes, deletion is forward through the file chain and that the (rare) possibility of losing memory capacity can't be omitted when following FAT specs...

One could however make a "system validate procedure" which cleaned the FAT table going through all cluster entries in FAT to see if all clusters is member of a valid chain..

laserman:

Quote:
As you've already looked at the MMC with Winhex, you can see that CHS etc doesn't come into it.

Sorry but both in MBR partition table and in BPB the CHS value i swritten by format command. What I said was that values in the boot record table is irrelevant and gives no valubal meaning. CHS addresses in MBR however is set to 0x010100 for offset 0x1bf and 0x07e0d2 for offset 0x1c3.

Regards
Vidar (Z)

----------------------------------------------------------

"The fool wonders, the wise man asks"

Top
Jepael
Jepael's picture
Level: Raving Lunatic
Joined: Mon. May 24, 2004
Posts: 6279 View posts
Location: Tampere, Finland
Posted by Jepael: Thu. Oct 19, 2006 - 11:22 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

zainka wrote:
Cliff:

One could however make a "system validate procedure" which cleaned the FAT table going through all cluster entries in FAT to see if all clusters is member of a valid chain..

[

That's exactly why rebooting win98 starts scandisk before loading windows, to retrieve the possibly lost clusters. If file deletion is interrupted, the last clusters of the file are still marked to be in use in FAT table, but there is nobody pointing at them, and this can be corrected in the next boot. Maybe you should do something like this when power is turned on in your device?

I think Win98 somehow marks if the file system was shut down properly or not and runs scandisk if it was not. But you can do this just by creating an empty file on bootup, and deleting it on shutdown. If the file already exists when you try to create it, then you know you have to do a FAT scan for lost clusters.

- Jani

Top
clawson
clawson's picture
Level: Moderator
Joined: Mon. Jul 18, 2005
Posts: 97332 View posts
Location: (using avr-gcc in) Finchingfield, Essex, England
Posted by clawson: Thu. Oct 19, 2006 - 11:34 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

And in the good old days it was left up to us, the users of MS-DOS to run the chkdsk utility from time to time and if you used the /F parameter (fix problems) it would not only find the orphaned file chains but write their contents to visible files in the root directory. The fact that file chains could be left orphaned in this way when the PC crashed during a file write (which used to be a common occurrence!) kind of suggests that the links WERE being deleted forwards.

Cliff

Top
laserman
laserman's picture
Level: Hangaround
Joined: Fri. Feb 7, 2003
Posts: 116 View posts
Posted by laserman: Fri. Oct 20, 2006 - 04:11 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Incidentally, if anyone looks at that Microsoft paper that I referenced a few posts ago, the FAT32 spec document doesn't come up easily if you search on the exact title they refer to in the first doc. Look for fatgen103.pdf (as opposed to fatgen102.pdf). one place you can find it is at i30www.ira.uka.de/~sdi/2006/doc/fat....

These docs are also available in French, at least.

Roger

Top
zainka
zainka's picture
Level: Hangaround
Joined: Fri. Nov 16, 2001
Posts: 475 View posts
Location: Grimstad, NO(r)WAY!
Posted by zainka: Fri. Oct 20, 2006 - 07:13 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Chkdsk and scandisk was in my mind.
To avoid memory leakage in this product, hwever, I think we will go for a variant of the "empty file" which Jani mentioned. Diference is that a dummy file is created on root before creating, writing to, or deleting a file, and dummy file will be deleted after we are finished with the target file.

This dummy file will use only one cluster and contain information aboute which filename, and path, which is going to be changed ++. I aint going through all scenarioes related to this methode but we are several here which finds that this might solve the problem. Also, if deleting a dummy file failed, it is only one cluster missed, we can also avoid this if we scan root folder for deleted dummy files and check that dreleased clustrer realy is free.

Thanks to all

Ahh, one more thing. I have thought of an idea to transfer FAT16 into a double linked list and still be compatible with FAT16 used in PC (i.e. a PC can still access the drive perfectly). The idea is to double the FAT table size and use upper half as normal FAT i.e. single linked list, as is pr. today, and second half is identical with upperhalf except that the chain goes the opposite way, pointing backwards. Doing it this way might make the system bakward compatible (needs further investigation).

However, there is many issues with this methode like, with two FAT tables (one for backup), there will now be 4 places where the FAT tables needs to be updated. Also, if you write to card from PC, the PC wont update second half of FAT (effectively making the double linked list non-valid), but a PC can read all content without problems making it usefull for backing up or transfering data. This is youst an irresponsible idea, havn't verifyed it yet and I wont implement it, but there is some places it might be usefull. youst wanted to share it. Lets call it FAT162

Regards
Vidar (Z)

----------------------------------------------------------

"The fool wonders, the wise man asks"

Top
zainka
zainka's picture
Level: Hangaround
Joined: Fri. Nov 16, 2001
Posts: 475 View posts
Location: Grimstad, NO(r)WAY!
Posted by zainka: Fri. Oct 20, 2006 - 07:50 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

One more thing...

Cliff:
I noticed that the link you reffered to for the FAT32 spec. is an older version. I have found the latest version at Microsoft. Rev. number is only going from 1.02 to 1.03, but still, it is always usefull to use latest rev. The link is below.

http://search.microsoft.com/resu...

This site contain link to both pdf and word formatted spec.

Regards
Vidar (Z)

----------------------------------------------------------

"The fool wonders, the wise man asks"

Top
Bingo600
Bingo600's picture
Level: Raving Lunatic
Joined: Sun. Apr 25, 2004
Posts: 4076 View posts
Location: Denmark
Posted by Bingo600: Sun. Oct 22, 2006 - 04:58 PM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

zainka wrote:
One more thing...

Cliff:
I noticed that the link you reffered to for the FAT32 spec. is an older version. I have found the latest version at Microsoft. Rev. number is only going from 1.02 to 1.03, but still, it is always usefull to use latest rev. The link is below.

http://search.microsoft.com/resu...

This site contain link to both pdf and word formatted spec.

Is it FAT32 or "just" the "long filenames" that M$ has a patent on ??

/Bingo

Top
zainka
zainka's picture
Level: Hangaround
Joined: Fri. Nov 16, 2001
Posts: 475 View posts
Location: Grimstad, NO(r)WAY!
Posted by zainka: Mon. Oct 23, 2006 - 07:23 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Good question which I have found better not to ask.
Wikipedia has some information according to licensing.
http://en.wikipedia.org/wiki/FAT...

This is one of the refferences in same wiki which I have found interesting. Pr. now, i think there is no issues with using FAT(12/16/32) as long as LFN isnt used.

http://news.com.com/Microsoft+FA...

Enjoy

Regards
Vidar (Z)

----------------------------------------------------------

"The fool wonders, the wise man asks"

Top