The WordPress.com stats helper monkeys prepared a 2014 annual report for this blog.
Here’s an excerpt:
A New York City subway train holds 1,200 people. This blog was viewed about 5,100 times in 2014. If it were a NYC subway train, it would take about 4 trips to carry that many people.
HTCIA – An Introduction to the Microsoft exFAT File System 1.01 Final
Session: Introduction to the Microsoft Extended File System (exFAT)
Abstract:
This session will examine the internals of the Microsoft Extended FAT file system (nicknamed FAT64) which was designed for use with removable storage devices and is the exclusive file system of the new SDXC digital media standard. This new format creates many challenges for the forensics examiner. With minimal documentation on the internals of exFAT, and with exFAT experiencing a very high adoption rate, the forensics examiner needs guidance on how to navigate the filesystem. This session will explain the various internal tables and directory formats and show the differences from previous legacy forms of FAT, such as FAT12/16 and FAT32.
Presentation ngiven at HTCIA International Conference 2014 at Hyatt Lost Pines Resort & Spa in Austin Texas on Tuesday August 26, 2014 at 8am, in room Lost Pines 3
|
|
NYC4SEC – An Introduction to the Microsoft exFAT File System 1.00 FinalNYC4SEC Meet-up group draft of the exFAT Introduction Slide Deck
In September of 2010 I presented exFAT at the 2010 HTCIA conference in Atlanta, and at the request of some that were in attendance, I posted the slide deck on SlideShare. That was 3 years ago, and the slide deck has been viewed over 5,200 times and downloaded at least 50 times.
Well next week I’ll be presenting at a NYC meet-up meeting of a group of persons with an interest in Digital Forensics. I put a lot of time and effort into the slide deck to bring it up to the times, and add new information that I was able to come into contact with.
I’ll be doing HTCIA 2014 in Texas at the end of August with basically the same deck, but this will be a dry practice run.
Although the deck is in PDF format, it is a speaker notes printout since I tried to put extra information and links into the notes area.
I’d appreciate feedback either here in the blog or at the SlideShare site
Enjoy!
In Windows XP for NTFS there is a command that sets the file lengths. It is called the FSUTIL command.
Ex: FSUTIL FILE setvaliddata
Set the valid data length for a file on an NTFS volume.
In NTFS, there are two important concepts of file length:
The EOF indicates the actual length of the file.
The VDL identifies the length of valid data on disk.
Any reads between VDL and EOF automatically return 0 in order to preserve the C2 object reuse requirement.
| Offset | Size | Description |
| ~ | ~ | Standard Attribute Header |
| 0x00 | 8 | File reference to the parent directory. |
| 0x08 | 8 | C Time – File Creation |
| 0x10 | 8 | A Time – File Altered |
| 0x18 | 8 | M Time – MFT Changed |
| 0x20 | 8 | R Time – File Read |
| 0x28 | 8 | Allocated size of the file |
| 0x30 | 8 | Real size of the file |
| 0x38 | 4 | Flags, e.g. Directory, compressed, hidden |
| 0x3c | 4 | Used by EAs and Reparse |
| 0x40 | 1 | Filename length in characters (L) |
| 0x41 | 1 | Filename namespace |
| 0x42 | 2L | File name in Unicode (not null terminated) |
Source: http://inform.pucp.edu.pe/~inf232/Ntfs/ntfs_doc_v0.5/attributes/file_name.html
In NTFS, the “End of File” can be referred to as the “allocated size of the file” while “Valid Data Length” would be the “real size of the file”.
There is a way to set these values via a program:
SetFileValidData function to set the “real size of the file”
http://msdn.microsoft.com/en-us/library/windows/desktop/aa365544(v=vs.85).aspx
SetEndOfFile function to set the “allocated size of the file” (Physical size)
http://msdn.microsoft.com/en-us/library/windows/desktop/aa365531(v=vs.85).aspx
Use of this function creates a concept of “file pre-allocation” where you can define the size of the file up front and in advance before or while writing in the file.
But wait …. You say – this is a blog on the exFAT file system, why am I talking about NTFS? This is not a NTFS Blog.
That is true, and you’re correct. BUT…
If we were to examine a legacy FAT directory entry, the end of the directory record is a 32 bit length of the file. A single length, because effectively Valid Data Length = Size of File.
Now, let’s look at exFAT:
From the Specification, we get:
| Stream Extension Directory Entry | |||
| Offset | Size | ||
| Field Name | (byte) | (byte) | |
| EntryType | 0 | 1 | |
| GeneralSecondaryFlags | 1 | 1 | |
| Reserved1 | 2 | 1 | |
| NameLength | 3 | 1 | |
| NameHash | 4 | 2 | |
| Reserved2 | 6 | 2 | |
| ValidDataLength | 8 | 8 | |
| Reserved3 | 16 | 4 | |
| FirstCluster | 20 | 4 | |
| DataLength | 24 | 8 | |
Source: https://www.google.com/patents/US8321439
We see from the definition of the Stream Extension Directory Entry two, not one: lengths. Just like NTFS, exFAT has two separate data length, each 8 bytes long which provide a maximum theoretical file length of 16 EiB. If we look at the definitions of each:
ValidDataLength Field
The ValidDataLength field describes how far into the data stream user data has been written. Implementations shall update this field as they write data further out into the data stream. On the storage media, the data between the valid data length and the data length of the data stream is undefined. Implementations shall return zeroes for read operations beyond the valid data length.
If the corresponding File directory entry describes a directory, then the only valid value for this field is equal to the value of the DataLength field. Otherwise, the range of valid values for this field is:
DataLength Field
If the corresponding File directory entry describes a directory, then the valid value for this field is the entire size of the associated allocation, in bytes, which may be 0. Further, for directories, the maximum value for this field is 256 MB.
So, what does this mean?
First, exFAT can support pre-allocation of a file in advance before writing into it.
Second, a file can be pre-allocated in a very large size to attempt to get many contiguous clusters in a single allocation so that it can use that bit to say the file is contiguous and therefore not have to update the FAT chain. Imagine this: A camera is recording a long HD movie, it gets 3 hours in the movie, then trips on an allocated cluster from another file, in order to continue recording, the camera would need to go back and build the FAT chain, in other words the camera went 3 hours without touching the FAT, it would have to go back and fill in every FAT for every cluster since the beginning of the file, and depending on cluster size – that could be a lot of fixing up to do.
So, this is a feature of the exFAT file system, taken from NTFS and not currently in legacy FAT. Although legacy FAT supports the VDL (Set Valid Data Length) feature, legacy FAT only has one length field in the directory record so VDL is also the physical EOF.
I spoke about Defragmentation of an exFAT volume.
In my write-up I made a statement that FLASH and SD Media should not be defragmented.
I’m seeing many other blog posts and articles that are actually saying the same thing – and for the same reasons. It even goes further to also include other forms of solid state drives, such as SSD. One of the main reasons why exFAT is considered “flash” friendly is that sine the life a a NAND gate is finite, you should not write unless you have to. That is why the FAT is not used or updated unless actually needed.
So, keep in mind the issue of defragmenting an exFAT volume should take into consideration the actual media below the file system. The primary reason/benefit for a defrag is to put data together to reduce the Seek and Latency portions of an I/O, which take forever compared to the actual read/write operation. These are physical components that slow down the I/O. When media, such as flash, is completely electronic, those physical components drop out of the equation and I/O speed should not improve after a defrag.
Another note, is that if you format the media a lot, which is good to rebuild the file system structures, do quick formats and not full or “wipe” formats. A wipe is going to write to all the memory and that could wear it down. Of course, when you d a quick format, any data on the media can be recovered forensically, so wipes may be necessary if you really need to make sure the data is physically destroyed and “wiped out”.
In my former post
where I provided a crude program to parse out the exFAT filesystem I started the definitions with OEM Parameters.
In searching for the structure again, I found a copy at http://technet.microsoft.com/de-de/office/ee490605 and it also shows up at other sites today.
One interesting statement is: #define OEM_FLASH_PARAMETER_GUID 0A0C7E46-3399-4021-90C8-FA6D389C4BA2
Now, when I was doing my research back in 2009, I did not have flash memory, did not have Windows CE, and even when the SDXC card came it on 2010, was not able to afford one.
But, a couple of weeks ago I bought two SANDISK 64GB SDXC Media cards and started to browse them using WinHex. Now the OEM Parameters are in a section of the VBR, after the MEBS.
The Main Boot Region of the VBR is composed of five sub-regions of a total of 12 sectors:
Although the OEM Parameters are defined, they are not being used by the desktop/server exFAT driver. Because of this, until I got my hands on the SDXC card, the OEM PARMS VBR record has always been empty for my research. Then, then I browsed the SDXC card, I came up with one entry with the following GUID:
467EC0C0A99332140 C890FA6D389C4BA2
So, I contacted SANDISK and asked them for the format of this GUID because if you look at the structure, and you look at the WinHex dump, there was only one value, it looked like a big value. Also, when I searched, I could find anything.
SANDISK cam back with an interesting response, which was essentially that this confidential information and they can’t tell me.
Well, sorry SANDISK, if it was confidential, it isn’t anymore.
But, if you are doing research, here is something you may need to know:
Part of the GUID is little endian, and part is not. That is why I could not find it on a search. I had read this somewhere else as well, but I thought I’d mention it here.
The proper notation for the GUID is 5 sections, where the first 4 are little endian, and the last is a string.
{0A0C7E46-3399-4021-90C8-FA6D389C4BA2}If you compare it to the number above, you will see what is reversed and what is not.
HTCIA has its annual conference, and this year they will be having it in Austin Teas during the last week of August, 2014.
For more information: http://www.htciaconference.org/
I was contacted and informed that I will be presenting my exFAT presentation on Tuesday morning of the conference. It is an early morning session, but I’ll survive (I’m not a morning person). I have spent a lot of time over the past few weeks working on updating the presentation, and found out some stuff that has happened over the past few years. I also managed to get a couple of SDXC cards and checking those out as well.
I have been invited by the meet-up group NYC4SEC to do a presentation on the exFAT file system on June 11, 2014 and by HTCIA at their 2014 International conference on Tuesday August 26, 2014.
Been updating the power point deck to pick up any good stuff to add. After I give the presentations I’ll post the new decks to this blog. I also expect to be adding sone new stuff as well. I have a SDXC card on order and will be doing some testing and acquisitions which I have been wanting to check out for a while now!
Steve Bunting is listed as either an author or co-author of two books in the forensics field that contain sections on exFAT – and gives a plug for my SANS paper on exFAT.
One book is:
By Steven Anson, Steve Bunting, Ryan Johnson, Scott Pearson
and the other is:
By Steve Bunting
Thanks Steve!
It looks like Windows 7 may have some interesting exFAT stuff. I am in the process of upgrading one of my laptops to Windows 8 and will the check the differences. Windows 8 allows creation of bootable exFAT media, so it will be interesting looking at the boot VBR records.
exFAT Extended FAT File System 