HTCIA International Conference 2014 exFAT Session Completed, Slide Deck Available


HTCIA – An Introduction to the Microsoft exFAT File System 1.01 Final

Session: Introduction to the Microsoft Extended File System (exFAT)

Abstract:

This session will examine the internals of the Microsoft Extended FAT file system (nicknamed FAT64) which was designed for use with removable storage devices and is the exclusive file system of the new SDXC digital media standard. This new format creates many challenges for the forensics examiner. With minimal documentation on the internals of exFAT, and with exFAT experiencing a very high adoption rate, the forensics examiner needs guidance on how to navigate the filesystem. This session will explain the various internal tables and directory formats and show the differences from previous legacy forms of FAT, such as FAT12/16 and FAT32.

Presentation ngiven at HTCIA International Conference 2014 at Hyatt Lost Pines Resort & Spa in Austin Texas on Tuesday August 26, 2014 at 8am, in room Lost Pines 3

 

 

NYSE4SEC exFAT Session Completed, Slide Deck Available


NYC4SEC – An Introduction to the Microsoft exFAT File System 1.00 FinalNYC4SEC Meet-up group draft of the exFAT Introduction Slide Deck

In September of 2010 I presented exFAT at the 2010 HTCIA conference in Atlanta, and at the request of some that were in attendance, I posted the slide deck on SlideShare. That was 3 years ago, and the slide deck has been viewed over 5,200 times and downloaded at least 50 times.

Well next week I’ll be presenting at a NYC meet-up meeting of a group of persons with an interest in Digital Forensics. I put a lot of time and effort into the slide deck to bring it up to the times, and add new information that I was able to come into contact with.

I’ll be doing HTCIA 2014 in Texas at the end of August with basically the same deck, but this will be a dry practice run.

Although the deck is in PDF format, it is a speaker notes printout since I tried to put extra information and links into the notes area.

I’d appreciate feedback either here in the blog or at the SlideShare site

Enjoy!

exFAT VDL Feature – Two Data Lengths – File Pre-Allocation Support


In Windows XP for NTFS there is a command that sets the file lengths. It is called the FSUTIL command.

Ex: FSUTIL FILE setvaliddata

Set the valid data length for a file on an NTFS volume.

In NTFS, there are two important concepts of file length:

  • End of File (EOF) marker
  • Valid Data Length (VDL)

 

The EOF indicates the actual length of the file.

The VDL identifies the length of valid data on disk.

Any reads between VDL and EOF automatically return 0 in order to preserve the C2 object reuse requirement.

 

Offset Size Description
~ ~ Standard Attribute Header
0x00 8 File reference to the parent directory.
0x08 8 C Time – File Creation
0x10 8 A Time – File Altered
0x18 8 M Time – MFT Changed
0x20 8 R Time – File Read
0x28 8 Allocated size of the file
0x30 8 Real size of the file
0x38 4 Flags, e.g. Directory, compressed, hidden
0x3c 4 Used by EAs and Reparse
0x40 1 Filename length in characters (L)
0x41 1 Filename namespace
0x42 2L File name in Unicode (not null terminated)

Source: http://inform.pucp.edu.pe/~inf232/Ntfs/ntfs_doc_v0.5/attributes/file_name.html

 

 

In NTFS, the “End of File” can be referred to as the “allocated size of the file” while “Valid Data Length” would be the “real size of the file”.

 

There is a way to set these values via a program:

 

SetFileValidData function to set the “real size of the file”

http://msdn.microsoft.com/en-us/library/windows/desktop/aa365544(v=vs.85).aspx

 

SetEndOfFile function to set the “allocated size of the file” (Physical size)

http://msdn.microsoft.com/en-us/library/windows/desktop/aa365531(v=vs.85).aspx

 

Use of this function creates a concept of “file pre-allocation” where you can define the size of the file up front and in advance before or while writing in the file.

 

But wait …. You say – this is a blog on the exFAT file system, why am I talking about NTFS? This is not a NTFS Blog.

 

That is true, and you’re correct. BUT…

 

If we were to examine a legacy FAT directory entry, the end of the directory record is a 32 bit length of the file. A single length, because effectively Valid Data Length = Size of File.

 

Now, let’s look at exFAT:

 

From the Specification, we get:

 

 
 
Stream Extension Directory Entry  
Offset Size
Field Name (byte) (byte)
EntryType 0 1
GeneralSecondaryFlags 1 1
Reserved1 2 1
NameLength 3 1
NameHash 4 2
Reserved2 6 2
ValidDataLength 8 8
Reserved3 16 4
FirstCluster 20 4
DataLength 24 8

Source: https://www.google.com/patents/US8321439

 

We see from the definition of the Stream Extension Directory Entry two, not one: lengths. Just like NTFS, exFAT has two separate data length, each 8 bytes long which provide a maximum theoretical file length of 16 EiB. If we look at the definitions of each:

 

ValidDataLength Field

 

The ValidDataLength field describes how far into the data stream user data has been written. Implementations shall update this field as they write data further out into the data stream. On the storage media, the data between the valid data length and the data length of the data stream is undefined. Implementations shall return zeroes for read operations beyond the valid data length.

If the corresponding File directory entry describes a directory, then the only valid value for this field is equal to the value of the DataLength field. Otherwise, the range of valid values for this field is:

  • At least 0, which means no user data has been written out to the data stream
  • At most DataLength, which means user data has been written out to the entire length of the data stream

 

 

DataLength Field

If the corresponding File directory entry describes a directory, then the valid value for this field is the entire size of the associated allocation, in bytes, which may be 0. Further, for directories, the maximum value for this field is 256 MB.

 

So, what does this mean?

First, exFAT can support pre-allocation of a file in advance before writing into it.

Second, a file can be pre-allocated in a very large size to attempt to get many contiguous clusters in a single allocation so that it can use that bit to say the file is contiguous and therefore not have to update the FAT chain. Imagine this: A camera is recording a long HD movie, it gets 3 hours in the movie, then trips on an allocated cluster from another file, in order to continue recording, the camera would need to go back and build the FAT chain, in other words the camera went 3 hours without touching the FAT, it would have to go back and fill in every FAT for every cluster since the beginning of the file, and depending on cluster size – that could be a lot of fixing up to do.

 

So, this is a feature of the exFAT file system, taken from NTFS and not currently in legacy FAT. Although legacy FAT supports the VDL (Set Valid Data Length) feature, legacy FAT only has one length field in the directory record so VDL is also the physical EOF.

 

 

 

 

 

 

 

 

 

Defragmentation: Revisited


I spoke about Defragmentation of an exFAT volume.

In my write-up I made a statement that FLASH and SD Media should not be defragmented.

I’m seeing many other blog posts and articles that are actually saying the same thing – and for the same reasons. It even goes further to also include other forms of solid state drives, such as SSD. One of the main reasons why exFAT is considered “flash” friendly is that sine the life a a NAND gate is finite, you should not write unless you have to. That is why the FAT is not used or updated unless actually needed.

 

So, keep in mind the issue of defragmenting an exFAT volume should take into consideration the actual media below the file system. The primary reason/benefit for a defrag is to put data together to reduce the Seek and Latency portions of an I/O, which take forever compared to the actual read/write operation. These are physical components that slow down the I/O. When media, such as flash, is completely electronic, those physical components drop out of the equation and I/O speed should not improve after a defrag.

Another note, is that if you format the media a lot, which is good to rebuild the file system structures, do quick formats and not full or “wipe” formats. A wipe is going to write to all the memory and that could wear it down. Of course, when you d a quick format, any data on the media can be recovered forensically, so wipes may be necessary if you really need to make sure the data is physically destroyed and “wiped out”.

exFAT OEM Parms and the GUID


In my former post https://rshullic.wordpress.com/2010/01/08/sample-c-program-to-parse-out-the-exfat-directory-structure/ where I provided a crude program to parse out the exFAT filesystem I started the definitions with OEM Parameters.

 

In searching for the structure again, I found a copy at http://technet.microsoft.com/de-de/office/ee490605 and it also shows up at other sites today.

One interesting statement is: #define OEM_FLASH_PARAMETER_GUID 0A0C7E46-3399-4021-90C8-FA6D389C4BA2

Now, when I was doing my research back in 2009, I did not have flash memory, did not have Windows CE, and even when the SDXC card came it on 2010, was not able to afford one.

But, a couple of weeks ago I bought two SANDISK 64GB SDXC Media cards and started to browse them using WinHex. Now the OEM Parameters are in a section of the VBR, after the MEBS.

The Main Boot Region of the VBR is composed of five sub-regions of a total of 12 sectors:

  • The Main Boot Sector (MBS)
  • The Main Extended Boot Sectors (MEBS)
  • The OEM Parameters
  • A reserved sector
  • The Checksum Sector

Although the OEM Parameters are defined, they are not being used by the desktop/server exFAT driver. Because of this, until I got my hands on the SDXC card, the OEM PARMS VBR record has always been empty for my research. Then, then I browsed the SDXC card, I came up with one entry with the following GUID:

467EC0C0A99332140 C890FA6D389C4BA2

So, I contacted SANDISK and asked them for the format of this GUID because if you look at the structure, and you look at the WinHex dump, there was only one value, it looked like a big value. Also, when I searched, I could find anything.

SANDISK cam back with an interesting response, which was essentially that this confidential information and they can’t tell me.

Well, sorry SANDISK, if it was confidential, it isn’t anymore.

But, if you are doing research, here is something you may need to know:

Part of the GUID is little endian, and part is not. That is why I could not find it on a search. I had read this somewhere else as well, but I thought I’d mention it here.

The proper notation for the GUID is 5 sections, where the first 4 are little endian, and the last is a string.

{0A0C7E46-3399-4021-90C8-FA6D389C4BA2}If you compare it to the number above, you will see what is reversed and what is not.

Back to HTCIA International


HTCIA has its annual conference, and this year they will be having it in Austin Teas during the last week of August, 2014.

For more information: http://www.htciaconference.org/

 

 

I was contacted and informed that I will be presenting my exFAT presentation on Tuesday morning of the conference. It is an early morning session, but I’ll survive (I’m not a morning person). I have spent a lot of time over the past few weeks working on updating the presentation, and found out some stuff that has happened over the past few years. I also managed to get a couple of SDXC cards and checking those out as well.