exFAT Defragmentation

I am starting to see some questions come up regarding defragmentation of an exFAT volume. Looking today at Diskeeper, it doesn’t appear that they provide support for their product on the exFAT filesystem.

Keep in mind that defragmentation can be a problem, especially if you set options to run the defrag constantly. exFAT was designed for removable media, such as USB sticks and SD cards, which use a flash type of solid state memory. Because the chips are gated circuits, they degrade over time as you write to them. Although some gates can withstand 10,000 or 100,000 or even millions of writes per gate over the life of the chip, your ability to write files is finite, and you can effectively wear out sections of the chip memory by excessive writes.

Since exFAT is a Microsoft proprietary file format, any program that implements the file system requires a license of the specification from Microsoft. Now, I am not a lawyer, and I am not that well versed with the Microsoft licensing scheme, but the way I think it works, if someone want to write a defrag program, and be legit, they may need the Microsoft license. If they don’t have the license, then you could be at risk – sort of hiring an unlicensed plumber to work on your pipes.

I see the need for at least two defragment scenarios.

The first scenario is the directory, which includes subdirectories and the root directory. In certain cases directory entries are marked “not in use” which is common when a file is either deleted or under special renaming circumstances. exFAT will tend to write new directory entries rather than reuse already inactive entries. I believe that the reason for this may be to spread the gate changes to reduce the wear on the flash memory. However, it is theoretically possible to compress the directory and if there is many “not in use” entries, could possibly free up clusters if you do a compress. The possible problem with a directory compress is that each file has a “search hash” for quick file searching. Actually, this hash is yet another Microsoft patent, and if the compression of the directory requires a recalculation of the search hash value, I think this is where Microsoft licensing can be an issue.

The other scenario is the defragmentation of the files themselves. Keep in mind that the subdirectories are also files as well, but if the subdirectory is defragmented – but not compressed – I don’t think the search hash values are affected. The difference between compress and defragment is that compress removed records imbedded in the fie itself, but defragment just puts the clusters in the proper order and puts them together.

exFAT will attempt to write files in a unfragemented manner. By using the allocation bitmap, when a file is allocated, exFAT will try to store it as unfragmented. Why? Because it is faster, and faster for 2 reasons. The more traditional reason as with any filesystem, even back to mainframe days, is that when a physical disk is used, if a file is fragmented, the physical read/write heads have to move to the physical cylinder/head/sector to read the block. If the file is unfragmented, head movement is minimal or non-existent except to seek to the first sector. Flash memory is electronic random access and this hardly applies as there is no physical movement. The 2nd reason, and part of the reason why exFAT is supposed to be faster than FAT32, is the cluster run. A cluster run is the chain in the FAT, in which the clusters are chained in a forward single direction link list. This means that to find the next cluster, the FAT is referenced to find the address of the next cluster. FAT12, FAT16 and FAT32 (what I call legacy FAT) all use cluster runs to track allocation of a file. exFAT does not use the FAT to track allocation (the allocation bitmap is used for that) and the cluster run – the cluster chaining – is only needed and used when the file is fragmented. If the file is NOT fragmented, there is no cluster run, the FAT is not written to, and a bit in the stream extensions directory record called the “FAT Chain Invalid” flag is set. This extra writing, reading, and management of the cluster run goes away if the file is not fragmented. So, for performance reasons, it is better to keep the file unfragmented so you don’t incur the cost of cluster runs.

So, what do you do if you really need to defragment the exFAT volume? Without the tools and utilities today, you are sort of out of luck. But there is a brute force method, it is risky, and it is time-consuming. But if you have to, then you have to.

You may be able to just copy a few files off the drive (I really mean MOVE) to free up the clusters and hope to create a big enough hole so that you can build a large contiguous space. Then move them back. The more you move, the higher the probability that you will remove the bottleneck. In the worse case scenario, move everything to a separate different drive, reformat the drive that should now be empty because you moved everything off, then move everything back. This may be your only hope right now if your stuck.


One thought on “exFAT Defragmentation

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s