large allocation unit (FAT32/NTFS Clusters)

December 20, 2009 at 11:57:25
Specs: wXP, n/a

I was wondering if the below setup is interesting:

I have this computer, on which I would like to use my C drive only as OS disk. For some reason I forgot, I've chosen NTFS as File System ... but, this is the question:

This drive will never be filled (I keep it relatively empty on purpose), so disk usage it not an issue.

But, speed is. For that, I come to understand that using larger allocation units, provides better performance results ... with the drawback of an inferior disk usage (particularly for small files).

I've also read that most systems assume that allocation units are 4K ... and some "things" may work less good when this is altered.

So, creating large allocation units, example 32K units, is it a good idea or not ?

Not sure how these allocation units link up to NTFS versus FAT


See More: large allocation unit (FAT32/NTFS Clusters)

Report •


#1
December 20, 2009 at 12:44:16

http://support.microsoft.com/defaul...
Overview of FAT, HPFS, and NTFS File Systems

Report •

#2
December 20, 2009 at 13:16:20

Also, don't neglect reading Define Cluster Size Properly in THIS article.

i_Xp/Vista/W7User


Report •

#3
December 20, 2009 at 13:30:21

I suggest Googling for this phrase "adjusting cluster size for speed". I skimmed through some of the hits to refresh my limited knowledge of the pros & cons. The concensus is that you may get a very small gain in speed with larger cluster sizes when working with larger files.

The issue seems to be that when files are written to a cluster and outgrow that cluster the adjacent cluster is probably already in use so the overflow may be weritten to a totally different section of the drive.

That would produce a performance hit. The solution is to keep your drive defragged.

If you are really into this you should use partitions designed with file type in mind and use cluster sizes to match the typical file size. For example if you have lots of music. Place it on its own partition. Smaller partitions equal faster access times to.


Report •

Related Solutions

#4
December 21, 2009 at 11:44:29

Data is written to a drive on a first available first use basis. So lets say we have 4kb clusters [cluster is a collection of 512k sectors] and a file that is 128kb in size. It would be written to the first available cluster and the next until it ran out of contiguous space. Then it would jump to the next available clusters and repeat the action.

Files are not written contiguous to the drive. Deleting files creates holes that are also available to be written. This is why file system become fragmented.

With large databases, video files, movies, etc there is a real advantage in both performance and storage if the clusters are sized according to usage.

For example, if you were writting 1gig files [1024mb] it would be advantageous to have a cluster size that was 1gig. The file would remain contiguous and it would only take one pass of the read/write heads to read the file. The drive would not fragment.

Unfortunately the practicality of this in nonexistant. Files only fall into averages. This is one of the reasons MS chose the 4kb size as default for ntfs. This maximizes disk usage [less slack- partially filled clusters] but also makes defragmenting the drive of much greater importance. I have seen 32k clusters work well for databases.

To answer your question, for everyday usage 4kb is just fine especially for the OS.


Report •

#5
December 22, 2009 at 03:42:28

Thanks for the info, I'll have a look on the size of files currently on the system ... but that is not a real mistery, on the drive, there would only be : Windows XP

- Program Files is not on that drive
- I'm using a defragmentation tool already (non-Microsoft)
- I don't use partitioning

I guess a lot of Windows DLL are larger than 4K


Report •

#6
December 22, 2009 at 11:33:41

Had a check:

- 56 DLL's in C:\Windows being 4096 bytes or higher
(max. 13360 KB)
- 8372 DLL's smaller than 4096 bytes


all files:
- 26817 smaller than 4096 (including directories)
- 136 being 4096 or bigger


So, indeed not a lot of files exceeding 4K in size, in there. Conclusion would still remain that in theory, 8K or 16K would result in better performance, but lots of storage lost.

In case of converting 4K to 16K, that would mean at least 26817 files residing in a 32K cluster instead of 4K, which means additional disk usage of (32-4)K*26817 = 750876 K = 733 M

Still, 700 megs is nothing, if I have 50 gigs free


Report •

#7
December 22, 2009 at 13:20:23

196 larger than 4kb
30,000+ smaller.

Does not support your conclusion "8K or 16K would result in better performance"


Report •

#8
December 23, 2009 at 11:35:29

Well, it does for those 196 files, and for the 30.000 it then doesn't matter since they would have same.

So, same + faster = faster

I know I'm simplifying things here, but if the whole thing with allocation units serves that purpose (performance through less allocation units), well, didn't we achieve that goal then ?


Report •

#9
December 23, 2009 at 13:34:47

You will see no difference. Just like you can't tell the difference between a nanosecond and a millisecond. As humans, we can't detect those differences.

Report •

#10
December 23, 2009 at 13:44:02

I think there must be a name for these kind of things ... I mean, I know many systems and many options, designed to improve stuff (like performance, but anything else can also be), but if you ask a practical example ... the end result is, argh, don't bother, it will end up with the same result (for us to see) but requiring more work.

I got to understand that this option would only be interesting if you run programs with large files, like databases. Then again, handling these files, even 512K is a joke, and too small.

Thanks for the feedback anyway


Report •

#11
December 23, 2009 at 14:32:04

> Also, don't neglect reading Define Cluster Size Properly
> in THIS article.

Some interesting things in there, like the "Define Cluster Size Properly" part ... and here, well, there're just determining that 16K is the best size (not even knowing or caring what files we have) : section 2.3 in http://ixbtlabs.com/articles/ntfs/i... (it also just says : "The more the size is, the better is the performance (especially for FAT32)."

There you go.

That one also says that small HD's, better use FAT.

And, something I read somewhere in the past : "having cluster size more than 4 KB compression is not supported"
So, no compression, but that is not a problem.


Report •

#12
December 23, 2009 at 14:42:43

You have to think about what you read and what it is actually saying.

For example in your example of 2.3 the larger fat32 cluster size reduces the FAT [file allocation table]. It is not the cluster size that makes the difference it's the file allocation table size that allows less to be read thru.

You are not using FAT but NTFS which uses a MFT [master file table] which is different from FAT.

Changing the cluster size in addition to no compression also changes other things like [if I recall correctly] the MS native defrag doesn't work as well as encryption I don't believe works either.

Performance is a combination of many things not just the hard drive cluster sizes.

Old school would have you placing data in the middle of the drive and directories at the innermost for faster reads but with todays modern drives this is a waste of lifespan.

Real performance gain at this level is having the files be contigous despite what cluster size you choose.
Netware wrote files to the server contigous. I always wonder why MS didn't do the same.


Report •

#13
December 23, 2009 at 15:04:17

> For example in your example of 2.3 the larger fat32
> cluster size reduces the FAT [file allocation table].
> It is not the cluster size that makes the difference
> it's the file allocation table size that allows less to
> be read thru.
>

I understand that, no problem there.

But still:
- The default cluster size is not suited for best performance
- I won't be using MS defragmentation, MS encryption or MS compression (so I don't care they work less good or not at all)
- NTFS is not a requirement, FAT is still an option


Report •

#14
December 23, 2009 at 15:57:22

If you elect to use FAT32 you will need to format with something other than the WinXP CD. Windows won't allow your to format a partition larger than 32GB using FAT32. Below are the default cluster sizes for FAT32 and NTFS.

I understand it is possilbe to format a FAT 32 partition with a non standard cluster size. However, I believe that doing so will prevent you from using the defrag tools in Windows.

If you are really concerned about speed then consider using smaller partitions. Average seek time should fall when using smaller partitions.

Cluster sizes

for FAT32 are as follows:
512MB to 8,191MB = 4KB
8,192MB to 16,383MB = 8KB
16,384MB to 32,767MB = 16KB
Larger than 32,768MB = 32KB

NTFS - All partitions on a PC = 4KB default


Report •

#15
January 3, 2010 at 08:57:11

Yes, speed of 1st concern ... for that disk. I'm going to re-instally my XP anyway, so I'm gonna have a test. I actually don't care if I can only use 32Gig, since that is more than what I need. Below is the current config, which is an NTFS with 4K clusters, which I will want to change into FAT32 with 32K clusters. I'll not be partitioning since I want to keep this disk free of any load. I'll indeed need to take care how to format the disk, at moment of XP installation with CD ...

C:\>chkdsk C:
The type of the file system is NTFS.
Volume label is OS.

WARNING! F parameter not specified.
Running CHKDSK in read-only mode.

CHKDSK is verifying files (stage 1 of 3)...
File verification completed.
CHKDSK is verifying indexes (stage 2 of 3)...
Index verification completed.
CHKDSK is verifying security descriptors (stage 3 of 3)...
Security descriptor verification completed.

72597703 KB total disk space.
17685756 KB in 75212 files.
24796 KB in 5748 indexes.
0 KB in bad sectors.
340999 KB in use by the system.
65536 KB occupied by the log file.
54546152 KB available on disk.

4096 bytes in each allocation unit.
18149425 total allocation units on disk.
13636538 allocation units available on disk.

C:\>


Report •


Ask Question