UPDATED – Oh the good ‘ol days, I remember when I first started using thin provisioning in late 2006, learning the ropes, learning ins and outs.. I can’t count how many times I did data migrations between volumes to reclaim space at the time.
Today times have changed, a few years ago many companies started introducing thin reclamation technologies which provides a means for the computer to communicate to the storage which blocks are not being used and can be reclaimed by the shared storage system for use in other volumes.
Initially software support for this was almost non existent, short of a Microsoft tool called sdelete, which was never designed with storage reclamation in mind (I assume ..) it would be the tool of choice for early reclamation systems since it had the ability to zero out deleted space in a volume. Of course it only worked on Windows boxes.
Later came support from Symantec in their Veritas products, and VMware in their VAAI technologies (though in VMware’s case it seems they suggest you disable support because storage arrays can’t keep up which drives latency up – wonder if that includes 3PAR ?). Then Oracle announced their own reclamation system which the built in co-operation with 3PAR. Myself at least, I have not seen many other announcements for such technology.
A few days ago I came across this post from Compellent which talks about the support of the SCSI UNMAP command in Red Hat Enterprise 6, apparently integrated into the ext4 file system (or somewhere else in the layer so that it is transparent, no special tool needed to reclaim space). That sounded really cool, my company is an Ubuntu shop at the moment, and as far as I can tell there is no such support in Ubuntu at this time (at least not with 10.04 LTS).
One of my co-workers pointed me to another tool called fstrim, which seems to do something similar as sdelete.
Fstrim is used on a mounted filesystem to discard (or “trim”) blocks which are not in use by the filesystem. This is useful for solid-state drives (SSDs) and thinly-provi-sioned storage.
I have not yet tried it. YAT (Yet Another Tool) I came across is zerofree, but this tool is not really too useful since it needs the file system to be in read only mode or totally unmounted.
fstrim is part of the util-linux package in newer Debian-based distributions (Ubuntu 10.04 excluded), so it should be supported by your distro if your on a current release. It is also part of the latest versions of Red Hat Enterprise.
Any other thin reclamation tools out there (for any platform), or any other thin reclamation support built into other software stacks ?
UPDATE – Upon further investigation I believe I determined that this new Linux functionality was introduced into the kernel as something called FITRIM which Ubuntu says was part of 2.6.36 kernel. I see an XFS page that mentions real time reclamation (what RHEL 6 does) is apparently part of the 3.0 kernel, so I assume RHEL back ported those changes.
on any linux you can just run DD to zero out space: http://linux.die.net/man/1/dd
example: dd if=/dev/zero of=/disk bs=1M count=1024 would create a 1gb file of zeros.
Comment by linux — March 26, 2012 @ 3:13 am
yes that is true, though kind of a pain in the ass depending on the size of the file system and what’s on there. Usually you’d want to zero out the entire file system while at the same time not causing any writes to fail when the file system is filled with zeros for that brief period of time.
thanks for the post!
Comment by Nate — March 26, 2012 @ 8:39 am
[…] script that writes out a ton of zeros to the file system to reclaim this space (since I recently learned that the kernel code required to do fancier stuff like fstrim [updated that post with new information at the end since I originally wrote it] doesn't exist on my […]
Pingback by 3PAR Zero detection and snapshots « TechOpsGuys.com — April 12, 2012 @ 9:03 am
Interesting post. I’ve also been playing with sg_unmap and sg_write_same but didn’t have much success. The command seem to always work on the actual scsi device rather than a volume from the volume manager on top of it. I was (naively) planning to “unmap” a volume from a thinly provisioned LUN in order to reduce the number of times I have to reprovision LUNs just because the volume sizes keep changing.
Of course I can still dd the volume with /dev/zero but then I have to run a reclamation on the array (which, in an EMC shop, is a migration to a new LUN with all the overhead it takes).
Proper reclamation inside the filesystem is still not worth the effort imho.
Comment by Martin — May 9, 2012 @ 12:47 am
When you say migration to a new LUN for reclaiming space I assume that is transparent to the application using the LUN (minus the I/O overhead needed)?
I agree that reclaiming space is still somewhat of a PITA given all of the moving parts, I’ve gotten much better results by just leveraging LVM on the volumes and controlling growth. I’m upgrading the software on my little 3PAR array in about a week which provides an improved array defragmentation algorithm (since only 128MB contiguous blocks(which may or may not map to 128MB contiguous blocks in the OS) can be fully reclaimed on 3PAR). There was some nasty bugs with regards to system overhead in their first generation defragmentation algorithm so they released a patch that disabled it entirely until they could come up with a fix. My array was installed late last year with this patch to disable defragmentation but I was never notified, it would of saved some time when troubleshooting why all the space was not being reclaimed. There’s at least a few hundred gigs of space waiting to get defragmented to be reclaimed.
thanks for the post!
Comment by Nate — May 9, 2012 @ 9:39 am
Starboard Storage Systems supports the SCSI unmap command for space reclamation but maybe more importantly avoids writing zeros to disk. It does this by stipping out the zeros after data is written to the built in SSD write cache and simply creating pointers to zero blocks. you can read more about it in this blog post. http://blog.starboardstorage.com/blog/bid/155959/The-Scourge-of-the-Zero-Why-nothing-may-be-ruining-the-efficiency-of-your-storage-array
Comment by Lee johns — May 24, 2012 @ 9:32 pm