VMware ESXi Snapshot corruption fix for datastore with no available disk space
by Chris Hall/December 1, 2016
Recently we had an issue with a client’s server where corrupted snapshots of a virtual server caused multiple snapshot files allocating more than 100% disk space to a single virtual server (a domain controller). The virtual server in question was still running but would frequently crash as there was not enough disk space to allow for virtual swap files. The fix (and a last stich effort option we didn’t have to use) is below.
-Multiple swap files (e.g.: servername_00001.vmdk, servername_00002.vmdk, etc.) allocating more than 100% disk space
-Datastore with little space available.
-Servers going offline.
-Stuck tasks shown in VMware ESXi SSH (see http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1013003)
Steps to Resolve:
1. Power off the problem VM (or preferably all VMs). If this is not done the VMs continue to use swap space and if you have stuck tasks they can to. If you have stuck tasks reboot the server in full and keep the VMs powered off.
2. Clear ANY space available on the VM. Do not delete or rename any VMDK files or anything else in a VMs directory.
3. Take another snapshot of the problem VM. Ensure to un-tick the options to ‘Snapshot the virtual machine’s memory’ and ‘Quiesce guest file system’. If these are kept ticked the snapshot will use more disk space and cause further complication.
4. Once complete, open Snapshot manager and delete all snapshots. Doing so should consolidate all files and return the VM and the ESXi server to full health. This process may take several hours to complete and it normally gets stuck for most of this time at the 99% mark (as this is when it is actually moving the data).
This option should not be your first one. It is possible (if you can get you VM powered on) to use VMware Converter to convert your VM (V2V) to another datastore or server. This can be a complicated process as it either requires another standalone VMware ESXi server or the addition of further internal storage (Normally SAS, SCSI or SATA) to the current ESXi server.
USB Drives cannot be used for this as they are not supported in ESXi for direct datastore usage. The process should be similar to the below:
1. Locate an appropriate disk (for example a 1TB SATA disk).
2. Power off the ESXi Server.
3. Install the SATA disk in a spare SATA slot.
4. Power on the ESXi Server.
5. Do a scan for the new storage.
6. Create a new datastore.
7. Move the affected VM to the new datastore (giving it more disk space).
8. Run the process in Option 1 again.
9. If the Option 1 process does not work, power on the VM and use VMware Converter to V2V back to the original datastore.