+
  
SERVICES
  
PRODUCTS
  
CONTACT US

VMware ESXi Snapshot corruption fix for datastore with no available disk space

by Chris Hall/December 1, 2016


Recently we had an issue with a client’s server where corrupted snapshots of a virtual server caused multiple snapshot files allocating more than 100% disk space to a single virtual server (a domain controller). The virtual server in question was still running but would frequently crash as there was not enough disk space to allow for virtual swap files. The fix (and a last stich effort option we didn’t have to use) is below.

Symptoms:
-Error messages stating virtual swap file cannot be created.

-Multiple swap files (e.g.: servername_00001.vmdk, servername_00002.vmdk, etc.) allocating more than 100% disk space

-Datastore with little space available.

-Servers going offline.

-Stuck tasks shown in VMware ESXi SSH (see http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1013003)

Steps to Resolve:

Option 1:
Correcting the snapshots by consolidation

1. Power off the problem VM (or preferably all VMs). If this is not done the VMs continue to use swap space and if you have stuck tasks they can to. If you have stuck tasks reboot the server in full and keep the VMs powered off.

2. Clear ANY space available on the VM. Do not delete or rename any VMDK files or anything else in a VMs directory.

3. Take another snapshot of the problem VM. Ensure to un-tick the options to ‘Snapshot the virtual machine’s memory’ and ‘Quiesce guest file system’. If these are kept ticked the snapshot will use more disk space and cause further complication.

4. Once complete, open Snapshot manager and delete all snapshots. Doing so should consolidate all files and return the VM and the ESXi server to full health. This process may take several hours to complete and it normally gets stuck for most of this time at the 99% mark (as this is when it is actually moving the data).

Note: Do not run the ‘Consolidate’ function at any stage, only the ‘Delete All Snapshots’. Running the consolidate function will again use more disk space than required.

Option 2:
Correcting the VM by VMware Converter

This option should not be your first one. It is possible (if you can get you VM powered on) to use VMware Converter to convert your VM (V2V) to another datastore or server. This can be a complicated process as it either requires another standalone VMware ESXi server or the addition of further internal storage (Normally SAS, SCSI or SATA) to the current ESXi server.

USB Drives cannot be used for this as they are not supported in ESXi for direct datastore usage. The process should be similar to the below:

1. Locate an appropriate disk (for example a 1TB SATA disk).

2. Power off the ESXi Server.

3. Install the SATA disk in a spare SATA slot.

4. Power on the ESXi Server.

5. Do a scan for the new storage.

6. Create a new datastore.

7. Move the affected VM to the new datastore (giving it more disk space).

8. Run the process in Option 1 again.

9. If the Option 1 process does not work, power on the VM and use VMware Converter to V2V back to the original datastore.

Note: The V2V process does not recreate the snapshot files but does create a new Virtual Disk without the previous issues.

← Back to News


Want to say hey or find out more?


If you're part of a small/medium or enterprise level we
can help address your IT Management needs.

Get in touch
    
View our Services
                
Make your data
work for you.
Contact Us

+61 438 661 875
inquiries@datacommand.com.au

     

Copyright © DataCommand Pty Ltd / Privacy Policy
ABN 98 205 256 068 / ACN 131 977 323