VM Inaccessible

So I had to remotely shut down my servers last night as the air conditioner crapped out on us. I shut down the VMs and then the hosts. Fired up the 2 hosts today which auto start the VMs. Datastores for the VMs are isci LUNs located on a Synology NAS that we have, the NAS was not shutdown. For some reason when I booted everything back up my email VM shows as inaccessible and I am getting all sorts of errors in the logs saying "ATS-ONLY VMFS volume 'Synology-Email server' not mounted. Host does not support ATS or ATS initialization has failed" all of my other VMs are perfectly fine with other datastores on the same NAS.

When browsing to the datastore in Vsphere client or vcenter then I cannot mount, unmount or browse files as they are all greyed out. I have tried restarting both the host and the NAS a few times.

ESXi 6.0 on both hosts

Solution 1:

BINGO!!! Alright, here is how this was resolved:

I referenced this artice for unmounting the datastore, https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2004605

Step 1: Manually unmount the datastore from the host with this command:

esxcli storage filesystem unmount [-u UUID | -l label | -p path ]

In my case I just used the label command.

Step 2: Verify that datastore is unmounted:

esxcli storage filesystem list

You see output similar to:

Mount Point Volume Name UUID Mounted Type Size Free
------------------------------------------------- ----------- ----------------------------------- ------- ------ ----------- -----------
/vmfs/volumes/4de4cb24-4cff750f-85f5-0019b9f1ecf6 datastore1 4de4cb24-4cff750f-85f5-0019b9f1ecf6 true VMFS-5 140660178944 94577360896
/vmfs/volumes/4c5fbff6-f4069088-af4f-0019b9f1ecf4 Storage2 4c5fbff6-f4069088-af4f-0019b9f1ecf4 true VMFS-3 146028888064 7968129024
/vmfs/volumes/4c5fc023-ea0d4203-8517-0019b9f1ecf4 Storage4 4c5fc023-ea0d4203-8517-0019b9f1ecf4 true VMFS-3 146028888064 121057050624
LUN01 4e414917-a8d75514-6bae-0019b9f1ecf4 false VMFS-unknown version 0 0

The Mounted field is set to false, the Type field is set to VMFS-unknown version, and that no Mount Point exists.

Step 3: Disabling ATS-Only lock settings.

I referenced this article:

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1033665

In vcenter (or through vsphere client) I went to Hosts and Clusters and selected my host that was giving the issues, browsed to Settings and Advanced System Settings. In Advanced System Settings I changed 3 values as follows (all of them were set to 1):

DataMover.HardwareAcceleratedMove = 0  
DataMover.HardwareAcceleratedInit = 0
VMFS3.HardwareAcceleratedLocking = 0

Step 4: Re-mounting datastore:

In vcenter I selected the datastore giving me problems, right clicked and selected Mount Datastore and then powered on my VM.

Then test and quit for the day.

Solution 2:

This tip worked for me in the past, if you can disable ATS-Only:

Cannot mount the VMFS5 datastore when using storage encryption appliances (2030416)

I guess no VM are up in your case, but they need to be close to be able to run the command.

Some storage encryption appliances may block certain SCSI commands, including ATS commands, sent from ESXi hosts. VMFS5 datastores newly created on ATS-capable devices are marked as ATSOnly after the first successful ATS command to the device. This allows all hosts sharing the datastore to utilize ATS without checking whether or not the storage array supports it. When the storage encryption device is then added to the SAN, it blocks the ATS commands. As a result, the host assumes that the array does not support ATS because it did not receive the expected response. The expected behavior is that the host does not fall back to using SCSI-2 Reservations for locking mechanism on the ATSOnly VMFS5 datastore, which then gets unmounted and cannot be mounted again. To work around this issue, disable ATSOnly on the VMFS5 datastore.

Note: All virtual machines must be migrated off the affected datastore, or powered off, prior to running the below steps. To disable ATSOnly on the VMFS5 datastore:

Connect to one of the hosts sharing the VMFS5 datastore with an SSH session. For more information, see
Run the following command:

vmkfstools --configATSOnly 0 /vmfs/devices/disks/device-ID:Partition

Where:

device-ID is the NAA ID of the LUN on which the VMFS5 datastore was created.
Partition is the partition number on which VMFS5 datastore was created. This is usually 1.

For example:

vmkfstools --configATSOnly 0 /vmfs/devices/disks/naa.6006016055711d00cef95e65664ee011:1

Note: It is sufficient to run this command on one of the hosts sharing the VMFS5 datastore. Other hosts automatically recognize the

change.

Run the following command to rescan for datastores:

esxcli storage filesystem rescan

The VMFS5 datastore should now mount successfully.