What is reasonable storage failover time that most OS (VM) can tolerate?

The disk driver typically will wait until a configurable timeout is exceeded before even reporting an error for the operation requested.

As you have found out, this is /sys/block/<devicename>/device/timeout in Linux and defaults to 60 30 seconds.

Windows is storing this configuration as a global setting TimeoutValue (REG_DWORD) in HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Disk\ with a 60 seconds default.

As long as no error is reported upstream, you will see no immediate action (like a ro-remount of the FS), even after the timeout is up you will typically see more error handler action (logging, resetting the device etc.) before an error is passed back to the upper layer.

But be aware that there will be other implications affecting overall availability.

• applications or system services might implement own timeouts and throw exceptions on expiration
• on servers with a high request turnaround, you will see queues fill up and memory exhaust as new clients keep submitting new requests with the old requests still waiting for the storage to respond.
• if you happen to have swap space on the failed device, all page in / page out requests will stall, effectively blocking the processes working on these memory pages.

In general, you will want to keep the failover time as low as possible while still operating without premature failovers due to occasional load spikes or network glitches. Determining the right value for your use specific case is very much trial-and-error work over a prolonged period of operation. For general-use server VMs I would aim for something in the magnitude of 10 seconds, if feasible and supported by your infrastructure.

FreeBSD has the geom_mountver (https://www.freebsd.org/cgi/man.cgi?gmountver), which can be used to make it tolerate any failover time. If you're using ZFS, you might need to disable the deadman timer; it will panic the box if an IO doesn't complete in 15 minutes (IIRC).