KVM guest time shift on host reboot

I have installed Ubuntu server 12.04 as a KVM host, and created an Ubuntu guest that is also running Ubuntu server 12.04. The servers are in the AST time zone which is UTC + 3 hours. After setting the time correctly in the guests using date -s I am able to reboot the guests as much as I like with the correct time being retained between reboots.

If however the KVM host is rebooted (the host has no time issues between reboots), then once I start my KVM guests all of them are exactly 3 hours behind the correct time. The output below shows the system time and hardware clock time on the host and guest.

root@kvmhost1:~# date
Wed Jan 16 14:10:38 AST 2013
root@kvmhost1:~# hwclock 
Wed 16 Jan 2013 02:10:40 PM AST  -0.187899 seconds

root@kvmguest1:~# date
Wed Jan 16 11:10:45 AST 2013
root@kvmguest1:~# hwclock 
Wed 16 Jan 2013 02:10:47 PM AST  -0.521226 seconds

Once the time is corrected on the guests then they can be rebooted without the time shifting incorrectly.

Things I've tried/tested

• I tried this on three separate physical servers all exhibit the same exact behaviour.
• One of the kvm hosts is running Ubuntu 11.04 and the same behaviour happened.
• I've considered using a time server to resolve the issue, but Ubuntu recommends not using NTP on the KVM guests. There is a good discussion of this in KVM guests clock synchronisation, which suggests that kvm-clock running in the guest should take care of this.
• I've run dmesg | grep kvm-clock on the guest and it confirmed kvm clock is in use.

Update

I set the hardware clock to use local time using the --localtime option on the hwclock command. The same issue still persists. The output is a little clearer though because the software and hardware clocks are set with the same time.

root@kvmhost1:~# date
Wed Jan 16 15:50:26 AST 2013
root@kvmhost1:~# hwclock 
Wed 16 Jan 2013 03:50:29 PM AST  -0.750378 seconds

root@kvmguest1:~# date
Wed Jan 16 12:50:29 AST 2013
root@kvmguest1:~# hwclock 
Wed 16 Jan 2013 12:50:31 PM AST  -0.849677 seconds

The solution

The answer by Michael Hampton below solved the problem. The guest was created using a kickstart file which by default doesn't set the hardware clock on UTC time. I added the --utc option to the kickstart timezone option and it solved this issue. The kickstart option is now as follows:

timezone --utc Asia/Bahrain

It appears you've somehow installed your guest in a way that it considers the hardware clock to be in local time. KVM guests should always consider their hardware clock in UTC time, unless they're Windows guests (but even those can be set to UTC hardware clock)...

Edit the /etc/default/rcS file, and change UTC=no to UTC=yes. Then reset your guest clock.