Why is terraform erring with 'SignatureDoesNotMatch: Signature expired'?

I started getting the following when running terraform commands:

$ terraform refresh
Error refreshing state: 1 error(s) occurred:

* SignatureDoesNotMatch: Signature expired: 20170226T035111Z is now earlier than 20170227T013047Z (20170227T014547Z - 15 min.)
        status code: 403, request id: 7626f995-fc8e-11e6-9b73-7990b2a1af41

What does it mean? How to fix it?

Solution 1:

The AWS APIs use an authentication mechanism that involves the client creating a signature representing the request. This signature includes a timestamp of the time the request is being made, as a measure to reduce the risk of replay attacks.

The signature mechanism assumes that the client and server both have somewhat-accurate system clocks, such that the client's time is not more than 15 minutes divergent from the server's.

So it's important that any system where AWS APIs are used -- whether via Terraform or otherwise -- has a reasonably accurate system clock. Virtual machines often suffer from system clock drift for various reasons, so it's a good idea to run ntpd on most virtual machines to prevent their system clocks from drifting too much.

Solution 2:

It turned out my virtual machine where terraform was installed was way out of sync with time. Using the date command and a current time resolved the issue.