Terraform: Choosing credentials for a remote state file

I have existing infrastructure in Terraform and have been using it for a while. Recently I had swapped the AWS credentials of my local laptop (the creds stored in ~/.aws/credentials) and it stopped working until I re-set those credentials back.

The problem is that I'm declaring the creds in the Terraform source itself but it doesn't seem to be using them at all.

terraform {
  backend "s3" {
    bucket = "example_tf_states"
    key    = "global/vpc/us_east_1/example_state.tfstate"
    encrypt = true
    region     = "us-east-1"
  }
}

provider "aws" {
  access_key = "${var.access_key}"
  secret_key = "${var.secret_key}"
  region = "${var.region}"
}



variable "access_key" {
  default = "<hidden_for_stack_exchange_post>"
}

variable "secret_key" {
  default = "<hidden_for_stack_exchange_post>"
}

variable "region" {
  default = "us-east-1"
}

The access ID permissions are 100% good. I am using the same account ID and secret key both for the aws configure settings that go into ~/.aws/credentials as I am in the above Terraform variable declarations.

Everything works fine as long as the creds are in ~/.aws/credentials but as soon as the OS level credentials are gone (ie rm ~/.aws/credentials) I get the following when trying to run Terraform operations, such as terraform plan:

Failed to load backend:
Error configuring the backend "s3": No valid credential sources found for AWS Provider.
  Please see https://terraform.io/docs/providers/aws/index.html for more information on
  providing credentials for the AWS Provider

Please update the configuration in your Terraform files to fix this error.
If you'd like to update the configuration interactively without storing
the values in your configuration, run "terraform init".

If I re-populate the ~/.aws/credentials by running aws configure it will work fine again.

I'm not understanding -- if my provider setting is explicitly declaring the credentials to use inside the Terraform source code, why does my OS-level AWS configuration matter at all?

How can I make Terraform only use the creds defined in my Terraform configuration and ignore what's in my OS user profile?

Edit, it's Terraform v0.11.7

Edit: Please note that I'm trying to solve the issue on why the statically declared creds are not being utilized in the provider declaration. Not looking for alternative methods or workarounds. Thanks.

Your first question

if my provider setting is explicitly declaring the credentials to use inside the Terraform source code, why does my OS-level AWS configuration matter at all?

The error message "Failed to load backend: Error configuring the backend "s3"" is referring to your Backend S3 configuration.

Look in the file ./.terraform/terraform.tfstate and you will see the S3 Backend configuration.

The Terraform S3 Backend is different than the Terraform AWS Provider. The error message "No valid credential sources found for AWS Provider." is misleading. It implies that the AWS Provider configuration is used, which is false. S3 Backend credentials are configured separately and stored in the terraform.tfstate file.

Your OS-level AWS configuration matters because if no S3 Backend credentials are specified, as documented here https://www.terraform.io/docs/backends/types/s3.html, then Terraform defaults to using the following, in order:

1. Environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
2. AWS Shared credentials file, default value is "~/.aws/credentials".

You didn't specify any credentials in your S3 Backend config so terraform is defaulting to the AWS Shared Credentials File.

Your S3 Backend configuration contains no credentials.

terraform {
  backend "s3" {
    bucket = "example_tf_states"
    key    = "global/vpc/us_east_1/example_state.tfstate"
    encrypt = true
    region     = "us-east-1"
  }
}

Your second question,

How can I make Terraform only use the creds defined in my Terraform configuration and ignore what's in my OS user profile?

First, Backends cannot contain interpolation, see https://www.terraform.io/docs/backends/config.html. So you cannot use any variables in the Backend config. e.g. this config is invalid

terraform {
  backend "s3" {
    bucket = "example_tf_states"
    key    = "global/vpc/us_east_1/example_state.tfstate"
    encrypt = true
    region     = "us-east-1"
    access_key = ${var.access_key}
    secret_key = ${var.secret_key}
  }
}

If you want to specify AWS credentials when running terraform init you specify backend configuration as options.

terraform init --backend-config="access_key=your_access_key" --backend-config="secret_key=your_secret_key"

This produces a S3 Backend config that looks like this, stored in the ./.terraform/terraform.tfstate file:

{
    "version": 3,
    "serial": 1,
    "lineage": "bd737d2d-1181-ed64-db57-467d14d2155a",
    "backend": {
        "type": "s3",
        "config": {
            "access_key": "your_access_key",
            "secret_key": "your_secret_key"
        },
        "hash": 9345827190033900985
    },

Again, the S3 Backend credentials are configured separately from your AWS Provider credentials.

Re-run terraform init and specify the credentials on the command line as --backend-config options to fix your error.

The error you're getting is specifically referring to configuring the S3 backend, which AFAIK does not inherit the settings from the AWS provider configuration; it too has access_key & secret_key configuration options which if you're not using ~/.aws/credentials you will need to explicitly configure.

You are better off setting up profiles in your ~/.aws/credentials files like

[profile1]
aws_access_key_id = xxxx
aws_secret_access_key = xxxxx
region = us-east-1

[profile2]
aws_access_key_id = xxxx
aws_secret_access_key = xxxx
region = us-west-2

Then in your provider you can tell it which profile to use

provider "aws" {
  profile = "profile2"
  region = "${var.region}"
}

It will keep the keys out of your terraform files which is a good thing if you ever want to put them into source control.