Proper access policy for Amazon Elastic Search Cluster

I've recently started using the new Amazon Elasticsearch Service and I can't seem to figure out the access policy I need so that I can only access the services from my EC2 instances that have a specific IAM role assigned to them.

Here's an example of the access policy I currently have assigned for the ES domain:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::[ACCOUNT_ID]:role/my_es_role",
        ]
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:[ACCOUNT_ID]:domain/[ES_DOMAIN]/*"
    }
  ]
}

But as I said, this doesn't work. I log into the EC2 instance (which has the my_es_role role attached to it) and attempt to run a simple curl call on the "https://*.es.amazonaws.com" end point, I get the following error:

{"Message":"User: anonymous is not authorized to perform: es:ESHttpGet on resource: arn:aws:es:us-east-1:[ACCOUNT_ID]:domain/[ES_DOMAIN]/“}

Does anyone know what I have to change in the access policy in order for this to work?

Solution 1:

You can lock access down to IAM-only, but how will you view Kibana in your browser? You could setup a proxy (see Gist and/or NPM module) or enable both IAM and IP-based access for viewing results.

I was able to get both IAM access IP-restricted access with the following Access Policy. Note the order is important: I could not get it working with the IP-based statement before the IAM statement.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::xxxxxxxxxxxx:root"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-west-2:xxxxxxxxxxxx:domain/my-elasticsearch-domain/*"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-west-2:xxxxxxxxxxxx:domain/my-elasticsearch-domain/*",
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": [
            "192.168.1.0",
            "192.168.1.1"
          ]
        }
      }
    }
  ]
}

My EC2 instance has an instance profile with the arn:aws:iam::aws:policy/AmazonESFullAccess policy. Logstash should sign requests using the logstash-output-amazon-es output plugin. Logstash running on my EC2 instance includes an output section like this:

output {
    amazon_es {
        hosts => ["ELASTICSEARCH_HOST"]
        region => "AWS_REGION"
    }
    # If you need to do some testing & debugging, uncomment this line:
    # stdout { codec => rubydebug }
}

I can access Kibana from the two IPs in the access policy (192.168.1.0 and 192.168.1.1).

Solution 2:

According to AWS doc and as you (and I) just tested, you cannot restrict access to an AWS ES domain to a role/account/user/... and simply cURL it!

Standard clients, such as curl, cannot perform the request signing that is required of identity-based access policies. You must use an IP address-based access policy that allows anonymous access to successfully perform the instructions for this step. (http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg-search.html)

So you have basically two solutions:

• change your access policy and restrict it to IP(s), I think you cannot use private IP because your ES cluster does not seems to belong to your VPC (default or not). Please use the public IP
• sign your request: http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains.html#es-managedomains-signing-service-requests

Signing your request is probably the best solution if you want to keep your access policy as is (which is more flexible than restricting to an IP), but it seems to be a bit more complex. I haven't tried so far and I cannot find any doc to help.