AWS lamba start_instances gives pending response but console still shows stopped

I have a lambda function that I have setup to start an instance:

import boto3
ec2 = boto3.client('ec2')
response = ec2.start_instances(
    InstanceIds=['i-xxx']
)
print(response)

The response looks good, showing it going to pending from stopped:

START RequestId: 26c0cf5e-6d70-4701-b0bd-68276b06d30d Version: $LATEST
{
    "StartingInstances": [
        {
            "CurrentState": {"Code": 0, "Name": "pending"},
            "InstanceId": "i-xxxxxx",
            "PreviousState": {"Code": 80, "Name": "stopped"},
        }
    ],
    "ResponseMetadata": {
        "RequestId": "fdab5818-0536-457f-a19e-17fea60100f4",
        "HTTPStatusCode": 200,
        "HTTPHeaders": {
            "x-amzn-requestid": "fdab5818-0536-457f-a19e-17fea60100f4",
            "content-type": "text/xml;charset=UTF-8",
            "content-length": "579",
            "date": "Wed, 16 Dec 2020 18:38:57 GMT",
            "server": "AmazonEC2",
        },
        "RetryAttempts": 0,
    },
}
END RequestId: f2ed2be9-e2f2-4beb-a69b-4cddee35bef4
REPORT RequestId: f2ed2be9-e2f2-4beb-a69b-4cddee35bef4  Duration: 1381.48 ms    Billed Duration: 1382 ms    Memory Size: 256 MB Max Memory Used: 97 MB  Init Duration: 688.96 ms    

However, when I look at the console, it still shows as stopped and never starts.

It doesnt seem like it failed to start:

                "StateReason": {
                    "Code": "Client.UserInitiatedShutdown",
                    "Message": "Client.UserInitiatedShutdown: User initiated shutdown",
                },

Execution Based Policy:

{
  "Version": "2012-10-17",
  "Id": "default",
  "Statement": [
    {
      "Sid": "xxxxxxx",
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": "lambda:InvokeFunction",
      "Resource": "xxxxxxFunction",
      "Condition": {
        "StringEquals": {
          "AWS:SourceAccount": "xxxxxxxx"
        }
      }
    }
  ]
}

It seems like it didn't really try to start. I've used this code to start other instances. I'm wondering if it is a permissions issue, but there is no error. The lambda function execution role has EC2fullaccess.

Note, another datapoint. I have tried further code that uses ssm send command to send a command once its running (after manually starting). If I try that while it's running it succeeds.

Your EBS volume is encrypted using a KMS key, and the IAM role your Lambda function is using does not have permission to decrypt using that KMS key. In this case, the EC2 instance starts, but then immediately stops.

So there are two possible solutions.

Solution 1: Add your IAM role as a "user" of your KMS key. This requires you to edit the permission policy attached to your KMS key, and add your IAM role as a "user" of the key (to encrypt/decrypt).

Solution 2: Add kms:CreateGrant to your Lambda's IAM role. With this on your IAM role, AWS will allow the EC2 instance to start.