Can't run AWS CLI from CRON (credentials)

Trying to run a simple AWS CLI backup script. It loops through lines in an include file, backs those paths up to S3, and dumps output to a log file. When I run this command directly, it runs without any error. When I run it through CRON I get an "Unable to locate credentials" error in my output log.

The shell script:

AWS_CONFIG_FILE="~/.aws/config"

while read p; do
 /usr/local/bin/aws s3 cp $p s3://PATH/TO/BUCKET --recursive >> /PATH/TO/LOG 2>&1
done </PATH/TO/INCLUDE/include.txt

I only added the line to the config file after I started seeing the error, thinking this might fix it (even though I'm pretty sure that's where AWS looks by default).

Shell script is running as root. I can see the AWS config file at the specified location. And it all looks good to me (like I said, it runs fine outside of CRON).

If it works when you run it directly but not from cron there is probably something different in the environment. You can save your environment interactively by doing

set | sort > env.interactive

And do the same thing in your script

set | sort > /tmp/env.cron

And then diff /tmp/env.cron env.interactive and see what matters. Things like PATH are the most likely culprits.

When you run a job from crontab, your $HOME environment variable is /

The Amazon client looks for either

~/.aws/config

or

~/.aws/credentials

If $HOME = /, then the client won't find those files

To make it work, update your script so that it exports an actual home directory for $HOME

export HOME=/root

and then put a config or credentials files in

/root/.aws/

I was able to solve this issue through the following:

export AWS_CONFIG_FILE="/root/.aws/config"
export AWS_ACCESS_KEY_ID=XXXX
export AWS_SECRET_ACCESS_KEY=YYYY

Put this code before your command line to be executed into crontab -e

SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin