aws --version error: bash: /usr/bin/aws: No such file or directory

I have an AWS CLI install problem that I found no solution for on the Web.

Namely, I installed awscliv2 as guided by the AWS installation docs. After the install, when I enter the command aws --version I get the following error:

bash: /usr/bin/aws: No such file or directory

Yet, when I enter the command using the full path /usr/local/bin/aws --version, it works fine, returning the version information.

The command which aws returns the full path, so just entering aws should work but it does not.

I hacked a solution via a symlink.

In more detail, please observe my command line session below, with comments and, if you can, please answer the question I ask below at the last line:

[2021-03-28 19:04:55 root:~]# #dga- I had downloaded and unzipped the AWS install per:
https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html

[2021-03-28 17:34:53 root:~]# ls -l aws
total 80
drwxr-xr-x 11 root root 4096 Mar 24 21:39 dist
-rwxr-xr-x 1 root root 4047 Mar 24 21:28 install
-rw-r--r-- 1 root root 1465 Mar 24 21:28 README.md
-rw-r--r-- 1 root root 68271 Mar 24 21:28 THIRD_PARTY_LICENSES

[2021-03-28 17:35:00 root:~]# # I installed AWS CLI as instructed in the above AWS link...

[2021-03-28 17:35:00 root:~]# ./aws/install -i /usr/local/aws-cli -b /usr/local/bin
You can now run: /usr/local/bin/aws --version

[2021-03-28 19:09:10 root:~]# which aws
/usr/local/bin/aws

[2021-03-28 19:09:23 root:~]# /usr/local/bin/aws --version
aws-cli/2.1.32 Python/3.8.8 Linux/5.4.0-1041-aws exe/x86_64.ubuntu.18 prompt/off

[2021-03-28 19:09:39 root:~]# aws --version
bash: /usr/bin/aws: No such file or directory

[2021-03-28 19:10:21 root:~]# #dga- WHAT? How is that possible? A mystery.

[2021-03-28 19:11:01 root:~]# ls -l /usr/local/bin/aws
lrwxrwxrwx 1 root root 37 Mar 28 17:35 /usr/local/bin/aws -> /usr/local/aws-cli/v2/current/bin/aws

[2021-03-28 19:13:45 root:~]# ls -l /usr/local/aws-cli/v2/current/bin/aws
lrwxrwxrwx 1 root root 11 Mar 28 17:35 /usr/local/aws-cli/v2/current/bin/aws -> ../dist/aws

[2021-03-28 19:14:03 root:~]# ls -l /usr/local/aws-cli/v2/current/dist/aws
-rwxr-xr-x 1 root root 4414504 Mar 28 17:35 /usr/local/aws-cli/v2/current/dist/aws

[2021-03-28 19:14:31 root:~]# # Note that /usr/local/aws-cli/v2/current is itself a symlink:

[2021-03-28 19:15:07 root:~]# ls -l /usr/local/aws-cli/v2/current
lrwxrwxrwx 1 root root 28 Mar 28 17:35 /usr/local/aws-cli/v2/current -> /usr/local/aws-cli/v2/2.1.32

[2021-03-28 19:15:31 root:~]# # Presumably future updates from AWS will modify the above "current" symlink to a future path of the form /usr/local/aws-cli/v2/X.Y.Z where X.Y.Z > 2.1.32

[2021-03-28 19:21:03 root:~]# ls -l /usr/local/aws-cli/v2/2.1.32
total 8
drwxr-xr-x 2 root root 4096 Mar 28 17:35 bin
drwxr-xr-x 11 root root 4096 Mar 28 17:35 dist

[2021-03-28 19:27:31 root:~]# ls -l /usr/local/aws-cli/v2/2.1.32/dist/aws
-rwxr-xr-x 1 root root 4414504 Mar 28 17:35 /usr/local/aws-cli/v2/2.1.32/dist/aws

[2021-03-28 19:27:58 root:~]# #dga- So the above path is the executable that the "current" symlink references.

[2021-03-28 19:28:53 root:~]# #dga- Here is my total HACK to fix the total bullshit mystery mentioned above...

[2021-03-28 19:29:40 root:~]# #dga- Again, here is what we do NOT want to see happen:

[2021-03-28 19:30:09 root:~]# aws --version
bash: /usr/bin/aws: No such file or directory

[2021-03-28 19:30:24 root:~]# pushd /usr/bin
/usr/bin ~

[2021-03-28 19:31:27 root:/usr/bin]# pwd
/usr/bin

[2021-03-28 19:31:35 root:/usr/bin]# ln -s /usr/local/aws-cli/v2/current/bin/aws

[2021-03-28 19:31:54 root:/usr/bin]# ls -l aws
lrwxrwxrwx 1 root root 37 Mar 28 19:31 aws -> /usr/local/aws-cli/v2/current/bin/aws

[2021-03-28 20:20:13 root:/usr/bin]# #dga- THAT is my hack and it appears to work:

[2021-03-28 20:20:13 root:/usr/bin]# popd
~ /usr/bin

[2021-03-28 20:20:19 root:~]# aws --version
aws-cli/2.1.32 Python/3.8.8 Linux/5.4.0-1041-aws exe/x86_64.ubuntu.18 prompt/off

[2021-03-28 20:21:28 root:~]# #dga- some additional context:

[2021-03-28 20:47:36 root:~]# cat /etc/issue
Ubuntu 18.04.5 LTS \n \l

[2021-03-28 20:47:39 root:~]# uname -a
Linux bamboo.oceanpark.com 5.4.0-1041-aws #43~18.04.1-Ubuntu SMP Sat Mar 20 15:47:52 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

[2021-03-28 20:47:44 root:~]# #dga- Can someone explain to me why the hack was needed?

It turns out the problem was that bash had cached the path /usr/bin/aws from a previous install of an old version 1 of aws cli. Prior to the commands shown in my question, I had deleted the old version of aws which removed /usr/bin/aws. My install of the new version 2 of aws cli installed aws to a different path, /usr/local/bin/aws. The which aws command returned that new path, yet bash still had cached /usr/bin/aws as the path to the aws executable.

So I learned that which <command> can return one path, but bash can resolve <command> to a previous no longer existing path since removing the old path does not update the bash cache! (I never even knew that bash has a cache!)

The bash cache can (and should under these circumstances) be updated via hash aws then verified via hash -t aws.

Also, the AWS CLI install documention should suggest doing hash aws after completing the install of awscli2.

See:

https://unix.stackexchange.com/questions/5609/how-do-i-clear-bashs-cache-of-paths-to-executables