Where is $BASH_ENV usually set?

I have twin Linux servers that should be configured identically, however ssh commands to one of them are failing for commands that require a path that's specified in ~/.bashrc. For example, I can use a command like pwd both interactively and through ssh, but if I try running a program that's located in an application bin folder, it only works in an interactive shell for one of the servers.

The /etc/profile and /etc/environment file on both servers are identical, however $BASH_ENV is set to ~/.bashrc on the server that is working properly. I want to set $BASH_ENV on the server that's not working, but I'd prefer to set it in the same location as it's set in the working server. What are the places that Linux will run at the time of a non-interactive login, such as an ssh command from another computer?

edit: The line in /etc/passwd for the user specifies /bin/bash on both servers. The ~/.bash_profile file for both servers is identical, and contains if [ -f ~/.bashrc ]; then . ~/.bashrc; fi. The only difference between the systems is that $BASH_ENV is a null string on the server that's not working, and I can't find where it's been set in the server that's working.

edit 2: The ~/.ssh/environment file on both servers has BASH_ENV=~/.bashrc

BASH_ENV is only going to be set via the environment, or another script that is sourced during initialization. For a non-interactive shell, it will only be trying to source additional files if that shell is also a login shell. (in which case it'll read ~/.bash_profile, ~/.bash_login, and ~/.profile...but if it was doing that, you wouldn't be experiencing an issue)

The first place to look is the environment in which the subshell is being invoked.

• An exported BASH_ENV variable will be passed through. Keep in mind that this may be buried in a sourced file.
• It can be fed in as a parameter on the same line calling the script, i.e. BASH_ENV=blah /path/to/somecommand.sh. This stands out like a sore thumb so you probably would have caught it.

If it's being set after you log in but you can't figure out where, you may need to look at what is responsible for constructing the login environment.

• All of the usual files that get sourced by a login shell. man bash for the exhaustive list.

• PAM: As freiheit suggested in the comments, check /etc/security/pam_env.conf, and any additional files that are referenced by pam_env.so. Other PAM modules could also be responsible, but if your PAM configs look identical this is probably not the case.

• sshd: It will scan the following files, in order:

  • ~/.ssh/environment (before changing to the home directory; only if PermitUserEnvironment is enabled in sshd_config)
  • ~/.ssh/rc (after changing to the home directory; always)
  • /etc/ssh/sshrc (if ~/.ssh/rc is not present)

Note: sshd will also scan for environment=value lines in the user's authorized keys file (if PermitUserEnvironment is enabled), but it is not clear from the man page where that step falls in the above sequence.