Best way to go about changing a root password on 3000+ Solaris, AIX, and Linux servers?

Long story short: Big old corporation, lots of UNIX/Linux servers.

I inherited responsibility for a bunch of scripts that that left a few years ago. One of them was a script that would be run every $X amount of months to globally update the root password across all of our servers.

The script is a mess of Shell Script and Expect, and it works on the SSH trust that is set up between all of our servers and a central command-and-control server.

The problem is, the script is a giant mess. The Expect commands are trying to account for every possible version of "passwd" that exists on any UNIX/Linux box out there - and they vary quite a bit.

As we're expanding and upgrading a lot of our infrastructure, the script is getting really unmanageable.

My question is: Is there a better way to do this? Assuming there's an already established SSH trust, what's the best way to change the root password on 3000+ servers at the same time?

Use Puppet.

Puppet is very flexible, easy to maintain and uses SSL. Might sound a bit overkill and you'll have to put some extra effort to build Puppet system up.

But. Most probably this is not the last mass-update you'll be doing to these machines. Puppet will and does save you a lot of time when actual whatever mass-update procedure begins and scripts are very readable/reusable.

At least this worked for me few years back and still I'm able to re-use some of those Puppet recipes (aka scripts). I have also used it in a bit smaller environments just make sure that every machine is actually having a known state.

I have proven it many times (at many companies) that all customised deployment scripts become pain in the butt after a while or when that next guy steps in. And as long as you carry a cellphone, old scripts will and do haunt you.

If you think that this actually sounds good, here's a great Puppet tutorial with virtual environment included to get you started.

I've used the Perl module Authen::PAM on Solaris with great success. Here's a sample script:

#!/usr/bin/perl

use Authen::PAM;

my $username = 'root';
my $password = '1234567';

die qq{Error: Unknown user\n} unless getpwnam($username);

die qq{Error: You must run this as root.\n} unless ($> == 0);

my $pamh;

sub my_conv_func
{
    my @res;
    while ( @_ )
    {
        my $code = shift;
        my $msg = shift;
        my $ans = "";

        if ($code == PAM_PROMPT_ECHO_OFF() )
        {
            if (($msg =~ /^New Password:/i) or ($msg =~ /^Re-enter new Password:/i))
            {
                $ans = $password;
            }
            else
            {
                die qq{Unknown message: $msg\n};
            }
        }
        else
        {
            print qq{$msg\n};
        }

        push @res, (PAM_SUCCESS(), $ans);
    }
    push @res, PAM_SUCCESS();

    return @res;
}

ref($pamh = new Authen::PAM("passwd", $username, \&my_conv_func)) || die "Error code $pamh during PAM init!";

my $res = $pamh->pam_chauthtok;

print $pamh->pam_strerror($res),"\n" unless $res == PAM_SUCCESS();

exit 0;

If you can write Perl, the module Net::OpenSSH::Parallel allows to write scripts that perform actions in parallel on remote hosts via SSH quite easyly.

It contains a sample script for changing passwords that you can use as a base. As it seems that you have an heterogeneous environment you would like to group the hosts by type and use a different dialogue handling sub for every one.

I don't know about "best", and whether it's possible for all of the non-Linux *nix machines in your mix, but have you looked at puppet or cfengine for this kind of activity?

There are also commercial (highly expensive) tools for identity management out there, two I've seen/used in the past are Oracle Identity Manager and the Novel equivalent.

After continuing to research this, I've learned a few things...

First and foremost, this is a really annoying task to automate, especially across many different environments. The most correct answer to this question is probably @tomi's: use Puppet.

Eventually I hope to get Puppet to manage out infrastructure, but deploying to the entire enterprise's UNIX servers for a root password change right now isn't that feasible of an option.

After reading many manpages and lots of Google-fu, I managed to come up with a script that loops through a list of target servers, opens an SSH connection, and runs one of the following:

# Solaris
# Generate new pass via crypt(newpass,salt) and insert it into /etc/shadow

# AIX
$ echo "root:newpass" | chpasswd -f NOCHECK

# Linux
$ echo "newpass" | passwd root --stdin

# IBM VIO (Virtual I/O)
$ echo "echo \"padmin:newpass\" | chpasswd -f NOCHECK" | oem_setup_env

# IBM HMCs (Hardware Management Consoles)
$ chhmcusr -u hscroot -t passwd -v "newpass"

It does quite a bit more than running just those commands, but those above are what work the magic.

I couldn't find any simple way to non-iteratively change the password on Solaris - so we resorted to modifying /etc/shadow on the fly.