Managing upgrades on hundreds of Debian servers

What do you think are the best practices to maintain dozens (if not hundreds) of debian servers up-to-date ? Keeping in mind that :

There are groups of servers (i.e identical webservers, DB Servers, ...)
There can be several Debian issues (lenny, etch)
Running a loop over all servers and doing apt-get update && upgrade is not acceptable (because it's what I'm doing at the moment :) ) It should be better than this !

Currently, when I finally finish all the upgrades, a new security update is posted, and I have to do it all over again.

Thanks in advance serverfault community !

Solution 1:

I use apt-dater to manage upgrading all my Debian boxes. Seems to do the trick well enough. Haven't tried to scale it up to hundreds of hosts though.

Google solved this with debmarshal:

http://code.google.com/p/debmarshal/

Which lets you approve packages from an upstream repository for installation on your production hosts.

Then you can just run cron-apt in fully automatic mode.

Here's an intro video:

http://www.youtube.com/watch?v=L3hRToC23mQ