Windows desktop virutalization instead of replacing work stations
Solution 1:
This type of solution exists in a continuum.
On one end of the spectrum you have client computers running a "thick" operating system (like Windows or a desktop Linux distribution) and connecting via client software to hosted applications (via RemoteApp shortcuts and the Remote Desktop Protocol (RDP), or via Citrix ICA protocol).
In the middle of the spectrum you have clients connecting via these same protocols to full-blown desktop sessions (rather than a single application), but using a shared operating system installation. This is typically the world of Windows "Terminal Services".
On the far end of the spectrum you have what's typically known as a Virtual Desktop Infrastructure (VDI) where client devices are very stripped down and only host client software to connect to a hosted operating system instance.
All of these situations are physically feasible, but you'd do yourself a favor to start investigating the licensing costs before you go down the road of spec'ing servers, etc.
The licensing costs in the Microsoft world include either Terminal Services Client Access Licenses or Windows Virtual Enterprise Centralized Desktop (VECD) licenses of operating systems to contend with for each device or user accessing the VDI solution. Licensing for your desktop application software, depending on where on the spectrum you're falling, may also be different than you currently use and this necessitate additional license purchases.
It's likely that you're going to find that the acquisition costs of a VDI infrastructure are similiar, if not more expensive, than going down the traditional "thick client" route. Phyisically and pratically using thin-client devices sounds like a "win", but software licensing expense has traditionally more than made up for any hardware cost savings, which leaves only "soft cost" management and TCO savings as justification.
Edit:
Ryan Bolger hit it right on the head with his answer (and I +1'd him) with respect to "soft cost" savings, which you're right to identify as the place to save money.
Learning how to centrally deploy software, manage user environments, and generally maintain the hell out of your network using Group Policy will build your personal knowledge of the "innards" and operation of a Windows network and will have far fewer "moving parts" than a VDI infrastructure. Even if you had a VDI infrastructure, frankly, I think you'd still be able to leverage immense benefits from Group Policy-fu.
VDI and remote application delivery is a great solution for very task-specific application, or delivery of applications over slow or unreliable network connections (think "shared Microsoft Access database over a T1-based WAN"). I don't think that desktop virtualization, at least in the current incarnation as an excessive-licensing-fee-based minefield, is "the answer".
I'll even jump out on a limb and say that, with proper "care and feeding" maintenance of very large fleets of client computers running Windows isn't really all that hard, using the built-in tools in Windows Server, WSUS, good knowledge of scripting, and an understanding of how Windows itself and your application software works. Automating your client computer build, removing users' Administrator rights, and getting a handle on your OS and application update deployment infrastructure will take you leaps and bounds ahead.
Solution 2:
I'd like to build a bit off of Evan's answer regarding the different ways to remotely host applications.
Your primary concern seems to be about reducing the administrative overhead involved with managing a bunch of disparate workstations and their individual software installations. You don't need to move to a remotely hosted application infrastructure to accomplish that goal.
With a single server setup as a domain controller and all of your workstations joined to that domain, you can do just about everything you need right out of the box. The domain itself handles centrally configured user accounts. Group Policy can handle configuring all of the system settings on the workstations. And Group Policy Software Deployment can handle your application installations. The built-in Windows Deployment Services combined with the free Microsoft Deployment Toolkit can even give you your OS deployment solution. WSUS is also free and can handle your OS and Microsoft software patching.
There's just a ton of stuff you can do with nothing more than a single server OS license and your workstation OS licenses. It all has a bit of learning curve, but it's no more difficult than the things you'll have to learn with a remotely hosted app or OS solution.