How can Steam be prevented from downloading a billion copies of DirectX?

The problem has been discussed on the Steam Forums. The problem is that each game has to have a binary-identical version of the vc++ redistributables. Since games link against different versions of these libraries, steam always installs them the first time as a safeguard.

Credit to mordi2k for finding the post
Here's a link to the Steam Support Article about it (forum post is no longer available). The text is quoted below.

Why does DirectX install with every game?

Running the DirectX installer is not a matter of making sure your overall DirectX install being up-to-date. Microsoft has a helper library with D3D called D3DX. You'll find binaries for this like d3dx9_43.dll in your Windows\system32 folder. There are over 40 different versions of the D3DX library for D3D9 alone, and many more for D3D10 and 11 as well. Each game that uses the D3DX helper library is linked to a specific version. As such the game must run the correct D3D installer version that it was specifically compiled with to ensure the binaries exist.

Even if a later version of the binary is already installed, that version cannot be used, and even if your DirectX install is up-to-date because you've run a more recent version of the installer that is not guaranteed to have installed all previous versions. Even worse, if a version is installed for x86 it doesn't guarantee the same version is installed for x64, so 64 bit and 32 bit games may need to run the same exact installer version but targeting different platforms when run.

Furthermore, Microsoft's licensing terms prevent anyone from distributing the files directly, the only way to distribute them is to run the installer, that's also the only supported method from Microsoft to check that the correct version installed. Trying to manually check for the correct versions is extremely complicated because there are numerous files that must all be present and individual system configuration options like dll search paths complicate the situation. In addition, the dependencies and required checks may change in each new version of the D3DX runtime. The code to check correctly and repair broken installs all exists in the installer and running it is a guarantee that the correct binaries will exist when you run the game and prevents lots of bad cases where a game would fail to launch with an obscure error if a windows install was either missing the correct version or somehow corrupted in the past.

Why don't Valve's Source Engine games run the DirectX Installer?

Games which don't use the D3DX helpers (such as Source engine games) don't require running the installer on first launch as they only depend on major d3d9/10/11 versions being installed. However, games that do use D3DX must run it as it's the only way Microsoft has allowed for distributing and checking the version info on the files.

Since August 2014, there is now the Steamworks Common Redistributables package, that replaces the old per game DirectX installer system.

The content of this package will be downloaded depending on the games that require VC++, DirectX, XNA, .NET, OpenAL or PhisX, and then later saved to C:\Program Files (x86)\Steam\steamapps\common\Steamworks Shared\_CommonRedist for them to be used in the future.

So far, only the following installers are available on the Common package, meaning that other games that require other programs or versions will still need to keep the redistributables per game copy:

• VC 2005 Redist (5.61 MiB)
• VC 2008 Redist (9.24 MiB)
• VC 2010 Redist (18.37 MiB)
• VC 2012 Redist (13.11 MiB)
• VC 2013 Redist (13.06 MiB)
• VC 2015 Redist (28.38 MiB)
• VC 2017 Redist (28.29 MiB)
• VC 2019 Redist (28.06 MiB)
• DirectX Jun 2010 Redist (98.16 MiB)
• .NET 3.5 Redist (231.50 MiB)
• .NET 3.5 Client Profile Redist (255.55 MiB)
• .NET 4.0 Redist (48.11 MiB)
• .NET 4.0 Client Profile Redist (41.01 MiB)
• .NET 4.5.2 Redist (66.76 MiB)
• .NET 4.6 Redist (59.14 MiB)
• .NET 4.7 Redist (80.06 MiB)
• .NET 4.8 Redis (111.94 MiB)
• XNA 3.0 Redist (7.61 MiB)
• XNA 3.1 Redist (7.32 MiB)
• XNA 4.0 Redist (6.73 MiB)
• OpenAL 2.0.7.0 Redist (791.10 KiB)
• PhysX System Software 8.09.04 (49.39 MiB)
• PhysX System Software 9.12.1031 (25.49 MiB)
• PhysX System Software 9.13.1220 (39.27 MiB)
• PhysX System Software 9.14.0702 (57.26 MiB)