What are the rationales behind that Java does not overwrite / remove its older updates after each update?

Can I uninstall the older updates?

Regarding updates, I am referring to those Update 1, Update 2, e.g. Java 7 Update 1, Java 7 Update 2, and so on.

I am speaking with respect to Java for Windows.

Should I uninstall older versions of Java?

We highly recommend that you uninstall all older versions of Java from your system. Keeping old versions of Java on your system presents a serious security risk. Uninstalling older versions of Java from your system ensures that Java applications will run with the latest security and performance improvements on your system.

_{Source: Why should I uninstall older versions of Java from my system?}

The reason that the update doesn't remove the older version is to allow users to install more than one version of Java at the same time and then select different available Java versions for different applications, even though this is not recommended in Windows. See Oracle Java™ Runtime Environment Version Selection.

For uninstalling the older Java versions or updates, Oracle has released a separate application for uninstalling out-of-date versions of Java called Java Uninstall Tool for Windows. The Java Uninstall Tool for Windows runs with Java 6u10 and later versions on Windows XP and later up to Windows 8.

In rare cases a newer version of Java is NOT compatible with your existing software.
(Java sometimes removes certain features when going to a new version. And, more often the case, some functions get a slightly different behavior, which breaks some applications that relied on the old behavior.)

This mostly happens with version upgrades: Like version 1.6 to version 1.7, but it can happen in very rare cases with a minor update like 1.7.1 - 1.7.2 as well.
And sometimes certain stupid software just insists on a specific Java version being present even though it will work flawlessly with a new version.
(I have seen software that checked for the presence of a specific Java folder belonging to version 1.5.x, but it would happily work on 1.6.x as long as the folder existed. There weren't even any files in that folder.)

Because of all this it must be possible to run 2 or more versions in parallel and removing the old version isn't default.

P.S. Applications that really rely on a specific version of the Java run-time should use a private copy of that specific version inside their own program environment, in stead of using the version installed at the OS level. (This embedded copy will never be upgraded, except via an upgrade of the application that uses it.)
This is the recommended way of doing this, but not all software plays by the rules.