Advantages of upgrading 32 bit Windows Server and SQL Server to 64 bit?

Strongly related: Good reasons to keep 32-bit Microsoft Windows desktop OSes

You are using a 64 bit OS. Server 2008 R2 was the first to only support 64 bit CPUs.

"Newer" versions of Windows aren't even designed for 32 bit. You maybe won't take advantage of anything, but there should neither be any disadvantages. That being said: Upgrade anyway, as Server 2008 R2 SP1 (which I hope you are using) will be EOL from 2020-01-14.

As for SQL Server 32 bit/64 bit: Your understanding is correct, if you won't ever need > ~3,75 GB of RAM (or >2 GB per process), you can use the 32 bit version without problems. But for newer versions there won't be any 32 bit version to install, as Microsoft switched to 64 bit only.

As already noted, you are using a 64-bit OS already. There are two advantages of switching to a 64-bit version of SQL Server and one disadvantage.

The sole disadvantage is that the 64-bit version of SQL Server will use 64-bit pointers. This means pointers will occupy twice as much memory, consume twice as much memory bandwidth, and so on. This is likely fairly negligible, but it is a disadvantage. It's partially compensated by the fact that switching to a 64-bit application will allow you to ditch the overhead of the compatibility layer 32-bit apps have to use to access a 64-bit OS's functions.

The major advantage is that numerous significant improvements were made in the CPU instruction set over time. Some of them were made along with the change to 64-bits and some of them were made previously.

But even for the ones made previously, the 32-bit build has to handle CPUs that don't have those features and to avoid the hassle of detection and switching between multiple instances, just doesn't use them even where they're present. For example, 64-bit CPUs must have SSE2, but 32-bit CPUs might not. So most 32-bit code just doesn't bother checking and assumes no SSE2. 64-bit code is assured SSE2 instructions are present and so will use it if it's the best option.

The biggest one is the increase in the number of named, general-purpose registers from 8 to 16. The number of 128-bit XMM registers was also doubled, from 8 to 16.

In addition, a 64-bit process can make use of large amounts of virtual memory. This is especially important with processes that access large amounts of structured data on disk. And, of course, they can use 64-bit integer operations which tend to improve the performance of encryption, compression, and even some filesystem operations on large filesystems.

Fundamentally: Yes. Assuming you never make updates that are then 4 bit only - not sure there even IS a 32 bit SQL Server more recent than 2008.

Issues with your question: "The 64 bit versions allow access to more than 4 GB of RAM," - make that 3gb ;) not 4. 1gb is always reserved.