What is the maximum memory a process (MySQL) can consume on a 32-bit OS?

It depends on your kernel and configuration. It is usually either 2Gb or 3Gb.

The application itself may have a self-imposed lower limit though. Some 32bti apps/services won't use more than 2G even if the OS would let them because their own internal memory management uses (or allows for) signed 32 bit integers for everything.

If your CPU, memory controller, OS and application all support PAE then that would allow more RAM to be used by the single process. PAE is supported by the Linux kernel and there are no doubt pre-build kernels in the RHEL repositories that have it enabled (it is not enabled in most default kernels as there are performance implications) - I don't know if MySQL will have support for it though.

If you are using an older kernel it might be worth looking into the memory map configuration - while most new kernels default to a 1G/3G split between kernel and userland mapped address space within each process, in older times (when machines with RAM close to the limit of a 32bit address space were practically unheard of) a 2G/2G split was the default but could be changed if you needed 1G/3G.

Without digging deep into the subject, my guess would be that (unless you are using an older kernel and that is what is imposing the 2G limit rather than a 3G one) MySQL will not use more then 2G without the upgrade to the 64-bit compile. The extra RAM is unlikely to be wasted though - the kernel will use it for cache/buffers so while MySQL using it directly would be much more efficient (as there would be many less cases of MySQL's cache and the OS cache both holding the same blocks of data) it won't be standing idle so some benefit will be seen.

Most modern distributions and processors support PAE, which is Intel's instruction set allowing memory addressing beyond the 32-bit limitation. PAE allows a maximum of 64GB of RAM to be accessed by the processor. PAE has additional CPU overhead.

32-bit Linux has a per process limit of 3GB addressable even with PAE. Typically, the kernel reserves approximately 1GB.

MySQL is a single process daemon. As such, it will be unable to address outside of the 32-bit limitation even with PAE.

Some useful references:

• PAE Wiki
• LinuxRamLimits
• MySQL Memory Allocation