Is there difference if you are running 32bit Linux on low end VPS instead of 64bit in memory consumption?

It really depends on what you're running on that Linux box. Generally speaking, yes, 32-bit instance will take less RAM to do the same job as 64-bit instance. However, there are two nuances:

  1. In some applications the difference is more pronounced than in others. For example, if it is a Ruby on Rails web application, difference is really big (from my experience).
  2. You can have a basic 64-bit system and run 32-bit apps on top of it (with the aid of proper libraries). Depending on your distribution, many of these libraries could already be installed for you.

32-bit memory, pointers, instructions, etc, are all smaller than their 64-bit counterparts. The 64-bit overhead gains big advantages when crunching lots of data or accessing lots of memory (e.g. 32-bit registers can't access over 3.6 gigs of RAM.) But when you're talking about sub-Gig memory the 32-bit OS starts to sound good. I've seen differences in a bare-install boot where a 64-bit OS took an extra 100+ megs of RAM. If you're running on a 512 meg VPS that's going to kill your performance quickly.