limits.conf to set memory limits

I would like to limit any process from using more than 500 MB of RAM. AFAIK this is done using RSS in /etc/security/limits.conf but the process called gnome-panel apparently is using 618436 kB of VmRSS. How can this be ?

/etc/security/limits.conf

*               hard    rss             512000

username@debian:~$ cat /proc/3002/status
Name: gnome-panel
State: S (sleeping)
Tgid: 3002
Pid: 3002
PPid: 2910
TracerPid: 0
Uid: 1000 1000 1000 1000
Gid: 1000 1000 1000 1000
FDSize: 64
Groups: 20 24 25 29 44 46 112 116 117 1000 1002 1003 
VmPeak:   916636 kB
VmSize:   916636 kB
VmLck:        0 kB
VmHWM:   618436 kB
VmRSS:   618436 kB
VmData:   601972 kB
VmStk:      104 kB
VmExe:      516 kB
VmLib:    29232 kB
VmPTE:     1760 kB
Threads: 1
SigQ: 0/14001
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000020001000
SigCgt: 0000000180000000
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: ffffffffffffffff
Cpus_allowed: 3
Cpus_allowed_list: 0-1
Mems_allowed: 00000000,00000001
Mems_allowed_list: 0
voluntary_ctxt_switches: 871965
nonvoluntary_ctxt_switches: 47553
PaX: PeMRs

username@debian:~$ cat /proc/3002/limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    0                    bytes     
Max resident set          524288000            524288000            bytes     
Max processes             100                  100                  processes 
Max open files            1024                 1024                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       14001                14001                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us

Found this solution after looking here: http://shortrecipes.blogspot.de/2009/04/limitsconf-virtual-memory-limit.html

For your example:

* hard as 512000

This seems to be equivalient to ulimit -v 512000, applied the moment you log into your account.

limits.conf(5) man page says:

rss
  maximum resident set size (KB) (Ignored in Linux 2.4.30 and higher)

I don't know of a way to limit this on a per-process basis, but you can restrict the total RSS (and other) size of a group of processes with the memory resource controller.

RSS limit in limits.conf won't work in kernels > 2.4.30.

You must use Cgroups