Secure way to run other people code (sandbox) on my server?

To limit CPU and memory, you want to set limits for groups of processes (POSIX resource limits only apply to individual processes). You can do this using cgroups.

For example, to limit memory start by mounting the memory cgroups filesystem:

# mount cgroup -t cgroup -o memory /cgroups/memory

Then, create a new sub-directory for each group, e.g.

# mkdir /cgroups/memory/my-users

Put the processes you want constrained (process with PID "1234" here) into this group:

# cd /cgroups/memory/my-users
# echo 1234 >> tasks

Set the total memory limit for the group:

# echo 1000000 > memory.limit_in_bytes

If processes in the group fork child processes, they will also be in the group.

The above group sets the resident memory limit (i.e. constrained processes will start to swap rather than using more memory). Other cgroups let you constrain other things, such as CPU time.

You could either put your server process into the group (so that the whole system with all its users fall under the limits) or get the server to put each new session into a new group.

chroot, jail, container, VServer/OpenVZ/etc., are generally more secure than running as an unprivileged user, but lighter-weight than full OS virtualization.

Also, for Java, you might trust the JVM's built-in sandboxing, and for compiling C++, NaCl claims to be able to sandbox x86 code.

But as Checkers' answer states, it's been proven possible to cause malicious damage from almost any "sandbox" in the past, and I would expect more holes to be continually found (and hopefully fixed) in the future. Do you really want to be running untrusted code?

Reading the codepad.org/about page might give you some cool ideas.

http://codepad.org/about