Running xserver from console as normal user and gdm3 disabled

Good afternoon to everyone. I am the inexperienced system administrator of a virtual machine with limited hardware resources, and Debian 6 installed. The main use of this machine will be as fileserver/webserver, so I decided that I don't need to waste resources running gnome (and the xserver more in general), and I disabled gdm3 from starting up at boot time. Unfortunately it could be that in the future some users will need a graphic environment, but not so often to justify an xserver running all the time on such a machine. So I thought that they could launch their own xserver when needed. I tried as simple user with startx and I recived:

xauth:  timeout in locking authority file .Xauthority
xauth:  timeout in locking authority file .Xauthority
X: user not authorized to run the X server, aborting.
giving up.

I tried as root and everything worked. I would like to avoid the use of

sudo startx 

for each user, because the session created in this way runs with all root privileges (sudoers is bypassed, even if strictly configured - I tried). I found that the .Xauthority file is missing for each normal user. How could I try to create one for each normal user? I studied a bit. I tried many combination of xauth,

xauth generate .
xauth generate :0 . untrusted
xauth generate :0 . trusted

but without any success. It does nothing or complains about the fact that .Xauthority could not be accessed. The problem is that root is the only one that has an .Xauthority file... so I cannot invoke xauth from a normal user and get this working... . I tried coping it and changing ownership and permissions in the user home but I received always .Xauthority access problem.

Probably I'm not facing the problem with the correct approach. Xauth is not very well documented, and from google I always find solution on how enable X for root from normal users .Xauthority files.

Any suggestion would be more than welcome! Thanks in advance G.

See contents of /etc/X11/Xwrapper.config. Setting called allowed_user

Check this: http://karuppuswamy.com/wordpress/2010/09/26/how-to-fix-x-user-not-authorized-to-run-the-x-server-aborting/

And take sure, that you are member of video group.