What is the correct way to kill a vncsession in linux?

Solution 1:

As you noticed, from the man vncserver:

  -kill :display#
          This kills a VNC desktop previously started with vncserver. It does
          this by killing the Xvnc process, whose process ID is stored in the
          file "$HOME/.vnc/host:display#.pid". It actually ignores anything
          preceding a ":" in its argument. This can be useful so you can write
          "vncserver -kill $DISPLAY", for example at the end of your xstartup
          file after a particular application exits.

The display number is connected to the port number of the display if one hasn't set that manually (and differently), where

Display number = (Port number) ‒ 5900

e.g. port 5901 → display :1. This information can be found in man Xvnc (vncserver is just a wrapper script that calls this tool) where it says:

   -rfbport port
          Specifies the TCP port on which Xvnc listens for connections from
          viewers (the protocol used in VNC is called RFB - "remote
          framebuffer").  The default is 5900 plus the display number.

If you don't know the number by heart (but you need to know it if you are going to connect to the server anyway), you can check e.g. ps ax | grep vnc for information. If I do that locally, I see the processes

25697 ?        S     55:38 Xvnc4 :1 [...]
[...]
30481 ?        S     17:57 Xvnc4 :2 [...]

and thus I know that they represent VNC servers with display numbers :1 and :2 respectively, and can be killed by

vncserver -kill :1
vncserver -kill :2

In your case, you see that the display number is :1 for the server listed in your ps output.

Solution 2:

I tried the answer above and it didn't work for me. It gave me an error message as in this question: Killing VNC Process Manually

So I had to kill them manually. I tried kill -9, and then I couldn't log in with rdp anymore. I got xrdp_mm_process_login_response: login failed when I tried to log in.

The answer was found here: http://linuxtoolkit.blogspot.com/2013/03/xrdpmmprocessloginresponse-login-failed.html

Basically, there is a session file not cleaned up when the Xvnc server is killed. The file is named for the display, so if you're on display :12, it's /tmp/.X11-unix/X12. Delete that file after kill -9 and you're back in business.