Solution 1:

Is it possible that the serverSocket is being closed from another thread? That will cause this exception.

Solution 2:

This is my example solution to avoid WSAcancelblablabla: Define your thread as global then you can use invoke method like this:

private void closinginvoker(string dummy)
    {
        if (InvokeRequired)
        {
            this.Invoke(new Action<string>(closinginvoker), new object[] { dummy });
            return;
        }
        t_listen.Abort();
        client_flag = true;
        c_idle.Close();
        listener1.Stop();
    }

After you invoke it, close the thread first then the forever loop flag so it block further waiting (if you have it), then close tcpclient then stop the listener.

Solution 3:

This could happen on a serverSocket.Stop(). Which I called whenever Dispose was called.

Here is how my exception handling for the listen thread looked like:

try
{
    //...
}
catch (SocketException socketEx)
{    
    if (_disposed)
        ar.SetAsCompleted(null, false); //exception because listener stopped (disposed), ignore exception
    else
        ar.SetAsCompleted(socketEx, false);
}

Now what happened was, every so often the exception would occur before _disposed was set to true. So the solution for me was to make everything thread safe.