why is sftp rmdir not working?

In my experience, rmdir prefers to work on an empty directory. If you're trying to delete the directory foo, I would do:

$rm foo/*
$rmdir foo

You have not specified, what SFTP client you are using. So I'm assuming OpenSSH SFTP (sftp).

Command rmdir in OpenSSH SFTP client maps directly to SSH_FXP_RMDIR SFTP protocol request. The SFTP spec for version 3 (the one used by OpenSSH) specifically mentions that the SSH_FXP_RMDIR operation may fail, "if the specified directory is not empty" (though it does not seem to mandate it).

If the directory does not have subdirectories, you can use rm foo/* (meaning OpenSSH SFTP command, not shell command) to remove all the files in the directory first. And then use rmdir.

For more complex cases, you will need a smarter SFTP client.
Or if you have a shell access, use rm -r * in the shell.