Why do I get "ls: Desktop: Operation not permitted" when I own Desktop?

I'm on Big Sur, and I understand there have been some changes with permissions in macOS since Catalina. I don't understand, however, how it is possible that I cannot ls ~/Desktop when ls -l ~ shows drwx------+ for permissions and me as the owner. Could someone explain this?

Edit: to be clear, I am not asking how to access this directory, but how it is possible that I cannot, given how Unix permissions work.

[This answer is about how it is possible to get a permissions error accessing a file, when the file permissions allow access.]

Recent versions of macOS have additional security layers that are independent of the normal unix-style access permissions. They (mostly) regulate access based on the identity of the program, rather than the user ID the program is running as. The new layers are System Integrity Protection (SIP), which regulates access to the core of the operating system, and Transparency, Consent and Control (TCC), which regulates access to "personal" information (as well as things like the camera and microphone).

Access can be vetoed by either these layers or the standard unix layers. That is, if SIP and TCC allow something but the standard unix permissions forbid it, it's forbidden. And if the standard unix permissions allow something, but SIP or TCC forbids it (as in your example), it's also forbidden.

These new security layers are necessary because we can't trust our software. Even if you set aside malware, experience is showing that many "legitimate" developers would love to spy on their customers for valuable personal data that we'd really rather they not have. So just because I'm running some program doesn't mean that program should have access to everything I have access to (e.g. my emails and contacts, location, camera, etc...).

Testing on a clean install of macOS Big Sur, in Terminal when I executed ls ~/Desktop I was presented with this dialog box:

I clicked: OK

I then executed ls ~/Downloads and was presented with this dialog box:

I clicked: Don't Allow

Executing ls ~Downloads in Terminal again, it returned: ls: Downloads: operation not permitted

I then went to System Preferences > Security & Privacy > Privacy > Files and Folders and had the following:

As you can see what I said OK to is checked and what I said Don't Allow is unchecked.

So, check to see that System Preferences > Security & Privacy > Privacy > Files and Folders > Terminal > Desktop is checked.

Notes:

If you do not want to be prompted each time on protected folders, then add Terminal to: System Preferences > Security & Privacy > Privacy > Full Disk Access

In doing so, Files and Folders will show Terminal as having Full Disk Access instead of what's shown in the screenshot above.

If your uncheck Terminal in Full Disk Access, then whatever applications previously were under Files and Folders > Terminal will reappear with the same settings as last set.