What is the similar meaning proverb in English?

We have a proverb in Bengali, if I translate it directly into English, it emerges as:

Who is in there in the temple? I did not eat the banana!

Meaning in the temple banana is used for prayer to the God. So stealing banana from temple is very bad. But the person who steals always wants to pretend that he is not responsible for that. But he has the fear in his heart. So if anybody enquires anybody, other than him, as to who has stolen the banana, the thief speaks the first(albeit he is not asked!) that he did not do anything indicating he is the thief! The whole proverb, in a short, signifies that who has done the wrong, speaks the first when asked to others, revealing he is the doer. So what might be the proverb in English or expression to express the same meaning?

Here is a related line from Shakespeare that has become idiomatic:

The lady doth protest too much, methinks.

It comes from Hamlet and you can read all about it here. As the Wikipedia article suggests, it has come to mean that one can "insist so passionately about something not being true that people suspect the opposite of what one is saying."

I have heard it in many forms, such as "methinks the lady doth protest too much," "the lady protesteth too much," or more simply "she/he protests too much."

The following proverb is a close enough fit to the situation described by the OP:

He who excuses himself, accuses himself.

The below proverb means that, a guilty person knows that he is guilty and no one needs to tell him that.

A guilty conscience needs no accuser.

Not a proverb, but I'm thinking of "The Tell-Tale Heart" by Edgar Allen Poe, a short story about a guilty conscience leading someone to confess.