What is the scientific term for a habit of doing something without realizing it until someone points it out to you?

Are you thinking of mannerisms?

a particular habit or way of speaking or behaving that somebody has but is not aware of

He has this irritating mannerism of constantly scratching his nose.

[Oxford]

Detectives and poker players refer to such behaviors as tells, while lay people might call them nervous habits.

Are you thinking of "tic" ?

A tic is a sudden, repetitive, nonrhythmic motor movement or vocalization involving discrete muscle groups.

Or maybe "OCD", "Obsessive–compulsive disorder" :

mental disorder where people feel the need to

• check things repeatedly,
• have certain thoughts repeatedly,
• and feel they need to perform certain routines repeatedly.

Just for your information, in french, "OCD" is abbreviated "TOC" and is often mixed up with "tic".