Enderman won't leave my house but didn't attack me?

Solution 1:

Definitely a glitch. Looking at the Enderman's head, or attacking him enrages him. Sometimes the game takes a couple seconds to trigger the aggro mode, but rarely more. Normally Endermen are peaceful, unless you trigger their aggro by the above methods; they can teleport in a rather wide range. They also can't fit into a 2-tall gap, which makes killing them from within a 2-tall shelter quite easy (fighting them in open field is unadvisaed; they hit hard and fast; they are also hurt by water so spilling a water bucket under your feet allows to form a defensive position if you're caught by one in the open.)

They can pick up and place a certain set of blocks (dirt, sand and their variants, netherrack and a couple more); When killed, in 50% cases they drop an Ender Pearl (more when killed with Looting enchanted sword); the Ender Pearls are essential in finding and opening the End portal - so if you plan to reach the endgame content (obtain Elytra and Shulkerboxes) you will have to kill quite a few of them.