How can I stop pillagers from spawning?

I have a little lake next to my house in my survival world, but every other day a group of pillagers including a patrol captain are just resting there so now I have like 20 illiger banners. I'm tired of having to kill them off all the time. Is there anything I can do? Why is this happening?

Adding on to @Hari5000's answer, you could mob-proof the area around your base, by putting carpets or slabs surrounding the home. That way, pillagers and other mobs won't spawn around your base, though the area beyond the mob-proof area may spawn patrols or other hostile mobs.

From the Minecraft Gamepedia on Patrols:

In Bedrock Edition patrols spawn around 24–48 blocks away from the player (or more than 48 blocks away from the player if they are in a village) in any biomes except in mushroom fields, mushroom field shore, nether and end biomes. Additionally, patrols can spawn in biomes where common monsters cannot spawn such as in:

• river
• frozen river
• snowy tundra
• snowy mountains
• legacy frozen ocean

Patrols spawn on solid blocks at the surface, like monsters. On Easy difficulty, patrols can spawn only at light level 0–7 on the surface. On Normal and Hard difficulty, patrols can spawn in any light level.

Patrols spawn naturally after the world age reaches 100 minutes (five in-game days), then after a delay between 10–11 minutes an attempt is made to spawn a patrol (20% chance of success), repeating the delay and spawn attempts.

If you want to find some peace, either find yourself a lovely mushroom biome, or spend more time exploring the Nether or the oceans.