Do pillager patrols despawn?
After some research, I could find no documented special despawn rules for pillager patrols. According to despawn mechanics, any mob 32 or more blocks away from the player after 30 seconds has a 2.5% chance per second to despawn at random.
What may have happened is that you moved far enough away before the despawn time could occur that the chunk had become unloaded. Mobs in an unloaded chunk do not despawn and are saved along with everything else within it.
If you are adamant on not fighting a patrol because you are ill-equipped, you should put yourself at a great distance just far enough that you can still see them but are definitely more than 32 blocks away and wait for a while.
If you are adamant on not fighting a patrol because you do not want to suffer the effects of "Bad Omen" you can avoid getting the effect by either drinking milk after receiving the effect or by killing the banner-carrying pillager in a way that will not be attributed to you, such as dumping a bucket of lava underneath their feet.