Why is villager trying to go to the closest village which is more than 35 blocks away?

I have a new potato farm, so I moved the villagers to it, I also have a villager farm on the other side of my house (about 35 blocks away from the potato farm). But the farmer doesn't care about the potatoes, he keeps trying to get to the village I am using for breeding? According to the wiki, they will forget the village in like 6 seconds or so as long as they are 32 blocks away from it.

A villager taken more than 32 blocks away from its village boundary will forget the village within about 6 seconds. Whether in a village or not, a villager is never prone to despawning."

UPDATE: We can assume that the "village boundary" is not the doors them selves but an area around them, which makes the distance less than 32 blocks and therefore the villager is right to keep trying to go back.

Solved by giving the potato farm its own doors to simulate a closer village.

Notice how your quote says: "more than 32 blocks away from its village boundary ". A village is by default defined as a 32 radius sphere(can be more in certain situations), centered around the center of the village. The center is the average coordinate of all doors.

Your potato farmer is 35 blocks away from your breeder. Do 35 - 32 (subtract the radius of the bounding sphere) and you'll see that your potato farmer is only 3 blocks away from the boundary. The villager must be at least 64 blocks away from your village (radius + distance to forget the village) to have it forget the village.

I wouldn't give the potato farmer its own doors, which was suggested by Denislav Karagiozov. This might cause the village of your breeder to merge with the village of your potato farmer. The village center gets moved and might break your breeder. Instead, I would suggest you to just move your potato farm or your breeder.