What does "pay you out" mean?

Solution 1:

It means ‘take revenge’, as in this quotation from William Thackeray in 1849: ‘You see if I don't pay you out after school—you sneak, you.’ It was still being used well into the twentieth century. It is appropriate in the work of the Rev. Awdry, but seems to be in decline, as this nGram shows (subject to the ususal cautions about nGrams).

Solution 2:

It's a somewhat "dated" form that would normally be expressed today as pay you back, meaning retaliate, settle the score.