"you would not be prepared to swear to me" Swear what?

Solution 1:

This is an unusual use — or now an unusual use — of swear to:

swear to phrasal verb

2. To affirm with an oath; to express assurance of the truth of (a statement), or the identity of (a person or thing), by swearing.

1908 R. Bᴀɢᴏᴛ Anthony Cuthbert xxiv. 315 You could swear to its authenticity, or the reverse, if necessary?

— OED

That is, just as swear to its authenticity means "to make an oath that it is authentic"; this use of swear to me means "make an oath that I am me."

It doesn't indicate that the speaker is administering an oath (as one might swear to a magistrate to do something).