Could someone prove they had a halting oracle?
Solution 1:
It's clear that you cannot possibly hope for more than a probabilistic result -- any testing procedure that can pass a true halting oracle after asking it $n$ questions will also pass a random oracle with probability at least $2^{-n}$.
But I much doubt that you can get even a probabilistic guarantee. Suppose that the purported oracle was in reality a proof-search oracle that answers correctly when the program can be proved (in some fixed but unknown formal system) to either terminate or diverge, and otherwise answers "Halts". In order to conclude that this is not a true halting oracle, we would need to ask it about something that diverges, but unprovably so. We could ask it to search for a proof of an undecidable Gödel sentence, but since the formal system the fake oracle uses is unknown, it could just have added a finite number of Gödel sentences to its static knowledge.