A question on elementary number theory
By the paper A. Dujella and C. Fuchs, Complete solution of a problem of Diophantus and Euler, J. London Math. Soc. 71 (2005), 33-52. (see Theorem 1b), there does not exist 4-element set the considered property and with all elements greater than 1.
For results on sets contaning 1, see e.g. N. C. Bonciocat, M. Cipu, M. Mignotte, On D(-1)-quadruples, Publ. Mat. 56 (2012), 279-304. In particular, if {1,b,c,d} has considered property, then b>10^{13}.
I have been trying to find a 4-element set that satisfies the property. I ran a computer program to check all 4-element sets of integers <= 20,000 and found nothing, so I am guessing that there are no such sets, although of course I have nothing conclusive.
The program also listed all of the 3-element sets that it found, and I've noticed that each set, taken modulo 4, is either {1, 1, 1} or {1, 1, 2}. I can now prove this:
It is fairly easy to see that, if a*b-1 is square, then {a, b} modulo 4 is one of {1, 1}, {3, 3}, {1, 2}, and {2, 3}.Next, looking through some notes on number theory that I printed from MIT OpenCourseware, there is a lemma which states that, because a*b is the sum of two squares (n^2 + 1^2), any prime factors of a*b which are congruent to 3 mod 4 must divide both squares, i.e., n and 1. Since no primes divide 1, this implies that a*b has no prime factors congruent to 3 mod 4, from which it follows that a and b are both not congruent to 3 mod 4.
Thus, {a, b} modulo 4 is one of {1, 1}, {1, 2}. From this, it is easy to see that any 3-element set satisfying the property is of one of the forms {1, 1, 1}, {1, 1, 2}, as desired.
@pipi: how did you prove the original problem "modulo 4"? Does your result imply that any set of the form {1, 1, 2} modulo 4 cannot be extended to a 4-element set? If so, by using your argument and then modifying it to work on {1, 1, 1}, we may be able to prove that no 4-element sets satisfying the property exist.