A set with a finite integral of measure zero?

Yes it is true. Consider the set $S$ on which the integral in question is bounded by a positive value $K$. Then $\mu$ cannot have measure greater than $K\epsilon^2$ on any interval of width $\epsilon$ intersecting $S$. However, any interval $[a,b]$ can be divided into $n$ intervals of width $(b-a)/n$. So, $\mu(S\cap[a,b])$ is bounded by $n K ((b-a)/n)^2$. Let $n$ go to infinity.

I didn't count, but I know that exceeded 30 words by a fair margin.

The argument is easily adapted to show that $\int\vert x-y\vert^{-1-\epsilon}\,d\mu(y)$ is infinite $\mu$-almost everywhere for any $\epsilon > 0$. The more difficult question is whether the same holds for $\int\vert x-y\vert^{-1}\,d\mu(y)$.

this is a clumsy solution and certainly too long - but anyway. The answer is yes. There is a (weakly) increasing function $f:I\to\mathbb{R}$ such that $\mu$ is the push-forward of the Lebesgue measure on $I$ to $\mathbb{R}$ ($f$ is, roughly speaking, the inverse of $F(x)=\mu((-\infty,x])$). Your integral becomes $\int_I (f(x)-f(y))^{-2} dy$ (where "your $x$" is "my $f(x)$"). Since $f$ is monotone, it has a derivative a.e. If $x$ is in the set where the derivative is $<A$ (for any positive $A$) then the integral is $\infty$ by comparison with $\int_I (x-y)^{-2} dy$. Hence the integral is $\infty$ for a.e. $x$.

(not very pretty - but possibly correct :)