Probability of exactly $r$ books between A and B

The subset $\{A,B\}\subset[n]$ can occupy ${n\choose2}$ different pairs of sites in the resulting order, all of them equiprobable. Exactly $n-r-1$ of such pairs have the required distance $r+1$ among the two constituents. It follows that the probability $p$ in question is given by $$p={n-r-1\over {n\choose2}}={2(n-r-1)\over n(n-1)}\ .$$

PS: Your own thoughts about the problem, even allowing of a different way of counting, are wide off the mark.

If there are exactly $r$ books between books A and B, then the block of books with books A and B at the ends has length $r + 2$. Consequently, the block must begin in one of the first $n - (r + 2) + 1 = n - r - 2 + 1 = n - r - 1$ positions. There are $2$ ways to choose whether book A or book B is at the left end of the block. There are $(n - 2)!$ ways to arrange the remaining books in order. Hence, the probability that there are exactly $r$ books between books A and B is $$\frac{2(n - r - 1)(n - 2)!}{n!} = \frac{2(n - r - 1)(n - 2)!}{n(n - 1)(n - 2)!} = \frac{2(n - r - 1)}{n(n - 1)}$$

Method 1

The arrangement on the book shelf looks somewhat like

$$\text{* }\mid \text{ * }\mid\text{ *}$$ where the $\mid$ denote either books $\text{A}$ or $\text{B}$. The stars denoted on the either ends can be empty, but the star denoted in between the bars must contain exactly $r$ books. Say, the leftmost star contains $x$ books, then the right most contains $(n-r-2-x)$ books. Totals number of ways of arranging the books on the self is $n!$. The desired number of permutations is $$ \sum_{x=0}^{n-r-2} 2! {n-2\choose r} r! {n-r-2\choose x} x!(n-r-2-x)!$$ $$ =\sum_{x=0}^{n-r-2} 2(n-2)! = 2(n-r-1)(n-2)!\\ $$ Divide this by $n!$ to obtain probability $=\frac{2(n-r-1)}{n(n-1)}.$ $\qquad \square$

Method 2

Consider the chunk of books consisting of $\text{A}$ and $\text{B}$ and the $r$ books in between them as a single object. Thus, now the number of effective objects is $(n-r-1)$. Their permutations are $(n-r-1)!$. Permutations within the chunk are $2r!$ The number of ways of choosing the $r$ books in the chunk is ${n-2\choose r}$, hence the probability desired $$=\frac{2{n-2\choose r} (n-r-1)!r!}{n!}$$ $$=\frac{2(n-r-1)}{n(n-1)}. \qquad \square$$