Show that if $a \neq b$ and a and b are positive then $\frac{a}{b}+\frac{b}{a}$ is never an integer

Let's solve the equation $x + 1/x = n$ where $n$ is any positive integer. It's equivalent to the quadratic equation $x^2 - nx + 1 = 0$. Its coefficients are integers so, if $a/b$ is a root with $\operatorname{gcd}(a,b) = 1$ and $a\neq b$ then $a$ divides the constant coefficient $1$ and $b$ divides the dominant coefficient $1$.

(I assume that $a$ and $b$ are both integers...)

Let $x=\dfrac{a}{b}+\dfrac{b}{a}=\dfrac{a^2+b^2}{ab}$. We can assume that $a$ and $b$ are relatively prime, since substituting $a/g$ and $b/g$ instead of $a$ and $b$ doesn't change the value of $x$.

If $x$ is integer, than $ab$ should divide $a^2+b^2$, so

$$a|(a^2+b^2) \; \Rightarrow\; a|b^2$$

$$b|(a^2+b^2) \; \Rightarrow\; b|a^2$$

Since we assumed that $a$ and $b$ are relatively prime, these formulas hold only when $a=b=1$, which is contradicted to the condition $a\neq b$.

Therefore, $x\not\in \mathbb{Z}$.

Presumably you want $a,b$ to be integers. If $x = a/b$, solve the equation $$x + \dfrac{1}{x} = n$$ to get $$x = \dfrac{n}{2} \pm \dfrac{\sqrt{n^2-4}}{2} $$ The question is whether $\sqrt{n^2-4}/2$ is ever rational. This would require $\sqrt{n^2-4}$ to be an integer (because the square root of an integer is only rational if it is an integer). But then if $m = \sqrt{n^2-4}$, we would have $n^2 - m^2 = 4$. The only squares of integers that differ by $4$ are $2^2$ and $0^2$, but that corresponds to $x = 1$, i.e. $a = b$.