Find all integer solutions of: $\;\frac{1}{m}+\frac{1}{n}-\frac{1}{mn^2}=\frac{3}{4}$

Hint: express as: $$4m+4n=3mn+\frac{4}{n}.$$

HINT: solving for $m$ is better, we do not need a quadratic equation: $$m=\frac{4(n^2-1)}{3n^2-4n}$$ We get $$m=3,n=2$$

As an alternative to Dr. Sonnhard Graubner's answer on how to find the complete set of solutions from

$m=\frac{4(n^{2}−1)}{3n^{2}−4n}$

consider the following:

We can factor that expression further to

$m=\frac{4(n+1)(n-1)}{n(3n−4)}$

That tells us a lot about $n$. First off, if $n$ is 1, then $m$ must be 0, but that is not a valid solution because the original expression involves a division by $m$. So $n$ cannot be 1. For the same reason, $n$ cannot be -1.

Since $n$ is an integer, both the numerator and denominator of that expression are integers. In order for their quotient $m$ to also be an integer, the numerator must be divisible by the denominator. Since $n\neq\pm1$, both $(n+1)$ and $(n-1)$ must be relatively prime to $n$, so neither $n$ nor any of its prime factors can divide either of these terms. Therefore, if $m$ is an integer, 4 must be divisible by $n$. The only candidates are $\pm1$, $\pm2$, and $\pm4$, and we already excluded $\pm1$.

Substituting $n=2$ gives us $m=3$, so $(m, n) = (3, 2)$ is a valid solution.

Substituting $n=4$ gives us $m=\frac{15}{8}$, which is not an integer and therefore not a valid solution. Likewise, $n=-2$ yields $m=\frac{3}{5}$ and $n=-4$ yields $m=\frac{15}{64}$.

The other answers may be more elegant, but here is a way to continue from the point you reached. Assume $m$ and $n$ are integers. From $$n = \frac{-2m \pm 2\sqrt{m^2-3m+4}}{4-3m}$$ you know that $-2m \pm 2\sqrt{m^2-3m+4}$ must be a multiple of the integer $4-3m$; in particular, it must be an integer. Therefore $2\sqrt{m^2-3m+4}$ is an integer (since it is a difference of the integer $-2m$ and another integer), which implies that $4(m^2-3m+4)$ is the square of an integer.

Suppose $4(m^2-3m+4) = k^2$ for some integer $k.$ Complete the square: \begin{align} k^2 &= 4(m^2-3m+4)\\ &= 4m^2 - 12m + 16 \\ &= (2m - 3)^2 + 7 \end{align} and therefore the difference between the two squares $(2m - 3)^2$ and $k^2$ is $7.$ This is possible only if the two squares are $9$ and $16.$ Therefore $k^2 = 16$ and $$(2m - 3)^2 = 9.$$ This quadratic equation in $m$ has two roots, $m = 0$ and $m = 3,$ but $m = 0$ cannot be true for any solution to the original problem (since we require $1/m$ to be defined); therefore $$ m = 3.$$ Plug this into your equation for $n$ and confirm that the only possible integer value of $n$ is $n = 2.$

Alternative method:

Suppose that you solved for $m$ as in several other answers, so that you found the equation $$ m = \frac{4(n^2 - 1)}{3n^2 - 4n}. $$

If you know a few facts about rational functions (polynomials divided by polynomials), you can sketch $m$ as a function of $n$ free-hand (without the assistance of a graphing calculator or software) and solve the problem graphically.

First, plot the functions $m = 4(n^2 - 1)$ and $m = 3n^2 - 4n$ as shown below.

Because $3n^2 - 4n = 0$ at $n=0$ and $n = \frac43,$ we know that the rational function has vertical asymptotes at the lines $n=0$ and $n = \frac43.$ By looking at the highest-order coefficients of $4(n^2 - 1)$ and $3n^2 - 4n,$ we know the rational function has a horizontal asymptote at $m = \frac43.$ We know the rational function has the same zeros as $4(n^2 - 1),$ namely $n = \pm1.$ And by carefully accounting for the direction in which $3n^2 - 4n$ crosses the $n$-axis and the sign of $4(n^2 - 1)$ at each crossing, we can find out how the rational function approaches each of its asymptotes. The result is something like the following graph:

Assuming we sketched this free-hand, we do not yet know how close the curve is to various points with integer coordinates. But since the middle branch has asymptotes $n=0$ and $n = \frac43,$ its only possible integer solution is at $n=1$ (which indeed occurs at $(1,0)$). Since the left branch passes through $(-1,0),$ its only possible other integer solution is at $m=1$ (at which we can solve for $n,$ discovering that $n$ is not an integer in that case). We can then chip away at the right branch, either by finding $m$ when $n = 2$ (the smallest possible integer value of $n$ on that branch) or by solving for $n$ when $m = 2$ (the smallest possible integer value of $m$ on that branch). Either way, we quickly find out that $(2,3)$ is a solution, that the only other possible integer solution would be at $m = 2,$ and that in fact the curve crosses $m = 2$ at a non-integer value of $n$; therefore there are no other solutions. We eliminate $(1,0)$ and $(-1,0)$ by referring to the original problem statement, leaving only $n = 2, m = 3.$

First off, your attempt is a good start, but leads down a somewhat difficult rabbit hole (though David K. gives a very nice explanation). Indeed, the next thing that you would need to do is determine values of $m$ such that $$ m^2 - 3m + 4 $$ is a perfect square. This could be a viable line of attack, but it seems hard. So, instead, let's put a pin in that argument and try again from the beginning.

Instead of trying to solve for $n$, let's try to solve for $m$ instead (this might not work any better, but if we get stuck again, we can always go back to what you were trying in the first place). \begin{align} \frac{1}{m} + \frac{1}{n} - \frac{1}{mn^2} = \frac{3}{4} &\implies 4n^2 + 4mn - 4 = 3mn^2 \\ &\implies 4mn - 3mn^2 = 4 - 4n^2 \\ &\implies (4n-3n^2)m = 4 - 4n^2 \\ &\implies m = \frac{4-4n^2}{4n-3n^2}. \end{align} This still looks super complicated, but let's see what we can do with it. First off, we might try looking at the graph of this function. This isn't going to give us a rigorous argument, but it might tell us where to look. So, putting $n$ horizontal axis and $m$ on the vertical axis, we get

We are interested in points on the graph that hit the corners of the grid. In the picture, it looks like there are potential solutions at $n = -5, -1, 1, 2$. Outside the bounds of the picture, it looks like the graph increases to a little more than $1$ as $n$ tends to $-\infty$, and decreases to the same value as $n$ increases to $+\infty$, hence there are no other potential integer solutions (this can be made rigorous by appealing to the first derivative test from an intro level calculus course). By guess-and-checking, we get \begin{align} f(-5) &\approx 1.01 && (\text{not an integer}) \\ f(-1) &= 0 \\ f(1) &= 0 \\ f(2) &= 3. \end{align} Thus, via this approach, we rule out one potential solution ($n=-5$), and obtain three potential solutions ($(m,n) = (0,-1), (0,1), (3,2)$). However, if we try to substitute $m=0$ into the original equation, we end up dividing by zero, which is bad news. Hence the only integer solution is $$ (m,n) = (3,2). $$

I see that while I was typing, user3553031 provided an answer that is essentially the same as the following. So now you get it two ways. :)

Tackling this a little more rigorously, let's return to the equation $$ m = \frac{4-4n^2}{4n-3n^2} = \frac{4(n^2-1)}{3n^2-4n} = \frac{4(n+1)(n-1)}{n(3n-4)}. $$ This will be an integer if and only if the denominator divides the numerator. However, as we observed above, we cannot have $n=\pm 1$, which implies that $n$ cannot divide either $n+1$ or $n-1$—indeed, $n$ is relatively prime to each of these, so no factor of $n$ can divide $(n+1)(n-1)$. But $m$ is an integer, which implies that $n$ must be a factor of the numerator, so it must be that $n$ is an integer which divides 4. Therefore $n = \pm 1$ (which we have already ruled out), $n = \pm 2$, or $n = \pm 4$. Testing each of these (using a calculator, because I am lazy), we obtain \begin{align} n = -4 & \implies m = 0.9375 \\ n = -2 & \implies m = 0.6 \\ n = 2 & \implies m = 3 \\ n = 4 & \implies m = 1.875. \end{align} Only one of these solutions gives integer values for both $m$ and $n$, so we must conclude that the unique solution to the original problem is $$ (m,n) = (3,2). $$