Pythagorean theorem expressed without roots in an old Tamilian (Indian) statement
Solution 1:
Let the legs measure $a$ and $ka$ with $k>1$. Then the approximation given is:
$\frac{7ka}{8}+\frac{4a}{8}=\frac{(7k+4)a}{8}$.
The real measure is $\sqrt{k^2a^2+a^2}=\sqrt{k^2+1}a$.
So how does $\frac{7k+4}{8\sqrt{k^2+1}}$ behave? Not that bad.
Here is a graph:
So as the graph shows, in the interval $(1,\infty)$ the best approximation occurs at $k=1$ for which we get $\frac{7+4}{8\sqrt{2}}=\frac{11}{2\sqrt2}\approx 0.972$. After this the ratio becomes smaller and smaller, the limit is $\frac{7}{8}=0.875$.
Can this be improved? yes it can, the formula always gives us a length shorter than the actual length, so we can obtain a better aproximation by taking slightly larger coefficients.
A better question is which is the best aproximation for $\sqrt{a^2+b^2}$ that is of the form $la+mb+n$ with $l,m,n\in \mathbb Q$. Now, the value of $l$ is going to be irrelevant because when $a$ and $b$ are large enough the $l$ won't matter much.
So we need to approximate $\sqrt{a^2+b^2}$ with $la+mb$. If we write $b$ as $ka$ then we need to approximate $\sqrt{k^2+1}a$ with $l+mk(a)$. So essentially what we need to do is approximate $\sqrt{k^2+1}$ with $l+mk$. This is the real problem.
The approximation for $\sqrt{k^2+1}$ provided in the question is $\frac{7k+4}{8}$. Now, if we were to approximate $\sqrt{k^2+1}$ by $mk+l$ I would think it would be in our best interest to make $m=1$ (so that at least when $k$ goes to infinity the limit becomes $1$. Then it is only a matter of finding a good $l$.
I didn't think a lot but taking $l=\frac{3}{7}$ seems to give a good result. Here is the graph of $\frac{k+3/7}{\sqrt{k^2+1}}$
This is a better approximation, the ratio of the correct measurement, versus the actual measurement when $k\geq 1$ reaches a maximum of approximately $1.088$ when $k=2.\overline 3$, this is the worst it gets, it improves when $k$ approaches $1$, reaching the minimum of $1.01$ and it also improves as $k$ goes to infinity, with a limit of $1$. (as an opposite of the first approximation this approximation always gives a hypotenuse longer than it actually is, which again tells us this is not actually the best approximation)
So a better way to approximate the hypotenuse is to add the longer leg's length plus three sevenths of the shorter leg's length.
Solution 2:
The method provides a not-so-great approximation when the two catheti are roughly of the same length. In this case, $$\frac{7}{8}a+\frac{1}{2}b\approx\frac{7}{8}a+\frac{1}{2}a=\frac{11}{8}a=1\mathrm.375 a\approx(1\mathrm.414...)a=\sqrt{2}{a}\approx\sqrt{a^2+b^2}.$$