A curious algebraic fraction that converges to $\frac{\sqrt{2}}{2}$

I presume you mean $$ \eqalign{ a_{n+1} &= 3 a_n + 2 b_n\cr b_{n+1} &= 4 a_n + 3 b_n\cr}$$

That is $X_{n+1} = A X_n$ where $X_n = \pmatrix{a_n\cr b_n\cr}$ and $A = \pmatrix{3 & 2\cr 4 & 3\cr}$. The reason this works is that $A$ has eigenvalues $3 \pm 2 \sqrt{2}$, with eigenvector $\pmatrix{1\cr \sqrt{2}\cr}$ for $3 + 2 \sqrt{2}$.

More generally, suppose $A$ is an $m \times m$ real matrix with a simple eigenvalue $\lambda > 0$, and all other eigenvalues have absolute value strictly less than $\lambda$. Let $V$ be a normalized eigenvector for $\lambda$. Then for almost any initial vector $X_0$, the vectors $X_n = A^n X_0$ satisfy $X_n / \|X_n\| \to \pm V$ as $n \to \infty$.

Interested by your post, I have been playing with $$a_{n+1}=\alpha\,a_n+(\alpha -1)\,b_n$$ $$b_{n+1}=(\alpha+1)\,a_n+\alpha\,b_n$$ using $a_0=b_0=1$. I shall not reproduce here the formulae for $a_n$ and $b_n$.

Focusing on the limit $$L_\alpha=\lim_{n\to \infty}\frac{a_n}{b_n}$$ I was able to find some interesting results $$L_2=\frac{1}{\sqrt{3}}$$ $$L_3=\frac{1}{\sqrt{2}}$$ $$L_4=\sqrt{\frac{3}{5}}$$ $$L_5=\sqrt{\frac{2}{3}}$$ $$L_6=\sqrt{\frac{5}{7}}$$ $$L_7=\frac{\sqrt{3}}{2}$$ $$L_8=\frac{\sqrt{7}}{3}$$ $$L_9=\frac{2}{\sqrt{5}}$$ which correspond to $$L_\alpha=\sqrt{\frac{\alpha-1}{\alpha+1}}$$ which, as Winther commented, correspond to the positive solution of equation $$x=\frac{\alpha x+(\alpha-1)}{(\alpha+1) x+\alpha}$$

I think we can split the ratio $(3a + 2b)/(4a + 3b)$ further. Let $a/b$ be a rational approximation to $1/\sqrt{2}$. Then I prove that $(a + b)/(2a + b)$ is a better approximation to $1/\sqrt{2}$ but in opposite direction.

Clearly we have $$\left(\frac{a + b}{2a + b}\right)^{2} - \frac{1}{2} = \frac{b^{2} - 2a^{2}}{2(2a + b)^{2}} = \frac{b^{2}}{(2a + b)^{2}}\left(\frac{1}{2} - \frac{a^{2}}{b^{2}}\right)$$ so that $Y = \left(\dfrac{a + b}{2a + b}\right)^{2} - \dfrac{1}{2}$ and $X = \dfrac{a^{2}}{b^{2}} - \dfrac{1}{2}$ have opposite signs and $|Y| < |X|$ and thus the claim in previous paragraph is established.

Applying the transformation $a/b \to (a + b)/(2a + b)$ twice on $a/b$ we get $$\frac{3a + 2b}{4a + 3b}$$ and we are thus ensured that if $a/b$ is a rational approximation to $1/\sqrt{2}$ then $\dfrac{3a + 2b}{4a + 3b}$ is a better approximation to $1/\sqrt{2}$ in the same direction. Since a bounded and monotone sequence is convergent it follows that repeated application of the transformation $a/b \to (3a + 2b)/(4a + 3b)$ to any positive rational number $a/b$ converges to $1/\sqrt{2}$.

BTW this comes from an exercise in Hardy's A Course of Pure Mathematics (page 11, Examples III, problem 3) but Hardy uses it for $\sqrt{2}$:

Show that if $m/n$ is a good approximation to $\sqrt{2}$, then $(m + 2n)/(m + n)$ is a better one, and that the errors in the two cases are in opposite directions.