Eigenvalues of large tridiagonal matrix
Solution 1:
Since $M_n(a,b)$ and $M_n(a,-b)$ have same real spectrum, we may assume that $b\geq 0$. Let $\lambda_n$ be the smallest eigenvalue of $M_n$. Since there exist hidden orthogonal polynomials, the real sequence $(\lambda_n)_n$ is non-increasing.
Assume that $a\geq 0$. Note that $e_1^TM_ne_1=a^2$; then $\lambda_n\leq a^2$. Denote by $B_n$ the matrix $M_n$ with a zero diagonal (only the $b$'s remain). Then $M_n\geq B_n$ and $\lambda_n\geq \inf(\text{spectrum}(B_n))\geq -2b$. Finally the sequence $(\lambda_n)_n$ converges to $\lambda\in [-2b,a^2]$.
Note that , if $\dfrac{b}{a^2}$ is small enough, then $M_n\geq 0$ and $\lambda\approx a^2$. If $a$ is fixed and $b$ tends to $+\infty$, then $\lambda\rightarrow -2b$.
Solution 2:
Let's call $M_n$ this matrix, and let's consider its characteristic polynomial $P_n := \det(XI_n-M_n)$. Expansion according to the last column yields the recurrence relation $$ P_n = (X-(a+n)^2)P_{n-1}-b^2P_{n-2}. $$ with initial conditions $P_0 = 1$ and $P_1 = X-a^2$.