Solution 1:

Let us examine the series

$$ y_{n+1} = \frac{ay_n + b}{cy_n + d}$$

We can split this into two series (numerator and denominator)

as

$$y_n = \frac{p_n}{q_n}$$

and

$$p_{n+1} = ap_n + bq_n$$ $$q_{n+1} = cp_n + dq_n$$

This can be rewritten in matrix form as

$$\begin{bmatrix}a&b\\c&d \end{bmatrix}\begin{bmatrix}p_n\\q_n\end{bmatrix} = \begin{bmatrix}p_{n+1}\\q_{n+1}\end{bmatrix}$$

And thus

$$ A^n \begin{bmatrix}p_0\\q_0\end{bmatrix} = \begin{bmatrix}p_n\\q_n\end{bmatrix}$$

where

$$A = \begin{bmatrix}a&b\\c&d \end{bmatrix}$$

If $A$ is diagonalizeable then we get that both $p_n$ and $q_n$ are of the form

$$\alpha \lambda_1^n + \beta \lambda_2^n$$

where $\lambda_i$ are the eignevalues (and $\alpha$ for $p_n$ is different from $\alpha$ for $q_n$).

There might be corner cases when the eigenvalues are equal, in which case I believe the general form is $$\alpha n \lambda^n$$ (not sure, though).

In your case the matrix $A$ is

$$A = \begin{bmatrix}1&b^2\\1&a^2+b^2 \end{bmatrix}$$

Which is diagonalizable, and the corner case of equal eignevalues occurs when $(a^2 + b^2)^2 = 4a^2$

In either case, if $\lambda_1$ is the greater root, then the sequence converges to the ratio of the $\alpha$(i.e. coefficients of $\lambda_1$) for $p_n$ and $q_n$.