Limit of the quotient of sums of recurrent relations
Solution 1:
$\newcommand{\ds}{\displaystyle}\newcommand{\eps}{\varepsilon}$ For $b\in]0,1]$, we have $$\label{equ}\tag{1}A(b,x)\sim -\frac1{\log(1+b)}\log(1-x)\mbox{ as }x\to1-$$ in the sense that the quotient of both sides tends to $1$. As a consequence, the limit of the OP is $$\lim_{x\to1-}\frac{A(bx)}{A(1,x)}=\frac{\log(2)}{\log(1+b)}.$$ Observe that the right hand side (which we call $Q(b)$) is not very far from the conjectured $\tilde Q(b)=1+\frac{(1-b)\log(2)}b$: Both have $Q(1)=\tilde Q(1)=1$ (of course) and $Q(b)\sim \tilde Q(b)\sim \frac1b\log2$ as $b\to0+$. Precisely, $b\tilde Q(b)$ is the linear function coinciding with $bQ(b)$ at the points $b=0$ and $b=1$.
Proof of (\ref{equ}) for $b=1$. Here we know from the OP that $a_n(1,x)=x^{2^n-1}$ for all $n$. The sequence $a_n(b,x)$ is decreasing, so by comparison of sum and integral we have $A(1,x)-1\leq \int_{0}^\infty x^{2^s-1}\,ds\leq A(1,x)$. Now we rewrite (using $\alpha=-\log(x)$ and $s=\frac1{\log2}\log(1+t)$) $$\begin{array}{rcl}\ds\int_{0}^\infty x^{2^s-1}\,ds&=& \displaystyle\int_0^\infty \exp(-\alpha(2^s-1)))\,ds= \frac1{\log(2)}\int_0^\infty\exp(-\alpha t)\frac1{1+t}\,dt=\\ &=&\displaystyle\frac{e^\alpha}{\log(2)}\int_1^\infty\exp(-\alpha u)\frac1u\,du= \frac{e^\alpha}{\log(2)}E_1(\alpha) \end{array} $$ with the exponential integral $E_1$. As stated in that web page, $E_1(z)\sim -\log(z)$ as $z\to0+$. Here $\alpha=-\log(x)\sim 1-x$ as $x\to1-$ and hence $E_1(\alpha)\sim -\log(\alpha)\sim-\log(1-x)$ as $x\to1-$. This proves (\ref{equ}) for $b=1$.
Proof of (\ref{equ}) for $0<b<1$. We fix $b$ and do not indicate the dependence upon $b$ unless necessary. The problem is that $a_n(x)$ remains close to $1$ for a long time if $x$ is close to 1. This has to be quantified if we want to find an equivalent of $A(x)$ as $x\to1-$.
The idea is to use that $p=p(x)=1+\frac{1-x}{bx}$ is the second fixed-point (besides 0) of the mapping $a\mapsto x\cdot(b\,a^2+(1-b)a)$ used in the recursion. Observe that $p$ is close to $1$ when $x\approx1$. Then we define $c_n(x)=p(x)-a_n(x)$ and find a recursion for them $$\label{eqb}\tag{2} c_0(x)=\frac{1-x}{bx},\ c_{n+1}(x)=x\,c_n(x)\,(1-b+2bp(x)-b\,c_n(x)). $$ The sequence $c_n(x)$ is positive and increasing since $a_n(x)\leq1$ is decreasing.
We choose $\eps>0$ and define $N=N(\eps,x)$ as the largest integer $n$ such that
$c_n(x)\leq\eps$. Hence $c_N(x)\leq\eps<c_{N+1}(x)$.
Now the quotients $c_{n+1}(x)/c_n(x)=x(1-b+2bp(x)-b\,c_n(x))$ vary between
$Q(x)=x(1-b+2bp(x))-(1-x)=1+bx$
and $q(\eps,x)=x(1-b+2bp(x)-b\,\eps)\leq x(1-b+2bp(x)-b\,c_N(x))<Q(x)$
for $n=0,\dots,N$.
Since $a_0(x)=1$, we have
$$\frac{1-x}{bx}q(\eps,x)^n\leq c_n(x)\leq \frac{1-x}{bx}Q(x)^n\mbox{ for }n=0,\cdots,N+1.$$
Finally the inequalities defining $N(\eps,x)$ imply
$$\begin{array}{rcl}\ds N(\eps,x)&>&\ds\frac1{\log(Q(x))}(-\log(1-x)+\log(bx)+\log(\eps))-1\\
\ds N(\eps,x)&\leq& \ds\frac1{\log(q(\eps,x))}(-\log(1-x)+\log(bx)+\log(\eps)).
\end{array}$$
This gives, as $x\to1-$:
$$\label{ineqN}\tag{3}\begin{array}l
\ds\liminf_{x\to1-}\frac{N(\eps,x)}{-\log(1-x)}\geq\frac1{\log(Q(1))}=\frac1{\log(1+b)},\\
\ds\limsup_{x\to1-}\frac{N(\eps,x)}{-\log(1-x)}\leq\frac1{\log(q(\eps,1))}=\frac1{\log(1+b-b\eps)}.
\end{array}
$$
Now we split $$A(x)=\sum_{n=0}^{N(\eps,x)}a_n(x)+\sum_{n=N(\eps,x)+1}^\infty a_n(x).$$ The first part is simply estimated by $1+(1-\eps)N(\eps,x)\leq \sum_{n=0}^{N(\eps,x)}a_n(x)\leq1+ N(\eps,x) $. For an estimate of the tail, observe that $a_n(x)\leq 1-\eps$ for $n>N(\eps,x)$ and hence the quotients $a_{n+1}/a_n(x)\leq x(b(1-\eps)+1-b)\leq x(1-b\eps)\leq1-b\eps$ for all $x<1$. Hence, for all $x<1$ and $n=N(\eps,x)+1+m$ we have $a_n(x)\leq (1-\eps)(1-b\eps)^m$ and hence $$\sum_{n=N(\eps,x)+1}^\infty a_n(x)\leq(1-\eps)\sum_{m=0}^\infty(1-b\eps)^m= \frac{1-\eps}{b\eps}.$$ As this is valid for all $x<1$, we find using (\ref{ineqN}) $$\begin{array}l \ds\liminf_{x\to1-}\frac{A(b,x)}{-\log(1-x)}\geq\frac{1-\eps}{\log(1+b)},\\ \ds\limsup_{x\to1-}\frac{A(b,x)}{-\log(1-x)}\leq\frac1{\log(1+b-b\eps)}. \end{array} $$ As $\eps>0$ can be chosen arbitrarily, this implies $\ds\lim_{x\to1-}\frac{A(b,x)}{-\log(1-x)}=\frac{1}{\log(1+b)}$ and hence (\ref{equ}) for $0<b<1$ as wanted.
Remark: Actually the second part of the proof also works for $b=1$.