Prove that $f'(0)$ exists and $f'(0) = b/(a - 1)$

Problem:
If $f(x)$ is continous at $x=0$, and $\lim\limits_{x\to 0} \dfrac{f(ax)-f(x)}{x}=b$, $a, b$ are constants and $|a|>1$, prove that $f'(0)$ exists and $f'(0)=\dfrac{b}{a-1}$.

This approach is definitely wrong:

\begin{align} b&=\lim_{x\to 0} \frac{f(ax)-f(x)}{x}\\ &=\lim_{x\to 0} \frac{f(ax)-f(0)-(f(x)-f(0))}{x}\\ &=af'(0)-f'(0)\\ &=(a-1)f'(0) \end{align}

I will show you a case why this approach is wrong:

\[f(x)= \begin{cases} 1,&x\neq0\\ 0,&x=0 \end{cases}\] $\lim_{x\to0}\dfrac{f(3x)-f(x)}{x}=\lim_{x\to0} \dfrac{1-1}{x}=0$
but $\lim_{x\to0}\dfrac{f(3x)}{x}=\infty$,$\lim_{x\to0}\dfrac{f(x)}{x}=\infty$

Does anyone know how to prove it? Thanks in advance!

This is a tricky question and the solution is somewhat non-obvious. We know that $$\lim_{x \to 0}\frac{f(ax) - f(x)}{x} = b$$ and hence $$f(ax) - f(x) = bx + xg(x)$$ where $g(x) \to 0$ as $x \to 0$. Replacing $x$ by $x/a$ we get $$f(x) - f(x/a) = bx/a + (x/a)g(x/a)$$ Replacing $x$ by $x/a^{k - 1}$ we get $$f(x/a^{k - 1}) - f(x/a^{k}) = bx/a^{k} + (x/a^{k})g(x/a^{k})$$ Adding such equations for $k = 1, 2, \ldots, n$ we get $$f(x) - f(x/a^{n}) = bx\sum_{k = 1}^{n}\frac{1}{a^{k}} + x\sum_{k = 1}^{n}\frac{g(x/a^{k})}{a^{k}}$$ Letting $n \to \infty$ and using sum of infinite GP (remember it converges because $|a| > 1$) and noting that $f$ is continuous at $x = 0$, we get $$f(x) - f(0) = \frac{bx}{a - 1} + x\sum_{k = 1}^{\infty}\frac{g(x/a^{k})}{a^{k}}$$ Dividing by $x$ and letting $x \to 0$ we get $$f'(0) = \lim_{x \to 0}\frac{f(x) - f(0)}{x} = \frac{b}{a - 1} + \lim_{x \to 0}\sum_{k = 1}^{\infty}\frac{g(x/a^{k})}{a^{k}}$$

The sum $$\sum_{k = 1}^{\infty}\frac{g(x/a^{k})}{a^{k}}$$ tends to $0$ as $x \to 0$ because $g(x) \to 0$. The proof is not difficult but perhaps not too obvious. Here is one way to do it. Since $g(x)\to 0$ as $x \to 0$, it follows that for any $\epsilon > 0$ there is a $\delta > 0$ such that $|g(x)| < \epsilon$ for all $x$ with $0 <|x| < \delta$. Since $|a| > 1$ it follows that $|x/a^{k}| < \delta$ if $|x| < \delta$ and therefore $|g(x/a^{k})| < \epsilon$. Thus if $0 < |x| < \delta$ we have $$\left|\sum_{k = 1}^{\infty}\frac{g(x/a^{k})}{a^{k}}\right| < \sum_{k = 1}^{\infty}\frac{\epsilon}{|a|^{k}} = \frac{\epsilon}{|a| - 1}$$ and thus the sum tends to $0$ as $x \to 0$.

Hence $f'(0) = b/(a - 1)$.

BTW the result in question holds even if $0 < |a| < 1$. Let $c = 1/a$ so that $|c| > 1$. Now we have $$\lim_{x \to 0}\frac{f(ax) - f(x)}{x} = b$$ implies that $$\lim_{t \to 0}\frac{f(ct) - f(t)}{t} = -bc$$ (just put $ax = t$). Hence by what we have proved above it follows that $$f'(0) = \frac{-bc}{c - 1} = \frac{b}{a - 1}$$ Note that if $a = 1$ then $b = 0$ trivially and we can't say anything about $f'(0)$. And if $a = -1$ then $f(x) = |x|$ provides a counter-example. If $a = 0$ then the result holds trivially by definition of derivative. Hence the result in question holds if and only if $|a| \neq 1$.

In this quickly closed question the case $a=2$ is considered, which allows the following simpler solution:

Define $g(x):=f(x)- bx-f(0)$. Then $g$ is continuous at $0$, $g(0)=0$, and $$\lim_{x\to0}{g(2x)-g(x)\over x}=0\ .$$ We have to prove that $g'(0)=\lim_{x\to0}{g(x)\over x}=0$.

Let an $\epsilon>0$ be given. Then there is a $\delta>0$ such that $|g(2t)-g(t)|\leq\epsilon |t|$ for $0<t\leq\delta$. Assume $|x|\leq\delta$. Then for each $N\in{\mathbb N}$ one has $$g(x)=\sum_{k=1}^N\bigl(g(x/2^{k-1})-g(x/2^k)\bigr)+g(x/2^N)\ ,$$ and therefore $$\bigl|g(x)\bigr|\leq\sum_{k=1}^N\epsilon\,{|x|\over 2^k} \ +g(x/2^N)\leq\epsilon|x|+g(x/2^N)\ .$$ Since $N\in{\mathbb N}$ is arbitrary we in fact have $\bigl|g(x)\bigr|\leq \epsilon|x|$, or $\left|{g(x)\over x}\right|\leq\epsilon$, and this for all $x\in\>]0,\delta]$.