Characterizing the solutions of the functional equation $ f ( 3 x ) - f ( 2 x ) = f ( 2 x ) - f ( x ) $
Solution 1:
In this answer, we prove that linear polynomials are indeed the only possible solutions:
Claim. Let $f : (0, \infty) \to \mathbb{R}$ be a continuous, non-decreasing function such that $$ f(x) = \frac{f(\frac{1}{2}x) + f(\frac{3}{2}x)}{2} \tag{FE} $$ holds for all $x > 0$. Then $f(x) = mx + c$ for some $m \geq 0$ and $c \in \mathbb{R}$.
Proof of the claim modulo key lemmas. We will use the following lemmas.
Lemma 1. Let $f : (0, \infty) \to \mathbb{R}$ be as in the above claim. Then for any $0 < a < 1 < b$ and $x > 0$, we have $$ f(x) \geq f(\tfrac{1}{2}ax) \biggl( \frac{2b-2}{2b-a} \biggr) + f(bx) \biggl( \frac{2-2a}{3b-2a} \biggr) \tag{1} $$ and $$ f(x) \leq f(ax) \biggl( \frac{3b-2}{3b-2a} \biggr) + f(\tfrac{3}{2}bx) \biggl( \frac{2-a}{2b-a} \biggr). \tag{2} $$
Intuitively, this lemma follows by repeatedly applying the functional equation $\text{(FE)}$ only to the terms $f(\square)$ satisfying $ax < \square < bx$. This argument can be conveniently implemented by using martingale. Since the proof is a bit lengthy, we will postpone it until the end.
Lemma 2. Let $g : \mathbb{R} \to \mathbb{R}$ be a bounded, continuous function such that $$ g(u) = \frac{1}{4}g(u-\log 2) + \frac{3}{4}g(u+\log(3/2)). $$ Then $g$ is constant.
Now let us check that this lemma indeed proves the desired conclusion. First, fix $b$ and set $x = 1$. Then $\text{(2)}$ shows that
$$ \biggl( \frac{3b-2a}{3b-2} \biggr) \biggl[ f(1) - f(\tfrac{3}{2}b) \biggl( \frac{2-a}{2b-a} \biggr) \biggr] \leq f(a), $$
hence $f(a)$ is bounded from below as $a \to 0^+$. Together with the montonicity of $f$, it follows that
$$c = \lim_{a \to 0^+} f(a)$$
exists. Now define $\tilde{f}(x) = f(x) - c$. Then $\tilde{f}$ also satisfies the condition of OP's problem, and in addition we have $\tilde{f}(0^+) = 0$. Then letting $ a \to 0^+$ to both $\text{(1)}$ and $\text{(2)}$ yields
$$\frac{2}{3b} \tilde{f}(bx) \leq f(x) \leq \frac{1}{b} \tilde{f}(\tfrac{3}{2}bx).$$
In particular, plugging $x=1$ shows that $ \tilde{f}(b)/b \leq 3\tilde{f}(1)/2 $, hence $\ell = \limsup_{b\to\infty} \tilde{f}(b)/b$ is finite. Then taking $\limsup$ as $b\to\infty$ to the above inequality, we obtain
$$ \frac{2\ell}{3} \leq \frac{\tilde{f}(x)}{x} \leq \frac{3\ell}{2}. $$
Now define $g(u) = e^{-u}\tilde{f}(e^u)$. The above inequality tells that $g$ is bounded. Also, $\text{(FE)}$ for $\tilde{f}$ tells that $g$ satisfies the condition of Lemma 2. Therefore $g$ is constant, proving the main claim.
Proof of Lemma 1. Let $X_1, X_2, \dots$ be i.i.d. random variables such that
$$ \mathbb{P}(X_i = \tfrac{3}{2}) = \mathbb{P}(X_i = \tfrac{1}{2}) = \frac{1}{2}. $$
Then the discrete-time process $M = (M_n)_{n\geq 0}$ defined by
$$ M_0 = 1, \qquad M_{n+1} = X_{n+1} M_{n} $$
is a positive martingale with respect to the natural filtration $\mathcal{F}_n = \sigma(X_1,\dots,X_n)$. Now this martingale is related to the original problem in such way that, if $f : (0, \infty) \to \mathbb{R}$ is as in OP's problem, then for each $x > 0$, the functional equation $\text{(*)}$ shows that
$$ \mathbb{E}[ f(M_{n+1} x) \mid \mathcal{F}_n] = \mathbb{E}[ f(X_{n+1}M_n x) \mid \mathcal{F}_n] = \frac{f(\frac{3}{2}M_n x) + f(\frac{1}{2}M_n x)}{2} = f(M_n x), $$
and so, $(f(M_n x))_{n\geq 0}$ is also a martingale. Now fix $0 < a < 1 < b$ and define
\begin{gather*} S = \inf\{ n \geq 0 : M_n \leq a \}, \\ T = \inf\{ n \geq 0 : M_n \geq b\}. \end{gather*}
Then by the SLLN or by other means, we can easily check that $\mathbb{P}(S < \infty) = 1$. Moreover, $\frac{a}{2} \leq M_{n\wedge S\wedge T}$ for all $n$, and so, both $(M_{n\wedge S\wedge T})_{n\geq 0}$ and $(f(M_{n\wedge S\wedge T} x))_{n\geq0}$ are bounded martingales. So by the optional stopping theorem,
$$ 1 = \mathbb{E}[M_0] = \mathbb{E}[M_{S\wedge T}] = \mathbb{E}[M_{S}\mathbf{1}_{\{S \leq T\}}] + \mathbb{E}[M_{T}\mathbf{1}_{\{S \geq T\}}] $$
as well as
$$ f(x) = \mathbb{E}[f(M_0 x)] = \mathbb{E}[f(M_{S\wedge T} x)] = \mathbb{E}[f(M_{S} x)\mathbf{1}_{\{S \leq T\}}] + \mathbb{E}[f(M_{T} x)\mathbf{1}_{\{S \geq T\}}]. $$
Now if we write $p_a = \mathbb{P}(S \leq T)$ and $p_b = \mathbb{P}(S \geq T)$, then $\frac{a}{2} \leq M_S \leq a$ and $b \leq M_T \leq \frac{3}{2}M_T$, together with the monotonicity of $f$, shows that
$$ \tfrac{1}{2}ap_a + bp_b \leq 1 \leq a p_a + \tfrac{3}{2}b p_b $$
and
$$ f(\tfrac{1}{2}a x)p_a + f(b x)p_b \leq f(x) \leq f(a x) p_a + f(\tfrac{3}{2}b x) p_b . $$
The first inequality, together with $p_a + p_b = 1$, implies that
$$ \frac{2b-2}{2b-a} \leq p_a \leq \frac{3b-2}{3b-2a}, \qquad\qquad \frac{2-2a}{3b-2a} \leq p_b \leq \frac{2-a}{2b-a}. $$
Then applying this to the second inequality proves the desired lemma.
Proof of Lemma 2. Let us consider the random walk $S_n = Z_1 + Z_2 + \dots + Z_n$ on the lattice $\mathbb{L} = \{a + ib \in \mathbb{C} : a, b \in \{0,1,2,\dots\} \}$, where $Z_k$'s are i.i.d. and
$$ \mathbb{P}(Z_k = 1) = \frac{1}{4} \qquad\text{and}\qquad \mathbb{P}(Z_k = i) = \frac{3}{4}. $$
Let $g$ be as in the assumption, and define $h : \mathbb{L} \to \mathbb{R}$ by
$$ h(a + ib) = g(- a \log 2 + b \log(3/2)). $$
Then $h$ is bounded and satisfies
$$ h(z) = \frac{1}{4}h(z+1) + \frac{3}{4}h(z+i) = \mathbb{E}[h(z+Z_1)] $$
for all $z \in \mathbb{L}$. In particular, this tells that $h(S_n)$ is a bounded martingale and hence converges a.s. by the martingale convergence theorem. We will show that $h$ is constant.
To see this, fix $z = a + ib \in \mathbb{L}$ and write $m = a + b$. Also, let $(Z'_k)_{k\geq 1}$ be an independent copy of $(Z_k)_{k\geq 1}$ and define $S'_n = z + Z'_1 + \dots + Z'_n$. Then
$$ Y_n := \operatorname{Re}(S_{m + n} - S'_n) $$
is a random walk on $\mathbb{Z}$ which is irreducible and recurrent. In particular,
$$T := \inf \{ n \geq 0 : Y_n = 0 \}$$
is a.s. finite. Now by noting that $S_{m+n} = S'_n$ if andy only if $Y_n = 0$, we find that the process $(\tilde{S}_n)$ defined by
$$ \tilde{S}_n := \begin{cases} S'_n, & n \leq T \\ S_{m+n}, & n > T \end{cases} $$
has the same law as $(S'_n)$. So
$$ h(z) = \mathbb{E}[h(S'_n)] = \mathbb{E}[h(S_{m+n})\mathbb{1}_{\{n > T\}}] + \mathcal{O}(\mathbb{P}(n \leq T)) $$
as $n \to \infty$. Now by the dominated convergence theorem,
$$ h(z) = \lim_{n\to\infty} \mathbb{E}[h(S'_n)] = \mathbb{E}\Bigl[\lim_{n\to\infty} h(S_{m+n})\mathbb{1}_{\{n > T\}}\Bigr] = \mathbb{E}\Bigl[\lim_{n\to\infty} h(S_{n}) \Bigr] = h(0). $$
So $h$ is constant as claimed. However, since $\frac{\log(3/2)}{\log 2}$ is irrational, the set
$$\{ - a \log 2 + b \log(3/2) : a, b \in \{0,1,2,\dots\}\}$$
is dense in $\mathbb{R}$. Since $g$ is continuous and is constant on a dense subset of $\mathbb{R}$, it follows that $g$ is also constant, completing the proof.