Is the function characterized by $f(\alpha x+(1-\alpha) y) \le f^{\alpha}(\alpha x)f^{1-\alpha}(y)$ convex?
Solution 1:
[Note. This is an edited answer based on the helpful discussions with OP, as shown in the comments.]
Thanks for the interesting question. Here are some of my observations, and please let me know if I did something wrong. Thanks.
Observation 1
With $x = y$, we have from eq. 1, $$ f(y) \le f^{\alpha}(\alpha y) \, f^{1-\alpha}(y). $$ Since $f(y)$ is nonnegative, $$ f(y) \le f(\alpha y). \qquad (1) $$ This means the function $f(y)$ is decreasing for $y \ge 0$ and increasing for $y \le 0$.
A corollary is that $$ f(x) \le f(0). \qquad (2) $$
Observation 2
Let $y < 0$ and $\alpha x = (1 - \alpha) (-y) > 0$, we have $$ f(0) \le f^{\alpha}(\alpha x)\, f^{1-\alpha}(y). $$ But by (2) we have $$ \begin{align} f(\alpha x) &\le f(0), \\ f(y) &\le f(0). \end{align} $$ So $$ f(0) \le f^{\alpha}(\alpha x)\, f^{1-\alpha}(y) \le f(0). $$ This is possible only if both $f(\alpha x)$ and $f(y)$ are equal to $f(0)$. In other words, $f(x)$ is a constant.
Observation 3
The above argument shows that eq. 1 is a very stringent condition on $f$, if $y$ can take any real number. However, if we impose the restriction $y \ge 0$, any monotonically decreasing function satisfies the requirement, eq. 1. This is because $$ \begin{align} f(y) &\ge f(y + \alpha \, (x-y)), \\ f(\alpha x) &\ge f(\alpha x + (1 - \alpha) \, y), \end{align} $$ So $$ f^{1-\alpha}(y)f^\alpha(\alpha x) \ge f(\alpha x + (1- \alpha)y). $$