Infinite self-convolution for a function

While I don't know about the general case of your $F^{(\infty)}(x)$ for any such generic function $f$, it seems that in some cases you can find explicit closed forms for $n$-fold convolutions for some functions $f$. Maybe then you can study the limit as $n\rightarrow\infty$ for your purposes?

For example, I found useful information in Grindstead and Snell's Introduction to Probability which has a page about this question for the case of a finite number of convolutions (chapter 7, pg 300 in my copy) and lists the equation

$$ f_{S_n}(x)=\frac{1}{\sqrt{2\pi n}}e^{-x^2/2n} $$

representing $n$ convolutions of a Gaussian, with $S_n=X_1+X_2+\dots+X_n$ where $X_1,X_2,\dots,X_n$ are independent Gaussian random variables with mean 0 and variance 1. $S_n$ is defined by their $n$-fold convolution as $f_{S_n}=(f_{X_1}*f_{X_2}*\dots*f_{X_n})(x)$. So for your Gaussian experiment, maybe this can provide some insight about the convergence to zero.

The book shows similar examples for uniform and exponential random variables. They note that this can be done for certain cases, implying that it is not generally applicable. They also provide the following reference for additional information (which I have not studied): J. B. Uspensky, Introduction to Mathematical Probability (New York: McGraw-Hill, 1937), p. 277.

I had a similar question for years. Only recently I was able to solve. So here you go.

As you have mentioned, you can assume $f$ as a pdf of a random variable multiplied by a scaling factor, since it satisfies all the required properties you've mentioned.

So following the approach, let me first consider a function $f(x)$, which is a pdf of a random variable $X$. Also consider a sequence of $n$ random variables, $X_1 , X_2 , X_3 , \dots , X_n $ that are iid ( Independent and Identically Distributed RVs ) with a pdf $f(x)$.

Now Central Limit Theorem says that \begin{equation} Y = \frac{1}{\sqrt n} \sum\limits_{i=1}^{n} X_i \end{equation} converges in probability to a normal distribution as $n$ approaches $\infty$. But by sum property of random variable, the pdf of the random variable $Y$ is simply $\frac{1}{\sqrt n} ( f(x)*f(x)*\dots f(x)) $.

This means that in your case $F^{\infty}(x)$ converges to $\sqrt n a^n \mathcal{N} (\mu,\sigma)$, which tends to $0$ as $n$ tends to $\infty$, if $|a| \leq 1 $, where $a$ is the scaling factor required to normalize the area under the curve to $1$ for the equivalent pdf. This is the reason why your function becomes flatter and flatter with increasing $n$. Now try the same experiment after normalizing the function with $ \sqrt n a^n$, you must get a smooth bell curve.Hope it helps.