How to prove that geometric distributions converge to an exponential distribution?

Solution 1:

Recall pmf of the geometric distribution: $\mathbb{P}(X = k) = p (1-p)^k$ for $k \geq 0$.

The geometric distribution has the interpretation of the number of failures in a sequence of Bernoulli trials until the first success.

Consider a regime when the probability of success is very small, such that $n p = \lambda$, and consider $x = \frac{k}{n}$. Then, in the large $n$ limit: $$ 1 = \sum_{k=0}^\infty \mathbb{P}(X = k) = \sum_{k=0}^\infty \lambda \left(\left(1 - \frac{\lambda}{n} \right)^{n \cdot k/n} \frac{1}{n} \right) \stackrel{n \to \infty}\rightarrow \int_0^\infty \lambda \mathrm{e}^{-\lambda x} \mathrm{d} x $$

Alternatively, you could look at the moment generating function for the geometric distribution: $$ \mathcal{M}(p, t) = \frac{p}{1-\mathrm{e}^t (1-p)} $$ To recover the mgf of the exponential distribution consider the limit: $$ \lim_{n \to \infty} \mathcal{M}\left( \frac{\lambda}{n}, \frac{t}{n} \right) = \lim_{n \to \infty} \frac{\lambda}{n - \mathrm{e}^{t/n}\left(n - \lambda\right)} = \lim_{n \to \infty} \frac{\lambda}{n \left(1 - \mathrm{e}^{t/n}\right) + \lambda \mathrm{e}^{t/n} } = \frac{\lambda}{\lambda-t} $$ which is the mgf of the exponential distribution with rate $\lambda$.

Solution 2:

It may be useful to construct the relevant distributions the other way round, that is, to start from a single exponential random variable $X_\lambda$ with density $\lambda\mathrm e^{-\lambda x}$ on $x\geqslant0$, for a given positive $\lambda$, and to consider the integer part $Y_a$ of $X_\lambda/a$, for every positive $a$.

Then, $[Y_a=n]=[na\leqslant X_\lambda\lt (n+1)a]$ hence $\mathrm P(Y_a=n)=\mathrm e^{-\lambda na}-\mathrm e^{-\lambda (n+1)a}$ for every nonnegative integer $n$, that is, $$\mathrm P(Y_a=n)=(p_a)^n(1-p_a),$$ with $$ p_a=\mathrm e^{-\lambda a}. $$ This proves that each $Y_a$ is geometric with parameter $p_a$. Note that, for a given $\lambda$, every parameter $p$ in $(0,1)$ is realized as $p=p_a$ for some $a$, hence, starting from a unique random variable $X_\lambda$, this construction yields a whole family of geometric random variables $(Z_p)_{p\in(0,1)}$ such that each $Z_p$ is geometric with parameter $p$ and such that $Z_p\leqslant Z_q$ with full probability, for every $p\geqslant q$ in $(0,1)$: simply define each $Z_p$ by $Z_p=Y_{\alpha(p)}$, with $$ \alpha(p)=-\log(p)/\lambda. $$ Coming back to your question, $X_\lambda\leqslant aY_a\lt X_\lambda+a$ with full probability, hence $aY_a\to X_\lambda$ almost surely when $a\to0$, in particular $\alpha(p)Z_p\to X_\lambda$ when $p\to1$. Thus, $-\log(p)Z_p$ converges almost surely (hence in distribution) to the standard exponential random variable $X_1=\lambda X_\lambda$.

Since $-\log(p)\sim1-p$ when $p\to1$, this shows that $(1-p)Z_p$ converges almost surely (hence also in distribution) to $X_1$. Now, the convergence in distribution of some random variables does not depend on their realization hence, by the remarks above, for every family $(T_p)_{p\in(0,1)}$ of random variables such that each $T_p$ is geometrically distributed with parameter $p$, the random variables $(1-p)T_p$ converge in distribution to the standard exponential distribution when $p\to1$.

Note Such a simultaneous and almost sure construction of geometric random variables $Z_p$, for every $p$ in $(0,1)$, from a single (exponentially distributed) random variable $X_\lambda$ is called a coupling. For more about this powerful idea, see the WP page on coupling in probability.

Solution 3:

The waiting time $T$ until the first success in a sequence of independent Bernoulli trials with probability $p$ of success in each one has a geometric distribution with parameter $p$: its probability mass function is $P(x) = p (1-p)^{x-1}$ and cumulative distribution function $F(x) = 1 - (1-p)^x$ for positive integers $x$ (note that some authors use a different convention where the random variable is $T-1$ rather than $T$, but that won't make a difference in the limit). The scaled version $p T$ converges in distribution as $p \to 0+$ to an exponential random variable with rate $1$, as for $x \ge 0$ $$ P(p T \le x) = F(x/p) = 1 - (1-p)^{\lfloor x/p\rfloor} \to 1 - e^{-x} $$