Proving a sample mean converges in probability to the true mean

Solution 1:

Use Chebyshev's inequality.

First recall Markov's inequality: If $\Pr(V\ge0)=1$ and $\mathbb E V=\alpha$ then $\Pr(V\ge x\alpha)\le\dfrac 1 x$. E.g. if incomes are non-negative, then no more than $1/15$ of the population can have more than $15$ times the average income, etc.

Apply this to $(X-\mu)^2$ where $\mu = \mathbb EX$ and $\sigma^2=\mathbb E((X-\mu)^2)<\infty$.

Markov's inequality then tells is that $\Pr((X-\mu)^2\ge x^2 \sigma^2)\le \dfrac 1 {x^2}$.

Consequently $\Pr\left(\dfrac{|X-\mu|}{\sigma}\ge x\right)\le \dfrac 1 {x^2}$. That is Chebyshev's inequality.

Let $\bar X = (X_1+\cdots+X_n)/n$, and recall that $\mathbb E \bar X=\mu$ and $\operatorname{var}(\bar X)=\sigma^2/n$. (This assumes $X_1,X_2,X_3,\ldots$ all have the same expected value and the same finite variance, and are uncorrelated. It is not necessary to assume they all have the same distribution, nor that they are independent, although those assumptions are certainly sufficient, since they are stronger than the assumptions we're making here.)

Applying Chebyshev's inequality to $\bar X$, we get $$ \Pr\left(\frac{|\bar X-\mu|}{\sigma/\sqrt{n}}\ge x\right) \le \frac {1}{x^2}, $$ so $$ \Pr\left(|\bar X-\mu|\ge \frac{x\sigma}{\sqrt{n}}\right) \le \frac 1 {x^2}. $$ This holds if $x$ is any positive number at all; hence it holds when $x=\varepsilon\sqrt{n}/\sigma$, where $\varepsilon$ is any positive number. That gives us $$ \Pr\left(|\bar X-\mu|\ge \varepsilon\right) \le \frac{\sigma^2}{\varepsilon^2 n}\to0\text{ as }n\to\infty. $$

All this works in cases where $\sigma^2<\infty$. In cases where that does not hold, there are counterexamples to the conclusion we're getting here, but I think some weaker assumptions than those we've made might be enough.

Solution 2:

You need to distinguish between the two concepts: the "true" mean of the underlying distribution, i.e. $\mathbb{E}(X_1)$ and the sample mean $\frac{1}{n}\sum X_i$. The goal is to show that $\frac{1}{n}\sum X_i \rightarrow \mathbb{E}(X_1)$ in probability as $n$ tends to infinity.