Example for finitely additive but not countably additive probability measure
Let $\mathcal{U}$ be a free ultrafilter on $\mathbb{N}$. Let $P(A)=1$ if $A\in\mathcal{U}$ and $P(A)=0$ if $A\notin\mathcal{U}$. I think it is impossible to give an explicit example of a finitely additive measure on a $\sigma$-algebra that is not countably additive, but our resident set theorists might be able to tell you more about that.
This question was from years ago, but I was just about to ask a similar question (I found this page from the stackexchange list of similar questions). My own question is whether it is possible to have an explicit example. The answers above are all non-explicit. Here is another (non-explicit) answer in a different form that I found to be helpful. It uses a Banach limit from functional analysis.
Define the natural numbers $\mathbb{N} = \{1, 2, 3, \ldots\}$ and define $2^{\mathbb{N}}$ as the set of all subsets of $\mathbb{N}$. Define $P:2^{\mathbb{N}}\rightarrow\mathbb{R}$ as follows: For each set $A \subseteq \mathbb{N}$, define $P(A)$ as a Banach limit of the sequence $\left\{\frac{|A \cap \{1, 2, ..., k\}|}{k}\right\}_{k=1}^{\infty}$.
Banach limit properties:
A Banach limit can be proven to exist and to have the following properties:
1) It is defined for all bounded real-valued sequences $\{x_k\}_{k=1}^{\infty}$, regardless of whether or not $x_k$ has a limit. In fact, the Banach limit is always a real number between $\liminf_{k\rightarrow\infty} x_k$ and $\limsup_{k\rightarrow\infty} x_k$.
2) The Banach limit is the same as the regular limit whenever the regular limit exists.
3) The Banach limit of a sum of two bounded sequences is the sum of the Banach limits of the individual sequences.
4) The Banach limit is nonnegative whenever $x_k \geq 0$ for all $k$.
There are many ways to define a real-valued function on bounded sequences that satisfies the above properties, so it is implicitly assumed that we consistently use one such function. The value of that function on a given bounded sequence is what we shall call the "Banach limit" of that sequence. Proofs of existence of such functions use nonexplicit things like axiom of choice or ultrafilters.
Using these properties:
Now if $A$ is a finite subset of $\mathbb{N}$ then $\frac{|A \cap \{1, ..., k\}|}{k} \leq \frac{|A|}{k}\rightarrow 0$, so the limit exists and $P(A)=0$. In particular, $P(\{n\})=0$ for all $n \in \mathbb{N}$. So:
$$ 1=P[\mathbb{N}] = P[\cup_{n=1}^{\infty} \{n\}] \neq \sum_{n=1}^{\infty}P[\{n\}]=0$$
Furthermore, $P(A)$ is nonnegative for all $A\subseteq \mathbb{N}$ (by the 4th property of Banach limits above). It also satisfies $P(A \cup B)=P(A)+P(B)$ whenever $A$ and $B$ are disjoint (which can be shown by the 3rd property above). So this $P(A)$ function is indeed a finitely-additive measure on all subsets of $\mathbb{N}$, but not a countably-additive one.
Remaining question:
The above pushes a bit more towards an explicit answer, but still uses Banach limits and hence is not explicit. Can a more explicit answer can be given? Now I'm not sure if I should formally ask this question on stackexchange or not, I suspect I would just get pointers back to your question.