Solution 1:

Chandru has provided the link to a complete solution, but as Eric notes, it is somewhat involved. Some months ago, being unaware of that solution, I had found a less involved proof if you assume that $f$ is not only continuous but also differentiable. I reproduce that proof below.

Let $\triangle$ be the set of points inside or on the triangle, and $a, b, c : \triangle \to [0,1]$ be scalar fields on $\triangle$ mapping any point $P$ to the distances $\overline{AP}$, $\overline{BP}$, $\overline{CP}$ respectively. Define the scalar field $$g := f \circ a + f \circ b + f \circ c = 0.$$ It is easily shown that $\nabla a$ is a unit vector pointing away from $A$, and similarly for $b$ and $c$. Then $$\nabla g = f'(a) \nabla a + f'(b) \nabla b + f'(c) \nabla c,$$ where $f'(x) = df/dx$. But $g$ is constant, so $\nabla g = 0$.

Take $P$ to be a point on the side $AB$. Then $\nabla a$ and $\nabla b$ are antiparallel, while $\nabla c$ is linearly independent. For their linear combination to be $0$, the coefficient of $\nabla c$, which is $f'(c)$, must be $0$. Since $P$ is an arbitrary point on $AB$, $c$ is any value between $\sqrt{3}/2$ and $1$, so over that range $f'$ is zero, and $f$ is constant.

Once we've shown that $f'$ is zero between $\sqrt{3}/2$ and $1$, for any other value $a$ it's fairly easy to pick a point inside the triangle that's $a$ units from $A$ and over $\sqrt{3}/2$ units from $C$. Now $\nabla g = f'(a) \nabla a + f'(b) \nabla b + f'(c) \nabla c = 0$. The last term drops out because $f'(c) = 0$. $\nabla a$ and $\nabla b$ are linearly independent in the interior of the triangle, so their coefficients $f'(a)$ and $f'(b)$ must be zero. Thus $f' = 0$ everywhere, so $f$ is constant. In particular, $f(1/\sqrt{3}) = 0$, so $f = 0$ everywhere.

Solution 2:

This is a problem asked in the American Mathematical Monthly. May be this link could be of some help:

  • http://www.jstor.org/stable/2323932

Solution 3:

Though Chandru has provided the link to the solution, I was wondering if the following approach could be of any help.

Instead of choosing $P$ to coincide with a vertex, let $P$ to fall on one of the sides say $AB$.

Let $AP = x$. Since $ABC$ is an equilateral triangle $BP = 1-x$ and $CP = \sqrt{1+x^2 - x}$.

So $\forall x \in [0,1]$, the function has to satisfy $$f(x) + f(1-x) + f(\sqrt{1+x^2 - x}) = 0$$

Can we squeeze a proof that $f(x)$ has to be zero out of this?