What is the probability that a point chosen randomly from inside an equilateral triangle is closer to the center than to any of the edges?

You are right to think of the probabilities as areas, but the set of points closer to the center is not a triangle. It's actually a weird shape with three curved edges, and the curves are parabolas.

The set of points equidistant from a line $D$ and a fixed point $F$ is a parabola. The point $F$ is called the focus of the parabola, and the line $D$ is called the directrix. You can read more about that here.

In your problem, if we think of the center of the triangle $T$ as the focus, then we can extend each of the three edges to give three lines that correspond to the directrices of three parabolas.

Any point inside the area enclosed by the three parabolas will be closer to the center of $T$ than to any of the edges of $T$. The answer to your question is therefore the area enclosed by the three parabolas, divided by the area of the triangle.

Let's call $F$ the center of $T$. Let $A$, $B$, $C$, $D$, $G$, and $H$ be points as labeled in this diagram:

The probability you're looking for is the same as the probability that a point chosen at random from $\triangle CFD$ is closer to $F$ than to edge $CD$. The green parabola is the set of points that are the same distance to $F$ as to edge $CD$.

Without loss of generality, we may assume that point $C$ is the origin $(0,0)$ and that the triangle has side length $1$. Let $f(x)$ be equation describing the parabola in green.

By similarity, we see that $$\overline{CG}=\overline{GH}=\overline{HD}=1/3$$

An equilateral triangle with side length $1$ has area $\sqrt{3}/4$, so that means $\triangle CFD$ has area $\sqrt{3}/12$. The sum of the areas of $\triangle CAG$ and $\triangle DBH$ must be four ninths of that, or $\sqrt{3}/27$.

$$P\left(\text{point is closer to center}\right) = \displaystyle\frac{\frac{\sqrt{3}}{12} - \frac{\sqrt{3}}{27} - \displaystyle\int_{1/3}^{2/3} f(x) \,\mathrm{d}x}{\sqrt{3}/12}$$

We know three points that the parabola $f(x)$ passes through. This lets us create a system of equations with three variables (the coefficients of $f(x)$) and three equations. This gives

$$f(x) = \sqrt{3}x^2 - \sqrt{3}x + \frac{\sqrt{3}}{3}$$

The integral of this function from $1/3$ to $2/3$ is $$\int_{1/3}^{2/3} \left(\sqrt{3}x^2 - \sqrt{3}x + \frac{\sqrt{3}}{3}\right) \,\mathrm{d}x = \frac{5}{54\sqrt{3}}$$

This gives our final answer of $$P\left(\text{point is closer to center}\right) = \boxed{\frac{5}{27}}$$

In response to Benjamin Dickman's request for a solution without calculus, referring to dtldarek's nice diagram in Zubin Mukerjee's answer (with all areas relative to that of the triangle $FCD$):

The points $A$ and $B$ are one third along the bisectors from $F$, so the triangle $FAB$ has area $\frac19$. The vertex $V$ of the parabola is half-way between $F$ and the side $CD$, so the triangle $VAB$ has width $\frac13$ of $FCD$ and height $\frac16$ of $FCD$ and thus area $\frac1{18}$. By Archimedes' quadrature of the parabola (long predating the advent of calculus), the area between $AB$ and the parabola is $\frac43$ of the area of $VAB$. Thus the total area in $FCD$ closer to $F$ than to $CD$ is

$$ \frac19+\frac43\cdot\frac1{18}=\frac5{27}\;. $$

P.S.: Like Dominic108's solution, this is readily generalized to a regular $n$-gon. Let $\phi=\frac\pi n$. Then the condition $FB=BH$, expressed in terms of the height $h$ of triangle $FAB$ relative to that of $FCD$, is

$$ \frac h{\cos\phi}=1-h\;,\\ h=\frac{\cos\phi}{1+\cos\phi}\;. $$

This is also the width of $FAB$ relative to that of $FCD$. The height of the arc of the parabola between $A$ and $B$ is $\frac12-h$. Thus, the proportion of the area of triangle $FCD$ that's closer to $F$ than to $CD$ is

$$ h^2+\frac43h\left(\frac12-h\right)=\frac23h-\frac13h^2=\frac{2\cos\phi(1+\cos\phi)-\cos^2\phi}{3(1+\cos\phi)^2}=\frac13-\frac1{12\cos^4\frac\phi2}\;. $$

This doesn't seem to take rational values except for $n=3$ and for $n\to\infty$, where the limit is $\frac13-\frac1{12}=\frac14$, the value for the circle.

The problem can be solved without calculus by using Archimedes Quadrature of the Parabola. To prove this Archimedes did, however, use techniques that are closely related to modern calculus.

According to Archimedes Quadrature of the Parabola the area between the parabola and the edge EF is $4\over3$ the area of triangle EFG.

Let's assume that EK has length x, then due to similarity DI has length ${3\over2}x$ and DH has length ${1\over2}x$. Because point G is on the parabola we have that the length of DG is half the length of DI, hence the length of DG is ${3\over4}x$. Now the length of HG is ${3\over4}x-{1\over2}x={1\over4}x$. So the length of DH is twice the length of HG

From this it follows that the area of triangle EFG is half the area of triangle EFD. The area between EF and the parabola is ${4\over3}\cdot{1\over2}={2\over3}$ times the area of triangle EFD.

As the area of triangle EFD is equal to ${1\over27}$ the area of triangle ABC we now find that the area of points closer to the middle then to the edges is equal to $3\cdot1{2\over3}\cdot{1\over27}={5\over27}$ the area of triangle ABC. Hence P(point is closer to the center)=${5\over27}$.

I am just passing by here. Perhaps the idea is to have something elegant, but this is just straightforward: an integral in polar coordinates with the help of Mathematica. I reuse the picture of the solution of Zubin. Let J be the midpoint of CD. By symmetry, we can restrict the analysis to CFJ. Without loss of generality, let the length of FJ be $1$. The total area of CFJ is $\sqrt{3}/2$. We need the area above the curve in CFJ. Pick an arbitrary point L on the curve inside CFJ. Let $\theta$ be the angle between FJ and FL. Some trigonometry gives us that the length of FL is $x = (1+ \cos\theta)^{-1}$. (It helps to draw the straight line that extends FL and meets CJ at some point K and note that the length of FK is $1/\cos \theta$.) So, the area above the curve is the integral of $(1+\cos \theta)^{-2}/2$ from $0$ to $\pi/3$. Using mathematica, the primitive is $((2 + \cos\theta)\sin\theta)/(6(1 + \cos\theta)^2)$. So, the area above the curve is $((2 + \cos(\pi/3))\sin(\pi/3))/(6(1 + \cos(\pi/3))^2) = 5 \sqrt 3/ 54$ . So, the probability to be above the curve is $(5 \sqrt 3/ 54)/(\sqrt 3/2) = 5/27$.

This solution generalizes to any regular polygone. We integrate from $0$ to $\pi/n$ instead of from $0$ to $\pi/3$ and the total area of CFJ becomes $\tan(\pi/n)/2$ instead of $\sqrt 3/2$, every thing else stays the same.