Solution 1:

Let $X$ be the anticlockwise distance along the circle's circumference from $B$ to $A$. Let $Y$ be the anticlockwise distance from $B$ to $C$.

Since the three points are independently uniformly distributed along the circumference of the circle, $X$ and $Y$ are iid variables with uniform distributions on $[0,1).$

Since the clockwise arc from $B$ to $A$ is $1 - X,$ the distance from $B$ to $A$ is the smaller of $X$ and $1-X$. That is, if we let $X_m = \min\{X, 1 - X\}$, then $X_m$ is the distance from $B$ to $A$.

Similarly, let $Y_m = \min\{Y, 1 - Y\}$, then $Y_m$ is the distance from $B$ to $C$.

By symmetry, $X_m \in [0,\frac12]$ and $X_m$ is uniformly distributed (glossing over the fact that the zero-probability events $X_m=0$ and $X_m=\frac12$ can each happen in only one way). Likewise, $Y_m$ also is uniform on $[0,\frac12]$. Moreover, $X_m$ and $Y_m$ are iid.

The nearest neighbor of $B$ is $A$ if $X_m < Y_m$, but is $C$ if $X_m > Y_m$. So $W = \min\{ X_m, Y_m \}.$

So we want the pdf (or cdf) of the minimum of two iid variables uniformly distributed on $[0,\frac12]$. For $0 \leq w \leq \frac12$ we have

\begin{align} P(W < w) &= 1 - P(W \geq w)\\ & = 1 - P(X_m \geq w) P(Y_m \geq w) \\ & = 1 - \left(2\left(\frac 12 - w\right)\right)^2 \end{align}

with $P(W < 0) = 0$ and $P(W < 1) = 1.$ The cdf is therefore $$ F_W(w) = 1 - (1 - 2w)^2. $$

The pdf is easily found from this.

Solution 2:

I will assume that the unit circle means a circle of unit length. Otherwise you should rescale the quantities by the factor $2\pi$.

Let the distance to the closest point be $w$. It leaves the other point the configuration space of $1-2w$.

Therefore the pdf of the least distance is: $$ \begin{cases} 4(1-2w),&w<0.5\\ 0,&w>0.5 \end{cases} $$

Solution 3:

I'm going to use David K's $X$ and $Y$ but compute $F_W(w)$ differently for $w \in (0,\frac12)$. I don't use $X_m$, $Y_m$.

$$P(W \ge w) = P(X \ge w, 1-X \ge w, Y \ge w, 1-Y \ge w) = P(w \le X \le 1-w, w \le Y \le 1-w)$$

$$= P(w \le X \le 1-w) P(w \le Y \le 1-w)$$

$$= \int_w^{1-w} 1_{(0,1)}(x) dx \int_w^{1-w} 1_{(0,1)}(y) dy$$

$$= (1-2w)^2$$

$$ \iff F_W(w) = 1-(1-2w)^2$$