calculating distance between a point and a rectangular box (nearest point)

Here is a single formula that avoids all the case logic. (I happen to be working in JS right now, so here's a JS implementation). Let rect = {max:{x:_, y:_}, min:{x:_, y:_}} and p={x:_, y:_}

function distance(rect, p) {
  var dx = Math.max(rect.min.x - p.x, 0, p.x - rect.max.x);
  var dy = Math.max(rect.min.y - p.y, 0, p.y - rect.max.y);
  return Math.sqrt(dx*dx + dy*dy);
}

Explanation: This breaks down the problem into calculating the x distance dx and the y distance dy. It then uses distance formula.

For calculating dx, here is how that works. (dy is analogous)

Look at the tuple being provided to the max function: (min-p, 0, p-max). Let's designate this tuple (a,b,c).

If p is left of min, then we have p < min < max, which means the tuple will evaluate to (+,0,-), and so the max function will correctly return a = min - p.

If p is between min and max, then we have min < p < max, which means the tuple will evaluate to (-,0,-). So again, the max function will correctly return b = 0.

Lastly, if p is to the right of max, then we have, min < max < p, and the tuple evaluates to (-,0,+). Once again, Math.max correctly returns c = p - max.

So it turns out all the case logic is taken care of by Math.max, which leads to a nice 3-line, control-flow-free function.

I think you need to analyze cases; there's no single formula. It's easier to illustrate in two dimensions:

1          2          3
    +-------------+
    |             |
4   |      0      |   5
    |             |
    +-------------+
6          7          8

The edges of the box (extended) divide the outside into 9 regions. Region 0 (inside the box) is solved by computing the distance to each edge and taking the minimum. Every point in region 1 is closest to the top left vertex, and similarly for regions 3, 6, and 8. For regions 2, 4, 5, and 7, you need to find the distance from the point to the closest edge, which is a fairly simple problem. You can determine which region a point is in by classifying it with respect to each edge. (It's easier to see how to do this by directing the edges say, counter-clockwise.) This will also tell you if the point is inside the box.

In 3D, the logic is exactly the same except that you classify with respect to the six faces and you have more cases.

The problem is simpler if the edges of the box are parallel to the coordinate axes.