Deciding which colliding segment to use as impact vector for reflection

Using the following illustration.

enter image description here

The box moves along the vector A-B. Calculate the intercepts between A-B and E-F and E-J as units along E-F, or E-J The point of intercept C will in the illustration be between lines A-B and E-J

ABx = B.x - A.x
ABy = B.y - A.y
uu = ABx * (A.y - E.y) - ABy * (A.x - E.x)

EFx = F.x - E.x
EFy = F.y - E.y
c = ABx * EFy - ABy * EFx
if (c !== 0) {
    u = uu / c
    if (u >= 0 && u <= 1) {
        // hits line E-F
    }
}

EJx = J.x - E.x
EJy = J.y - E.y
c = ABx * EJy - ABy * EJx
if (c !== 0) {
    u = uu / c
    if (u >= 0 && u <= 1) {
        // hits line E-J
    }
}

If the boxes are axis aligned AABB then you can simplify by canceling out zeros

ABx = B.x - A.x
ABy = B.y - A.y
uu = ABx * (A.y - E.y) - ABy * (A.x - E.x)

c = -ABy * (F.x - E.x)  
if (c !== 0) {
    u = uu / c
    if (u >= 0 && u <= 1) {
        // hits line E-F
    }
}

c = ABx * (J.y - E.y) 
if (c !== 0) {
    u = uu / c
    if (u >= 0 && u <= 1) {
        // hits line E-J
    }
}

Note that point E represent the common corner as the test is only between the two sides you wish to test. You will never need to test more than two sides.


The accurate way to do this is to treat objects in motion as a function of time.

Currently, you do a full update's worth of motion, then check if the objects overlap:

(x, y) := (x+dx, y+dy)

Instead, you can treat the object's position as a function of time (from the previous frame to new one). This makes the output of the collision logic also a function of time: starting at "no" collision (as you have already checked the previous frame), and possibly changing to "yes" before the end of the update period.

pos(t) = (x+t*dx, y+t*dy)  # for t in [0..1]

Solving for earliest collision point should also let you determine which side the collision is on: each side will have a collision condition, and the first one that fails is the one that is hit. In addition, this method avoids the case where your objects are moving so fast that they can pass through each other.