Inside /outside test of the point on 2D plane

Solution 1:

For a plane with equation $ax+by+cz+d=0$ defining two halfspaces, a good guideline is as follows:

"Two points $(x_1,y_1,z_1)$ and $(x_2,y_2,z_2)$ are in the same halfspace if and only if $(ax_1+by_1+cz_1+d)$ and $(ax_2+by_2+cz_2+d)$ have the same sign."

If one of them is the position of the observer, and that this position is the origin $(0,0,0)$, this test becomes :

Point $(x_1,y_1,z_1)$ is on the same side of the plane as the observer iff $ax_1+by_1+cz_1+d$ and $d$ have the same sign.

You can call this halfspace the "halfspace of the observer" ("front halfspace" is kind of ambiguous).

Example: Consider the plane with equation $x+y+z-1=0$ (intersecting the axes in $(1,0,0), (0,1,0)$ and $(0,0,1)$), a point like $(x,y,z)=(-2,-3,-4)$ is on the same halfplane as the origin because $-2-3-4-1=-10$ has the same sign as $d=-1$.

Edit 1: If your plane is defined by three points $P,Q,R$ in this order, it defines a natural orientation of the plane, then a natural normal by taking the cross product $\vec{N}=\vec{PQ} \times \vec{QR}$.

Edit 2: If the coefficient $c$ of variable $z$ is non zero, you can extract:

$$z=a'x+b'y+c'$$

In this case, you consider it as a surface, and you have un-ambiguous above and below halfspaces.

Solution 2:

The sign of D tells you on what "side" the origin is:

• When $D = 0$, $Ax+By+Cz+D = 0$ contains the origin. Notice that $(0, 0, 0)$ satisfy the equation $Ax+By+Cz+D = 0$.
• When $D > 0$, $Ax+By+Cz+D > 0$ contains the origin. Notice that replacing $(0, 0, 0)$ in the equation $Ax+By+Cz+D = 0$ leaves you with $D = 0$, and $D$ is positive. Meaning that the origin satisfies $Ax+By+Cz+D > 0$ when $D$ is positive.
• When $D < 0$, $Ax+By+Cz+D < 0$ contains the origin. By the same logic as above, replacing $(0, 0, 0)$ in the equation $Ax+By+Cz+D = 0$ leaves you with $D = 0$, but in this case $D$ is negative. Meaning that the origin satisfies $Ax+By+Cz+D < 0$ when $D$ is negative.

And, of course, points that give you the same sign are on the same side. See also: How to determine if a point lies on the same side of the plane that origin does?

And I remind you that you could flip all the sings on a plane equation (that is, go form $Ax+By+Cz+D = 0$ to $-Ax-By-Cz-D = 0$) and it defines the same plane. However, you flipped the "sides".

From there, it is convention. What convention?

For a clear example, if we are working in camera space and we want to know if something is "behind" a plane. Where "behind" means on the other side from the camera. Remember that the camera in camera space is at the origin. So "behind" is the side that does not contain the origin.

Be aware that different conventions might apply to different contexts. If in doubt, pick one and stick to it.