Detecting polygon self intersection

Solution 1:

There is the obvious algorithm of comparing all pairs of edges, which is $O(n^2)$ but probably is ok for small polygons. There is the Bentley–Ottmann algorithm sweep algorithm, which is $O(n \log n)$ but is harder to implement and probably only needed if $n$ is large. I think there is a $O(n)$ algorithm by Chazelle, which is very likely to be impractical.

In any case, note that you can test whether two line segments intersect without finding the intersection point. It's a simple matter of comparing signs of a few determinants. See http://algs4.cs.princeton.edu/91primitives/.

Solution 2:

The Shamos-Hoey algorithm is designed for the task of simply reporting whether a polygon has self-intersections or not.

There is an excellent writeup in this blog and there are many implementations available (include my own in javascript which also includes the original PDF of the paper).