How to intersect two polygons?
Solution 1:
Arash Partow's FastGEO library contains implementations of many interesting algorithms in computational geometry. Polygon intersection is one of them. It's written in Pascal, but it's only implementing math so it's pretty readable. Note that you will certainly need to preprocess your edges a little, to get them into clockwise or counterclockwise order.
ETA: But really, the best way to do this is to not do this. Find another way to approach your problem that doesn't involve arbitrary polygon intersections.
Solution 2:
If you are programming in .NET Framework, you may want to take a look at SqlGeometry class available in .NET assemblies shipped as Microsoft SQL Server System CLR Types
The SqlGeometry class provides STIntersection method
SqlGeometry g1 = SqlGeometry.Parse("POLYGON ((...))");
SqlGeometry g2 = SqlGeometry.Parse("POLYGON ((...))");
SqlGeometry intersection = g1.STIntersection(g2);
Solution 3:
What I think you should do
Do not attempt to do this yourself if you can possibly help it. Instead, use one of the many available polygon intersection algorithms that already exist.
I was strongly considering the following codebase on the strength of their demonstration code and the fact that they mentioned their handling of most/all of the weird cases. You would need to donate an amount (of you/your company's choice) if you use it commercially, but it's worth it to get a robust version of this kind of code.
http://www.cs.man.ac.uk/~toby/gpc/
What I actually did was to use a polygon-intersection algorithm that is part of the Java2D libraries. You can possibly find something similar in MS's own C# libraries to use.
There are other options out there as well; look for "polygon clipper" or "polygon clipping", since the same basic algorithms that handle polygon intersection also tend to be usable for the general clipping cases.
Once you actually have a polygon clipping library, you just need to subtract polygon B from polygon A to get your first piece of output, and intersect polygons A and B to get your second piece of output.
How to roll your own, for the hopelessly masochistic
When I was considering rolling my own, I found the Weiler-Atherton algorithm to have the most potential for general polygon-cutting. I used the following as a reference:
http://cs1.bradley.edu/public/jcm/weileratherton.html
http://en.wikipedia.org/wiki/Weiler-Atherton
The details, as they say, are too dense to include here, but I have no doubt that you'll be able to find references on Weiler-Atherton for years to come. Essentially, you split all the points into those that are entering the final polygon or exiting the final polygon, then you form a graph out of all the points, and then walk the graph in the appropriate directions in order to extract all the polygon pieces you want. By changing the way you define and treat the "entering" and "exiting" polygons, you can achieve several possible polygon intersections (AND, OR, XOR, etc.).
It's actually fairly implementable, but like with any computational geometry code, the devil is in the degeneracies.