What is the probability of having a pentagon in 6 points

If the probability that $5$ random points in the plane whose horizontal coordinate and vertical coordinate are uniformly distributed on the interval $\left(0,1\right)$ occur to be the vertices of a convex pentagon is $\frac{49}{144}$, what is the probability that a subset of $6$ random points in the plane whose horizontal coordinate and vertical coordinate are uniformly distributed on the interval $\left(0,1\right)$ occurs to be the vertices of a convex pentagon? Thanks a lot.

Solution 1:

This is the same as the probability that no two of the six points lie within the convex hull of the other four.

The number of cases for this complementary event to happen is $${6\choose2}=15$$ and all these cases are (with 100% probability) mutually exclusive and equiprobable, except that we have triple-counted the cases when three of the points actually lie within the convex hull of the other three, of which there are $${6\choose3}=20.$$ These cases are also all equiprobable, they are mutually exclusive except on a set of null measure, and they have to be re-included twice. This completes the inclusion-exclusion.

Given this, it is easy to solve the original problem if we can calculate the probability distribution of the area of the convex hull of the first four randomly chosen points and the marginal probability q that the last two uniformly and independently chosen points both lie within this area, and likewise the marginal probability r that the last three points all lie within the convex hull of the first three.

Then the probability that any five of the six points chosen uniformly and independently on $(0,1)\times(0,1)$ form a convex pentagon is $$p=1-15q+40r$$

So let $H_3(x)$ be the marginal cumulative distribution function of the area of the convex hull of the first three points, and $H_4(x)$ be the marginal cdf of the area of the convex hull of the first four points. Then $$q=\int_0^1 x^2dH_4(x);$$ and $$r=\int_0^1 x^3dH_3(x).$$ Computing $H_3(x)$ and $H_4(x)$ is tedious but straightforward calculus. Otherwise the problem is solved.

(Small hint: it may be easier to compute $H_4(x)$ after computing $H_3(x)$. The integrals are actually the second and third moments about the origin for these distributions, respectively. They are taken over the Kolmogorov space $[0,1]$ as the cdf ranges from 0 to 1.)

For $H_3(x)$ see The PDF of the area of a random triangle in a square. For $H_4(x)$ see PDF of area of convex hull of four random points in square.