How Do You Cut the Maximum Volume Cuboid from a General 3D Shape?
A search for largest rectangle in a polygon shows that this problem has been studied in the plane.
Here is a solution that would find the rectangle in your illustration (it's a computer science student project, well done and well documented).
I suspect the three dimensional problem is a fair bit harder.
(This link only answer posted at the OP's request.)