Why the 20px gap at the top of my UIViewController?

The Problem

Consider the following controller hierarchy:

UINavigationController
- UIViewController
  - UITableViewController

The presence of the UIViewController is affecting layout. Without it, the UITableViewController takes up the entire bounds of the UINavigationController:

enter image description here

However, if I add a vanilla UIViewController between the UINavigationController and UITableViewController, a 20px gap appears between the top of the UIViewController and the top of the UITableViewController:

enter image description here

Even if I reduce my code down to the simplest possible thing, I still observe this behavior. Consider this app delegate code:

public override bool FinishedLaunching(UIApplication app, NSDictionary options)
{
    window = new UIWindow(UIScreen.MainScreen.Bounds);

    var tableView = new UITableViewController();
    var intermediateView = new UIViewController();
    var navigation = new UINavigationController(intermediateView);

    navigation.View.BackgroundColor = UIColor.Red;
    intermediateView.View.BackgroundColor = UIColor.Green;
    tableView.View.BackgroundColor = UIColor.Blue;

    intermediateView.AddChildViewController(tableView);
    intermediateView.View.AddSubview(tableView.View);
    tableView.DidMoveToParentViewController(intermediateView);

    window.RootViewController  = navigation;
    window.MakeKeyAndVisible();

    return true;
}

The above still shows a 20px gap between the top of the UIView and the top of the UITableView.

My Understanding of the Problem

I understand that something is erroneously allocating space for a status bar. Using Reveal I can see that the Frame of the UITableViewController has a Y value of 20.

Things I've Tried

Unsuccessful

Set WantsFullScreenLayout to true on the UIViewController, UITableViewController, and both
Playing with EdgesForExtendedLayout and ExtendedLayoutIncludesOpaqueBars for both the UIViewController and UITableViewController
Played with AutomaticallyAdjustsScrollViewInsets on the UIViewController
Played with PreservesSuperviewLayoutMargins in the UITableView
Tried overriding PrefersStatusBarHidden and returning true in both the UIViewController and UITableViewController

Successful

Overriding ViewDidLayoutSubviews in my UITableViewController thusly:

public override void ViewDidLayoutSubviews()
{
    base.ViewDidLayoutSubviews();
    this.View.Frame = this.View.Superview.Bounds;
}

Things I want to know

is there a clean(er) way of achieving my goal?
what is actually responsible for adding the 20px gap? The UIViewController? The UITableViewController?
what are the best practices for ensuring my view controllers remain usable in different contexts? Presumably overriding ViewDidLayoutSubviews couples my view controller to expectations as to where it will be displayed in the visual tree. If it were to be hosted higher up the controller stack, things would not look right. Is there a way to avoid this coupling and thus increase reusability?

The behaviour you are seeing is not a bug at all but merely a side effect of your misuse of adding views into a hierarchy.

When you add the tableView into a view you need to tell UIKit how you want that tableView to size relative to its parent. You have two options: Auto-Layout or Autoresizing Masks. Without describing how you want your view to layout UIKit simply pops it onto the hierarchy and the default implementation will lay your view under the top layout guide (which just happens to be the height of the status bar). Something as simple as this would do the trick:

tableVC.View.Frame = rootVC.View.Bounds
tableVC.View.Autoresizingmask = UIViewAutoresizing.FlexibleWidth
tableVC.View.TranslatesAutoresizingMaskIntoConstraints = true// always nice to explicitly use auto layout

This behavior is not actually exclusive to UITableViewController but also to UICollectionViewController. I believe their default viewLoading implementation insets the view under the status bar. If we make our childController a simple UIViewController subclass none of this behaviour is exhibited. Don't see this as a bug though, if you explicitly declare how you want their respective views to be laid out you won't have this issue. Naturally, this is the primary function of a container controller.

Heres what your appDelegate should look like:

Xamarin

public override bool FinishedLaunching(UIApplication app, NSDictionary options)
{
    window = new UIWindow(UIScreen.MainScreen.Bounds);

    var tableVC = new UITableViewController();
    var rootVC = new UIViewController();
    var navigationVC = new UINavigationController(intermediateView);

    navigationVC.View.BackgroundColor = UIColor.Red;
    rootVC.View.BackgroundColor = UIColor.Green;
    tableVC.View.BackgroundColor = UIColor.Blue;

    rootVC.AddChildViewController(tableVC);
    rootVC.View.AddSubview(tableVC.View);


    //YOU NEED TO CONFIGURE THE VIEWS FRAME
    //If you comment this out you will see the green view under the status bar
    tableVC.View.Frame = rootVC.View.Bounds
    tableVC.View.Autoresizingmask = UIViewAutoresizing.FlexibleWidth
    tableVC.View.TranslatesAutoresizingMaskIntoConstraints = true
    tableVC.DidMoveToParentViewController(rootVC);

    window.RootViewController  = navigationVC;
    window.MakeKeyAndVisible();

    return true;
}

Swift

var window: UIWindow?
var navigationControlller: UINavigationController!

func application(application: UIApplication, didFinishLaunchingWithOptions launchOptions: [NSObject: AnyObject]?) -> Bool {
    let rootVC = UIViewController(nibName: nil, bundle: nil)
    let tableVC = UITableViewController(style: .Plain)
    let navVC = UINavigationController(rootViewController: rootVC)
    navVC.navigationBarHidden = true

    navVC.view.backgroundColor = UIColor.redColor()
    rootVC.view.backgroundColor = UIColor.greenColor()

    rootVC.view.addSubview(tableVC.view)

    //YOU NEED TO CONFIGURE THE VIEWS FRAME
    //If you comment this out you will see the green view under the status bar
    tableVC.view.frame = rootVC.view.bounds
    tableVC.view.autoresizingMask = .FlexibleWidth | .FlexibleHeight
    tableVC.view.setTranslatesAutoresizingMaskIntoConstraints(true)

    rootVC.addChildViewController(tableVC)
    tableVC.didMoveToParentViewController(rootVC)

    window = UIWindow(frame: UIScreen.mainScreen().bounds)
    window?.rootViewController = navVC
    window?.makeKeyAndVisible()

    return true
}

Note I wrote a post recently describing the ways you can configure a view to fill its superview

Your problem seems to relate to some kind of fundamental issues with UITableViewController being directly added as a child controller of another.

Looking around this is not a new issue: iOS 7: UITableView shows under status bar

Relative to that article, I took your code and tried the following options (once I realized it was in C# 8^)):

Instead of using a UITableViewController, add a UIViewController and then add a UITableView as a child of the UIViewController. If you do this then there is no 20pt issue. This highlights that the problem is with the UITableViewController class.

This approach is a relatively clean workaround and only a couple of extra steps on what you have already.

I did try adding constraints in your original code to force the UITableViewController frame to the top, but could not get this to work. Again this could be down to the table controller overriding things itself.
If you build your demo using storyboards, everything works. This I believe is down to the fact that IB itself uses a container view to embed the new view controller. So if you use storyboard, the way apple does it is to add a view which you can set the frame of using constraints and it then embeds the UITableViewController inside that view via an embed Segue.

Hence as per 1), using a view in the middle seems to solve the issue and again it seems that having control over the middle views frame is key.

I notice in your only viable workaround, that changing the frame was the answer. However post iOS7, changing the frame does not seem to be recommended due to the issues it can have clashing with constraints which also want to manipulate the frame.
Trying other options like edgesForExtendedLayout all seemed to fail. These seem to be hints for container view controllers and UITableViewController is ignoring them.

IMHO I think option 1) seems the safest approach as you have total control over the layout and are not fighting the system with frame overrides which may cause you issues later. Option 2) only really works if you use storyboards. You could try doing the same thing manually yourself, but who knows what goes on in an embed Segue.

EDIT

It would seem that there was a missing step as highlighted by Arkadiusz Holko in his answer and setting the frame explicitly for the table view does fix the issue.

You missed one step when adding a child view controller – setting up its view's frame.

See the second step:

- (void) displayContentController: (UIViewController*) content;
{
  [self addChildViewController:content];                 // 1
  content.view.frame = [self frameForContentController]; // 2
  [self.view addSubview:self.currentClientView];
  [content didMoveToParentViewController:self];          // 3
}

Source: https://developer.apple.com/library/ios/featuredarticles/ViewControllerPGforiPhoneOS/CreatingCustomContainerViewControllers/CreatingCustomContainerViewControllers.html

20 pixel is taken by the status bar. If you are using an xib file or storyboard with auto layout you can set the top constraint to top layout guide so that the 20 pixel difference is handled

Is there a clean(er) way of achieving my goal?

Just change the frame.

tableView.View.Frame = intermediateView.View.Bounds;
tableView.View.AutoresizingMask = UIViewAutoresizingFlexibleHeight | UIViewAutoresizingFlexibleWidth;

what is actually responsible for adding the 20px gap? The UIViewController? The UITableViewController?

You can log UIViewController.view.frame after you new a UIViewController. You can see UITableViewController.view.frame always have a 20px gap. Why? I think Apple just initialize UITableViewController.view with screen size with a 20px top padding.

what are the best practices for ensuring my view controllers remain usable in different contexts? ...

If you want to add a UIViewController.view to another UIViewController.view.The best way is use story board and use Container View.

If you don't want or can't use story board. I suggest just subclass UIView. addChildViewController sometimes have annoying problems with life circle and layout.