Formal / academic sounding phrase for "uncharted territory"
At least you are avoiding the mistake of "Unchartered territory" I hear more frequently.
I think uncharted territory is plenty formal. It is well understood and does not sound the least casual. One alternative is Terra Incognita which only means land we do not know. Though it sounds remote it is really just other land, not a magic place. It includes the elements of unknown and uncertainty you are looking for. It does bristle with Latinate which can be annoying to the larger herbivores. With that you could just say New land and be done with it.