Can "let us" always be used in place of "let's"?

Me: Perhaps we need to make a left turn at Albuquerque
Him: Let us try that

Now I would have said, "Let's try that". "Let us" sounds wrong to me in this instance. Is it? Are there contractions which are improper to use in their full form? I've always thought that in formal writing contractions are avoided. Are there any contractions that should be used even in formal writing?

Let us in your example only sounds wrong to you because it is very rarely used in that context. It is, however, correct.

One might furthermore argue that any in situation in which let us would be incorrect, let's, its contraction, would also be incorrect. (A contraction should only be used where its umm... anticontraction would also be correct.)

In your example of casual conversation, I would say "let's". But in other conversations I would prefer "let us", e.g. "let us now turn to (such-and-such example)" in a classroom. To me "let's" is folksy and informal, which is often what you want, and "let us" is more formal.