In American English is it okay to say "that" followed directly by "the" or "there"?

As used in the question, "that" serves as a conjunction to introduce a clause (see, for example, M-W discussion here). When serving as a conjunction, it is not unusual for "that" to be followed by "the" or "there", which would be part of the clause being introduced. In this usage, "that" serves a different purpose than "the" or "there", so they aren't redundant.

In my comment on the question, I wrote, "What gave you the impression that there might be an issue?" "...there might be an issue" is the clause introduced by "that".

For English Language Learners landing here, there might be additional confusion based on general guidance about "that" and "there" being redundant in a certain common non-standard use. As Janus Bahs Jacquet points out in a comment, this is taught in schools as being "wrong" in terms of not conforming to "standard English", but is seen in certain dialects.

An example of this usage is a case like, "Look at that there red house." "There" is used as a "directional" to clarify what house you're referring to, which is redundant when it immediately follows "that", which is also serving to identify the house. If you require "there" to specify the house, "that" doesn't really apply; it isn't adequate to do the job of identifying what you are referring to. You would say either, "Look at that red house" or "Look at the red house there".

However, it is not unusual for both "that" and "there" to be combined in speech as long as they are not consecutive. "That" would precede the object and "there" would follow it, like "Look at that red house there".