Position of verb for object clause
The Object clause how strong ..., since it starts with a Wh-word, is an Embedded Question Object Complement clause. Embedded Wh-Questions differ from real Wh-Questions in that they don't undergo Subject-Auxiliary Inversion.
So that's what lets out the third example, which has undergone Subject-Auxiliary Inversion.
Both of the first two examples are fine, because they both come from different potential placements of the prepositional phrase on Y. Being a locative adverbial, it can be inserted into any available niche in the sentence, including
- The effect of X is Z on Y.
- The effect of X on Y is Z.
Both of these, if "Z" is a real variable, coreferential with What, produce
- What is the effect of X on Y?
as a non-embedded Wh-Question. However, as an Embedded question, either order is available, since the auxiliary verb is doesn't get inverted with the subject Noun Phrase the effect.
Summary:
- We investigate how strong the effect of X on Y is -- OK.
- We investigate how strong the effect on Y of X is -- OK.
- We investigate how strong is the effect of X .. -- Not OK.
While I personally think the second example is stylistically strongest, there's nothing really wrong with the first. It just makes the reader parse "X on Y" as a prepositional phrase before getting to the copula, and "the effect of X" is easier to digest as a unit than "the effect of X on Y."
Moreover, although the third example may have an odd feel to it, there is still nothing wrong with that word order either. I can imagine the formulation being used to emphasize the word strength. (I'm not sure what XZY is supposed to mean; did you intend to write "X on Y" there as well?)
Imagine an expressive voice reading the line:
We investigate how strong is the effect of X on Y.
And consider a parallel construction:
We wonder how foolish is the man who thinks otherwise.
A little bit poetic, perhaps a bit dramatic — but certainly no glaring solecism to be worried about.
Addendum
On the surface, this obviously conflicts with John Lawler's view about the third example. He is answering as a grammarian and for most uses his view on that example would be considered standard. I only mentioned my point about No. 3 because English is really more flexible than "standard grammar" would allow.
The "poetic" or "dramatic" effect I refer to is called anastrophe. It refers to an inversion of word order used for effect as a rhetorical device. From Ward Farnsworth's Classical English Rhetoric:
Some standard purposes of [anastrophe]:
a. The unexpected placement of words calls attention to them. Pushing a word into an especially early or late position often creates emphasis in itself; then the emphasis is still greater because the ordering mildly violates the reader's expectations.
b. Inversion may put words in an order that creates an attractive rhythm.
c. Inversion may compress a meaning into fewer words.
d. Inversion sometimes causes the full meaning of a sentence to become clear only late in its progress; this bit of suspense makes the finish more climactic when it arrives
That said, in most cases it would be safer to avoid anastrophe if you don't really know how to use it or intend to do so — to use it, as @AndrewLeach suggests, you first need to understand that you are deviating from a standard or "normal" word order.