"But for all his foolish pranks, he was worshipped in the ranks"

Solution 1:

1. How did I know this intuitively? The context doesn't seem to help.

Native speakers of English can intuit the meaning correctly, because they understand the firmly established definition of the idiomatic expression for all —— :

for all ——

In spite of ——:

2. What can my wife look out for to spot this unusual meaning of "for" in the future?

Your wife can learn from this experience that, in the context of contrast, for all is often interpreted as a unit of meaning, rather than as the combined meaning of for and all.

This idiomatic interpretation is not universal, especially when there is not a contrast being made. This can be seen on page 355 of The works of Thomas Moore, comprehending all his melodies, ballads, etc.:

When Solomon travelled, the eastern writers say, " he had a carpet of green silk on which his throne was placed, being of a prodigious length and breadth, and sufficient for all his forces to stand upon...

Again for all is not idiomatic on page 313-14 of Jim Newton's Justice for All: Earl Warren and the Nation He Made:

On January 15, he hosted a lunch for all his brethren.

The Cambridge Dictionary of American idioms provides a usage note that is helpful:

for all something
despite

used to introduce a fact that is completely different than the information that follows it

Emphasis mine

A simple example from page 56 of Robert Kolb's For All the Saints: Changing Perceptions of Martyrdom and Sainthood in the Luthereran Reformation:

For all his literary skills, Rabus did not organize his entire project before he began.

The adverbial yet tends to signal or reinforce the idiomatic meaning of for all:

adverb

3 In spite of that; nevertheless:

ODO

Notice the example on page 316 of Jim Newton's Justice for All: Earl Warren and the Nation He Made:

And yet for all his reservations, Jackson could not bring himself to uphold school segregation.

The conjunction but can also signal or reinforce the idiomatic meaning of for all:

conjunction

1 Used to introduce a phrase or clause contrasting with what has already been mentioned:

ODO

Another example from page 314 of Jim Newton's Justice for All: Earl Warren and the Nation He Made:

But for all his many strengths, Eisenhower was a dunce on matters of race.

The intuitive contrast between foolish pranks and worship strongly suggests the idiomatic meaning: in spite of, but the whimsical mood of the song leaves wiggle room for the literal due to, implying that the silly people of Katmandu, who worship the one-eyed yellow idol, also worshiped the Mad Carew because of his foolish pranks. It is quite possible the author intended this interpretive ambiguity for the sake of interest.

Interpreting this use of for all requires attentive examination of the larger context of the phrase it introduces, and a deeper investigation of the poem itself, which are beyond the scope of this answer.

Solution 2:

"For all the good it will do...", meaning "despite the fact it won't do any good..." comes to mind.

As to what "rule" may be found or cooked up in order to spot these exceptional uses of "for"... that's a great question! Hmmm...

Aha- in each example of this kind of usage we have so far, we do not have "for", but "for all".