Do metaphorical agents follow special rules?
Solution 1:
A construction includes a syntactic frame and an associated meaning.
The example seems weird to you because the construction X have [passive S] has a meaning that X caused S to happen indirectly, and usually intentionally: e.g.,
I'll have you thrown out on the streets!
Since fatigue is not an intentional agent, it's a metaphorical use of the construction, which is why it sounds a bit odd at first, but the metaphorical meaning is clear.