Nationality modifier vs. Language modifier
Solution 1:
To your first question, the answer is 4: probably both. You're right, the modifier Chinese is ambiguous, but only in terms of what language the author uses. (Note that ambiguity in headlines is not necessarily something to reject. Eradication of ambiguity often requires the sacrifice of attention, and attention-grabbing is paramount in a headline.) In my opinion, Chinese author is not ambiguous about the nationality of the author. As slight confirmation of this, Mo Yan sounds Chinese, so I'd be comfortable inferring that he is a native. As to the language he writes in, it's very likely to be Chinese as well, but not absolutely necessarily.
To the second question, the obvious way is to say what you mean in as many words: Mo Yan, Chinese native and author of Chinese literature, wins Nobel prize. You can see why that wouldn't "take". You could drop the "native" part, since the name and the fact that he writes in Chinese could be enough to go on: Mo Yan, author of Chinese literature, wins Nobel prize or simply Author of Chinese-language novels wins Nobel Literature Prize, if you're willing to drop the name in favor of conciseness.
But here it becomes obvious that a newspaper headline sometimes needs that trivial ambiguity to stay more interesting. If you need to dispel ambiguity, read the article.
Solution 2:
I'm going to disagree slightly with the other answers: I believe "Chinese writer" refers exclusively to the nationality of the writer (your option 2). The rest we assume from context, i.e. from what we know about the world.
Think about it: do you know whether Mo Yan writes in Cantonese or Mandarin? Or what about if the headline mentioned "Swiss writer Juste Olivier"? Would you have any idea what language he wrote in? In both cases, the answer is "no", because the language wasn't specified.
Solution 3:
I would say "3 1/2 -- probably but not necessarily both" :)
"Chinese writer/author" pretty strongly implies "Chinese national" to me. It seems natural that one would specify the nationality of the winner of an international award. Especially since at least one of the articles stresses the committee's trend of picking Europeans.
It stands to reason that a Chinese-nationality author would write primarily (if not exclusively) in Chinese. But that's not necessarily true.
I don't think the potential ambiguity matters in this context. Newspaper style is often ambiguous in favor of being pithy. Unless he was notable for being a Chinese national who won the prize for a book written in English, the language of his work is almost certainly irrelevant to the story.
If you had to be very specific for some reason, you could do something like: "Chinese national Mo Yan won the Nobel Prize in Literature for his Chinese-language tale (whatever title)." But I wouldn't.