Is it wrong to say "The sun's rays are primarily responsible for skin damage."

The sun's rays are primarily responsible for skin damage.

To me this sounds like it means that the primary activity of the sun's rays is damaging skin. However the intention is obviously that the primary cause of skin damage is the sun's rays.

Is this sentence actually wrong, or just ambiguous?

I would say it was ambiguous; I originally read it as you intended, but it could be read the other way as well.

Simply put it like you already have in your question:

The primary cause of skin damage is UV radiation from the Sun.

I would say it is ambiguous and also incorrect from a logical point of view. Here are a couple of ways to rephrase the sentence without changing too many of the words:

The sun's rays are the the primary cause of skin damage.

Or using the passive,

Skin damage is primarily caused by the sun's rays.

Hope that helps you.