What is Delphine Talking About?

I think this conversation is about the Age of Aggression song.

I don't imagine she would support the Stormcloaks.

She might not support the Stormcloaks, but some of the townsfolk definitely do, which explains why she wouldn't like the song being played there.

For example if after the intro of the game you decide to follow Ralof (a Stormcloak soldier) to Riverwood you get to hear a conversation that he has with his sister Gerdur, brother-in-law Hod and nephew Frodnar, from which becomes clear that this family is not very fond of Imperials.

As small towns with few families usually share similar opinions I think a song that contains the text "down with Ulfric" is generally disliked here.