Failing the quest "Of Flickering Hearts"
Simply put, you need to complete Of Flickering Hearts before giving the bandana to Phillipa, because the quest is given by the Elf Eleyas who will leave with Iorveth when he temporarily leaves Vergen (this happens a little later down the mob quest, but it seems that Eleyas disappears immediately).
In short, you're doing nothing wrong, it's only the order that the game has found fault with.