Does it matter if people like you?

Yes, whether characters like you can make a difference. Based on posts in this thread, you can be declared an enemy of a faction (including your own!) if you upset the wrong people.

After the battle, I ran into my faction leader on the road who insisted that I was now an enemy of his faction due to my choices, and proceeded to attack me.

However, when I defended myself it said "You have started a fight." and I ended up losing affinity with nasty.

So I kill him, and then return to the faction base where william informs me that he's declared me an enemy of my own faction.

Now all the named NPCs in the town attack me on sight, but worse when I defend myself it says "You have started a fight." Shopkeepers won't sell to me, and all the trainers are hostile (or dead)

According to a few places, like this gamepressure guide, dialogue can even be affected by who is your companion at the time.

During a conversation, the other person pays attention to what you are saying and, depending on your choices, can change their opinion about you. A badly selected dialog line can offend them which means that this choice can result in an unwanted end of a dialog or even provoke a fight.

You should know that your companion also has an impact on a dialog. They can often partake in conversations and, depending on their background, they can block some dialog lines. It works in both ways – your companion hears what you are talking so speaking against their faction will not be tolerated by them.

Furthermore, based on information given in this review, some quests are hidden behind dialogue choices, which I would imagine could also be affected by an NPC's affinity towards you:

There are tons of quests (at least 300 based on the achievements) and many quests are very easy to miss as they could be 'hiding' behind some innocent dialogue comment that you decided to skip. Quite often something you thought was meaningless or flavor will turn out to be meaningful in some ways.