Why don't the Rattay soldiers like me?
Some have suggested that there isn't much you can do. Caving in to their demands when they want to search you helps, also keep talking to them and exhausting every dialogue options.
Of course breaking the law will drop it so try to avoid that.