Why can I sometimes take half of a shops items while the oher half is considered stealing?

The more people like you, the more willing they are to let you "borrow" their stuff (and sell it back to them, in many cases).

The higher their disposition (there are 4 ranks), the greater the value of items they're okay with allowing you to take.