Do stolen quest items stay in inventory after being caught by guards?
Quest items you stole will be given back to you/kept. If you go in jail you will get them once you escape. If you open the chest with your other belongings you will also find them there. The same goes for when paying bounties.