What to call something that is used to impart flavour to food, but is not actually ingested?
Solution 1:
Flavoring
...something that is added to food or drink to give it a particular taste:
-Cambridge dictionary online
This seems to be a generic term. I cannot think of anything I would put in food that is not actually edible in powdered or crumbled form. Even Bay leaves have been called poisonous, and cooks are told to remove them, but the real reason for removing spices such as these before serving is that they do not pass through the digestive tract easily.