How do I make "potions with unknown effects"?
Solution 1:
"potion with unknown effects" just means you don't know whether the combination will result in a valid potion or not, you can discover new potions by trial-and-error.
Each ingredient has four effects, you need to combine ingredients that contain a certain effect at least twice all together, that effect will be added to the effects of the resulting potion. So if you put two ingredients in a potion, only the effects that are present in both ingredients will be effects of the potion. If the ingredients have no effects in common, the result will be the failure to create any potion as you observed.
You can see the effects of an ingredient once you discovered it by creating a potion that used that effect.
Solution 2:
Adding to the previous answerer, eating an ingredient will discover the first effect, as well as inflicting whatever the effect is (i.e. "Restore Health" will Restore some health).