What is mixed damage in league of legends?

You can't tell the percentages, all Mixed damage means is that damage came in multiple types from one skill. Sometimes it's a combination of Physical and True (Irelia), Magical and True (Ahri), or Physical and Magical (Fizz).

Liandry's Torment procs off of all activated abilities (Not on-hit effects), whether physical, magical, or True. It will work for abilities like Cho'Gath's Vorpal Spikes, as they trigger a spell activation on attack, rather than an on-hit effect. The same will hold true for Rylai's Crystal Scepter, so this item combination will work very well together.

If you hover your mouse over the icon of the attack on your death recap, it should tell what is magic damage and what the physical damage is.