Word or phrase for "expensive mentally"

In your particular context, you could use any of the words that imply hard work, e.g. an arduous task, or a laborious process, and this would be understood to mean in a mental sense. Tedious is also used to mean this sometimes (which you've already used in your example).

Alternatively, if you want to be explicit about the mental aspect of such a word, you could say it's mentally taxing, mentally consuming, mentally demanding or (as mentioned in the comments elsewhere) mentally draining

The term is cognitive load: total amount of mental effort being used.

A phrase from computer science is computationally expensive. One can think of the mental effort of understanding ancient mathematical theory as being a lot to compute. (And it's not too great a distance to remember when people held the profession of computer.)

I don't know if this is a big improvement over the OP's mentally expensive, as one would may need to explain computationally expensive to those other than computer scientists or mathematicians.

In other contexts, the OP might prefer some of the suggestions made in comments:

• taxing
• tiring
• exhausting

A superscription that has made its way into many theses and dissertations is:

Be warned, my son, of anything in addition to them. Of making many books there is no end, and much study wearies the body. (Ecclesiastes 12:12, NIV)

"It is a task that requires a lot of mental effort..." - It is a mentally strenuous task. - It is a task which bears a heavy burden on the mind. - It is a task that is taxing on the brain. - It is a task that requires great mental duress. - It is a task with a high cognitive load. - It is a mentally demanding task.

There are probably a few more suggestions I could throw out, but these are probably the ones that capture the essence of what you mean to say, though the meanings are not always 100% exact.