How should one properly punctuate layered argument?

Even at risk of being accused by some members of writing an inexpert answer, may I suggest that punctuation is best kept simple and unambiguous. I should recommend the use of just one pair of brackets (or one pair of dashes) at a time, and feel it is better not to mix both in the same sentence: especially not dashes within brackets!

This sentence is not so easy to decode with a single reading:

I sometimes deal with complex concepts, some of which - if they are to be properly expressed (by which I mean precisely expressed) to a reader new to the concept - are dealt with in a sentence which consists of three layers

Moreover, multiple breaks in the form of brackets and dashes interrupt both the flow of the reading and the expression of your ideas.

Part of the problem is that the proper interpretation of a multi-layered complex sentence may be very clear to us -- especially if we wrote it -- but it can potentially confuse readers, which is not what we want, is it! Some great author said, "if it looks likely your reader would lose his way and need to read the sentence again from the beginning, then it is always better to rewrite your sentence."

My English writing goal this year is to write as clearly as I possibly can, and absolutely minimize ambiguity, for which I frequently try to mentally re-read my sentences from the point of view of the general reader. I should recommend this practice to polish our syntax and punctuation for maximum clarity, whenever the option is available.

Two notable closely-related previous questions that have some good answers for your situation are

(Parentheses (inside parentheses))

Is it acceptable to nest parentheses?

I shall also try to update this answer with style guide references that give authoritative guidelines on how to approach punctuation in such cases.