Grammatical structure of a complex sentence

Solution 1:

The sentence means:

• The lecture would maybe have been difficult and boring, if the professor hadn't made it 'lighter' with some jokes.

Three things make the sentence difficult. The first two are:

1. This is the adjective trying (not the verb). It means 'annoying and difficult'.

2. This is a conditional sentence, but there is no if. We understand the same meaning as 'if' because in the subordinate clause, the subject the professor, and the auxiliary had have inverted - they have changed places. The usual order would be: the professor had not. Some more examples of this:

   • Had you taken the train, you would have arrived on time.
   • Were you a bit kinder to him, he would be much more helpful.
   • Should you see Bob, could you ask him to phone me?

The word not can be confusing if you aren't used to this kind of inversion. It may not be clear which word(s) not belongs to. In your sentence, it is part of 'the professor had not'. It has just been separated from had because of the inversion.

Back to the rest of your question. Might shows that it is possible, but not definite, that the lecture would have been boring without some jokes. The speaker is using might, not may, because a past tense modal verb is needed in this type of conditional.

This is the type of conditional speakers use when they want to say what the logical outcome of a situation was (whether the situation was real or not). They usually use it, however, when they know or believe that the situation or event did not happen. We use past perfect verb forms in these conditionals, so we see might have in the sentence instead of just might.

The third thing that makes the sentence difficult is the very long noun phrases. We need to find the head noun in the noun phrase. If we understand the head noun, we will understand the important information in the sentence. In your particular example:

• The lecture on number theory and its applications - lecture
• the non-specialists in the audience - non-specialists
• humorous asides - asides [ - extra comments]

Lastly, not a grammar point, but in case it's helpful: the verb leaven usually means to make a situation feel lighter or less serious.

I hope this is helpful!

Solution 2:

First, the present participle trying does not act as a component of a progressive construction here; it acts as an adjective, meaning "annoying" or "tedious".

The lecture ... might have been tedious ...

Consequently, the verb in the consequence clause (then clause, apodosis) of this conditional sentence is simply might have been: the ordinary past irrealis form for MAY BE.

The condition clause (if clause, protasis) is expressed without if by subject/auxiliary inversion.

... if the professor^† had not leavened ... → ... had the professor not leavened ...

The verb in this clause is, again, a past irrealis, expressed with a past perfect.

You may perhaps see the structure more clearly if we restore it to canonical if ... then order:

If the professor had not leavened the lecture ... [then] it might have been trying ...

The irrealis forms tell us that the condition (had he not leavened) did not occur—rather, the professor did leaven his lecture—and that as a result the consequence which actualization of the condition would have triggered (might have been trying) did not occur either.

The sentence thus signifies that the professor leavened his lecture with humorous asides, which prevented it from being trying for nonspecialists in the audience.

I cannot understand your question about using not at the end of the sentence; if I have not resolved it, please let me know what troubles you about the not and I will endeavour to satisfy you.

^† I take it that your bare professor is a typo; I have arbitrarily read it as the professor rather than Professor X.