Is it true that it's OK to omit the pluperfect (past-perfect) in casual speech?

The pluperfect is used to indicate what is relatively earlier than the compared clause. But in conversation I hear people omit it all the time. Examples:

I haven't spoken to the President since the inauguration. We had a few conversations before that.

vs.

I haven't spoken to the President since the inauguration. We'd had a few conversations before that.

Or,

The incident occurred yesterday, but things started deteriorating before that.

vs.

The incident occurred yesterday, but things had started deteriorating before that.

If I were writing something I would take care to include the pluperfect, but in conversation I often don't have time to do the comparison. It seems to be omitted a lot by people. Am I correct in this perception?

It can be omitted because it is superfluous in meaning. Insert “by then” instead of “before that”, and your past perfect will come alive and necessary. It will make clear how one event preceded another and did not overlap.