Should I use "or" or "nor" in the following case?
I wasn't forcing myself to say anything, or/nor trying to be funny.
"Or" sounds more grammatically correct to me. But "nor" has more Google results. So I'm confused.
Solution 1:
Garner in his Modern American Usage (p571) would recommend or in Op's sentence. As Garner states:
When the negative of a clause or phrase has appeared at the outset of an enumeration, and a disjunctive conjunction is needed, or is generally better than nor. The initial negative carries through to all the enumerated elements-e.g:
"There have been no bombings nor [read or] armed attacks by one side against the other." William D. Montalbano, "Links to IRA Seen in Rash of Violence in Northern Ireland," L.A. Times 12 Jan 1996.
Peters, in the Cambridge Guide to English Usage (p378) has a more nuanced discussion:
The use of nor is probably declining, even in its core domain of coordinating two negative phrases. Compare:
The gallery will not be open on Sundays or public holidays.
The gallery will not be open on Sundays nor public holidays.
Both sentences are perfectly acceptable English, but the first shows that nor is not really needed to extend the negation over to "public holidays". Rather it may seem to overdo the expression of the negative for the purposes of a single announcement. This use of nor for the second coordinate underscores the parallelism of the two phrases, and in the context of fine writing, with more extended coordinates, it would seem to have its place. For example:
"The word universal is never the name of anything in nature, nor of any idea or phantasm found in the mind ... "
Substitute or for nor in that sentence, and the structure and meaning are still perfectly viable. The negative scope of never carries over to the second coordinate. But the use of nor helps to reaffirm the negative after a complex phrase, and to lift the latter part of the sentence.
The last point that Peters makes above endorses what FumbleFingers says in his or her answer.
Note of course that nor is mandatory if the sentence continues with a new clause:
I wasn't forcing myself to say anything, nor was I trying to be funny.
Solution 2:
It's entirely a stylistic choice whether to use nor or or here, but my guess is the vast majority would opt for or. If we consider a simplified version (still using "continuous" verb participles, but without the syntactically-irrelevant adjunct clauses)...
I wasn't shouting or arguing
...I think almost no native speakers would use nor. To my mind, the only real justification for using nor at all in such contexts is when the first activity (shouting in my example, forcing myself to say anything in OP's) is such a long phrase the reader might need "reminding" of the continuing negating effect of the initial wasn't when he finally reaches the point in the text where the other thing that I wasn't doing is reached. And I don't think that need particularly applies here.
Solution 3:
Either is potentially possible: it essentially depends on where you want to place the "not" in the imaginary structure of your sentence. You can have a single negative ("not"/"n't") that negates both constituents:
wasn't ( .... or ....)
or a negative for each constituent:
was (not ... nor ....)
I would concur with FumbleFingers that, of the two choices, the first is probably more idiomatic in everyday native speaker usage.