Nonrestrictive Clauses and Coordinating conjunction separating two independent clauses
I know that, if you join two independent clauses with a coordinating conjunction (like "and"), you need a comma before the coordinating conjunction.
For example: I walked to the store, and I bought some cheese.
You do not need a comma when using conjunctions to join clauses. Maybe, you could say, you definitely do for dependent clauses, but you certainly can get by without them - in formal writing - when joining ind. clauses. Even your example would be fine:
I walked to the store and I bought some cheese.
(Native speakers might drop the repeated subject and just have "I walked to the store and bought some cheese")
Commas are introduced as a matter of necessity with lists, though, so you may be thinking about the actions as a serial sequence of events. But that's unnecessary in your example.
Commas are punctuation, not grammar. Punctuation marks how the reader should approach or interpret text so that they have the same understanding when they read it that the author had when he wrote it. It is also used, along with constructions like meter used to help speakers when they are reading texts aloud.
Generally punctuation and grammar align very nicely and everyone's happy. But as you explore language as you have with your examples, I would say the best operating assumption for commas is that 1) they mark a pause (like a breath), and 2) they should not be overused or you break the cadence of the sentence.
Returning to your examples:
My question is if this holds up when you throw in a non-restrictive clause after the coordinating conjunction.
For example: I walked to the store, and, despite having plenty of cheese at home, I bought some cheese.
or
I walked to the store and, despite having plenty of cheese at home, I bought some cheese.
I would go with the second form. It works like the dependent clause mention earlier; you are setting aside the dep. clause "despite having plenty of cheese at home" from the complex clause of the main sentence.
Good question. I suffered the angst of comma placement for years. It never really goes away =)
Edit: Exception cited by request, NYT article from today:
Before she was found, the authorities said Ms. Pais had traveled to Denver and was considered armed and “extremely dangerous,” leading to the decision to keep about half a million students home in two dozen school districts.
Or, Ms. Pais had traveled to the store and Ms. Pais was considered to have bought cheese
This answer is in addition to Carly's answer. I am addressing this from a style point of view. Style is somewhat subjective though parallelism can be one of its elements.
Parallelism:
I walked to the store and I bought some cheese. [parallel]
I walked to the store and bought some cheese. [reduced for style by removing the repetition of the pronoun, but also parallel]
Parallel structure as given about where two sentences are joined by "and" do no required a comma. They are not correlative structures. [see link]
Compare parallel structure or correlative structure
Anything introduced after this such as a modifying phrase, would require being set off by commas, to wit:
- I walked to the store and, after thinking about it, I bought some cheese.
Since a phrase has been introduced before the second SVP (subject, verb, predicate), a comma is required just at it would be here (regardless of whether the sentence is compound/complex or not, as above):
- After thinking about it, I bought some cheese. [comma required for a preposition plus a gerund when pre-positioned] Compare:
- I bought some cheese after thinking about it. [no comma required]
Awkwardness reduction:
ORIGINAL: I walked to the store and, after thinking about it, bought (or I bought) some cheese.
The sentence above is not very elegant. I would call it clunky. It can be improved in terms of flow by doing this:
REWRITE: I walked to the store. And after thinking about it, I bought some cheese.
Suggestion: I suggest reading it out loud. Style is indeed subjective. In poetic terms (sound and rhythm), it can be argued that the prosody of the rewrite sounds better than the example above it. Perhaps the reason for this is that the thinking occurs after reaching the store. It comes prior to the act of buying cheese and breaks the the plodding nature of using just a single sentence.
Finally, I have assumed - perhaps totally inaccurately - that these sentences would appear within a large context.
I walked to the store. And after thinking about it, I bought some cheese. That's when the realization hit me [etc.]
There is plenty of evidence in support of the OP's claim that there exists the 'rule' if you join two independent clauses with a coordinating conjunction (like "and"), you need a comma before the coordinating conjunction. For example:
Use commas to separate independent clauses when they are joined by any of these seven coordinating conjunctions: and, but, for, or, nor, so, yet. (Owl at Purdue)
...the comma separates coordinated main clauses (Garner's Modern American Usage, p676)
When independent clauses are joined by and, but, for, or, nor, so, yet, or any other conjunction, a comma usually precedes the conjunction. (The Chicago Manual of Style 16th edition, section 6.28)
Nevertheless many writers do not feel compelled to follow this style advice, particularly on the British side of the Atlantic. Steven Pinker in The Sense of Style (p283) has a good introductory passage on the issue:
The problem for the writer is that punctuation indicates prosody in some places, syntax in others, and neither of them consistently anywhere. After centuries of chaos, the rules of punctuation began to settle down only a bit more than a century ago, and even today the rules differ on the two sides of the Atlantic and from one publication to another. The rules, moreover, are subject to changes in fashion, including an ongoing trend to reduce all punctuation to the bare minimum. They fill scores of pages in reference manuals, and no one but a professional copy-editor knows them all. Even the sticklers can't agree on how to stickle.
In order to remove multiple commas, which can make text clunky to read and are problematic in the OP's second sentence, it can be useful to do some reordering. For example, a little tweaking can produce a version that requires no commas at all:
Original: I walked to the store, and, despite having plenty of cheese at home, I bought some cheese.
Revision: I walked to the store and bought some cheese despite having plenty at home.
As a British English speaker who tends towards bare minimum punctuation this would be my preferred version in what appears to be an informal context. In a similar sentence structure in more formal writing I would put a comma after cheese.