Why isn't "innard" a word?
Innards is defined as "the internal organs of an animal".
Then shouldn't "innard" denote a single internal organ? Or is this a case where a singular noun looks like a plural?
If we see the word as a plural, it is what is called a plurale tantum or lexical plural.
A plurale tantum (Latin for "plural only"; plural form: pluralia tantum) is a noun that appears only in the plural form and does not have a singular variant for referring to a single object. These are used in English for objects that function as pairs or sets (glasses, pants, scissors, clothes, electronics, bagpipes, genitals).
That does not mean there never is a singular form:
In English, some plurale tantum nouns in fact have a singular form, but one that is used only attributively. That is, phrases such as "trouser presses" and "scissor kick" contain the singular form, even though it is considered non-standard to say "a trouser" on its own. This accords with a general preference for singular nouns in attributive positions in English; however, some words are used in the plural form even as attributive nouns (e.g. "clothes peg", "glasses case").
Another source also gives some nice examples and groups of words that behave like this:
Doesn't it seem just a little loopy that we can make amends but never just one amend; that no matter how carefully we comb through the annals of history, we can never discover just one annal; that we can never pull a shenanigan, be in a doldrum, or get a jitter, a willy, a delerium tremen, a jimjam, or a heebie-jeebie?
However, according to etymoline, it is not necessarily a plural. As a dialectical form of inwards, we can read it as a plural (the singular inward being something on the inside, analogous to the bowels that etymonline mentions), or as a singular substantivised (nouned) adverb.
Inwards as an adverb is a variation on inward though, and because of the analogy with bowels, I would see it as a plural.
As WS2 mentions in his comment, the transformation of inwards to innards is quiet a common one. It seems the w following a consonant can be elided in certain words (I have heard awfull without a pronounced w) and certainly in certain dialects: the Norfolk dialect treats backwards as backards, and forward as forrard:
*Arter I et tha' rubub, I kept a-go'rn backards and forrards to the little house at the top of the garden.
A w followed by a consonant is very often hardly pronounced, if at all, as in playwright, well-wrought or awry. It seems the w sound just doesn't sit well next to consonants.