The origin of the word "bats" to mean "mad" or "insane"

It comes from "bats in the belfry", a figure of speech drawing an analogy between a person's head and a church belfry, and between bats inhabiting a belfry and mental disturbance.

To follow up on my earlier comment under @Robusto's answer, I've been trying to track down the first print use of the "bats in the belfry" idiom, generally agreed to be the origin of the terms bats and batty to mean not quite sane. Etymonline has 1901 and The Phrase Finder has 1900, but antedatings were not difficult to find via Google.

There's three worth highlighting, all from 1899. The first is from Billy Baxter's Letters by William J. Kountz in which the author humorously describes a scene from an opera:

The band cut loose something fierce. The leader tore out about $9.00 worth of hair, and acted generally as though he had bats in his belfry. I thought sure the place would be pinched.

This reference is significant because it can be specifically dated to early 1899 by the preface of the book and because the author claims the book is full of "up-to-date slang."

The second reference is from Elbert Hubbard's Little Journeys to the Homes of Eminent Painters. In a chapter on the French artist Gustave Doré, Hubbard quotes painter James Whistler dismissing Doré with this comment:

Doré—Gustave Doré—an artist? Why, the name sounds familiar! Oh, yes, an illustrator. Ah, now I understand ; but there is a difference between an artist and an illustrator, you know, my boy. Doré—yes, I knew him—he had bats in his belfry!

This reference is interesting because Hubbard later refers to the phrase as a joke and a bon mot. It's possible that the joke part is in reference to the bat-like wings of devils portrayed in many of Doré's works.

The third reference, from a late 1899 edition of The Child-Study Monthly, actually alludes back to Hubbard's use of the phrase:

A parent or teacher who would do this is to our mind "born a button short" or, to use a term from the expressive vernacular of the streets, but dignified by that noble and intensely human literary Philistine, Fra Elbertus, such a person has "bats in his belfry" which, being interpreted, meaneth "rats in his garret" or "wheels in his head."

This passage not only confirms the phrase's origin in current slang rather than from Hubbard's book, but also links the phrase with another alliterative idiom for insanity, rats in the garret. While I can't find this idiom in any modern phrase dictionary, I found use of it back to the 1841 Charles Dickens-edited Pic-Nic Papers.

And one final curious reference from an 1883(?) treatise on hypnosis, Private Instructions in the Science and Art of Organic Magnetism by Mrs. Chandos Leigh Hunt Wallace. Among her list of humorous suggestions a hypnotist might put to a mesmerized group, there is this:

You may tell them there are bats in the belfry, and they should catch them. Each will excitedly describe those they see, some throwing their hats and handkerchiefs up to strike them.

There may be no connection between this British reference and the development of the idiom in America 15 years later, but the description here of people being made to look crazy while imagining bats in the belfry seemed too coincidental not to mention.

Edit 4/13/11:

Doug at Etymonline has edited his listing for batty to account for these 1899 references of bats in (one's) belfry.

The usual term I hear is not bats but batty, about which Etymonline says this:

batty

1580s, "pertaining to bats," from bat (n.2). Slang sense "nuts, crazy" is attested from 1903, from the expression (to have) bats in (one's) belfry, also meaning "not right in the head" (1901).