Was "man" a gender-neutral word in common usage at some point?

Man in Old English could be either gendered or non-gendered. We inherited that ambiguity.

In Old English, man referred to both an adult male and a human being of either sex. Here is Stephen A. Barney in Word-Hoard: An Introduction to Old English Vocabulary, entry 8:

Mann serves for both "adult male" and "human being (of either sex)," in English; the other Germanic langs. adopted distinct words for the two senses: ModG Mann and Mensch "human being." The latter form occurs in OE (not in our texts) as mennisc (adj.) "human(s)," which survived to the 12th c. The OE terms which discriminate sexes are wer (Lat. vir) and wif (+ man = woman). ModG, like OE, has man in nom. (unstressed) meaning "one" (cf. French on).

Compounds include an early version of mankind, man-cynn. So what we have is a word that can refer to either adult males or human beings. Furthermore, it was sometimes hard to distinguish these uses. While the context makes telling them apart easily enough in this usage OED:

lOE Anglo-Saxon Chron. (Laud) anno 639 Þæs dohter wæs gehaten Ercongota halifemne, & wundorlic man. (This daughter was called Eorcengota holywoman and wonderful man)

Can you tell which man is meant in the next two uses? Testicles makes the second men obvious. The first manne (referring to men being hanged for thievery) is rather harder, unless you believe that only men could be hanged for thievery.

lOE Anglo-Saxon Chron. (Laud) (Peterborough contin.) anno 1124 Þet wæron on þa litle hwile ealles feower & feowerti manne, & six men spilde of here ægon & of here stanes. (There were in a little while four and forty men*, and six **men spoiled of their eyes and their testicles.)

Or how about the following usage? In the Old English Homilies, did Jesus became both man (an adult male) or man (a human being) for us? Either fits.

a1200 MS Trin. Cambr. in R. Morris Old Eng. Homilies (1873) 2nd Ser. 199 (MED) He bi-com man for us.

Also, to the extent we've been discussing a word in a language that still had grammatical gender, man in Old English is masculine. While grammatical gender should not be confused with biological gender (see Latin poeta, agricola, incola, nauta, all appear feminine in form but function masculine when modified by adjectives and most likely refer to men), grammatical gender doesn't help us distinguish semantically gender-neutral cases here.

So it would oversimplify to calling this usage gender-neutral OR strictly gendered, since the reading as adult male and the reading as human being may lead to ambiguity. Furthermore, such ambiguity doesn't exist for any word that refers to women - wif, bryd, faemne, cwene, and other words for women do not enjoy double-status as words referring to humans in general. Thus, while Old English man could be used to refer to human beings (including individual women), man (and the adult male it sometimes refers to) possesses a dual quality that woman and other female terms lack.

I'm old enough to remember when "man" or the combining for "-man" was just common usage. "All men are created equal" was just taken for granted as meaning "All persons were created equal." Words like spokesman, craftsman, chairman, etc. were easy to use and didn't raise any gender questions.

Of course "man" could also be used in a strictly male sense. "Be a man, my son." "This is a job for a strong man." etc.

So you have a word which COULD be gender-specific or COULD be used otherwise. Languages aren't always consistent.

Yes. From the Oxford English Dictionary (subscription required):

Man was considered until the 20th cent. to include women by implication, though referring primarily to males. It is now frequently understood to exclude women, and is therefore avoided by many people.

and then, before beginning its selection of quotations, remarks

In some of the quotations in this section, it is difficult or impossible to tell whether man is intended to mean ‘person’ or ‘male human being’.

The 1662 A Brief and Easie Explanation of the Shorter Catechism says:

Q.[...] Did God create man both Male, and Female, after his own Image?
...
A. God created Man, Male and Female, after his own Image ...

The 1674 The Body of Divinity, Or, a Confession of Faith says:

God made man in his own image, male and female he created them. He created Man both Male and Female ; for by the word Man, is intended Mankind, i. e. both Man and Woman.

There have been many historical uses of man to mean all people well into the 20th century as well. In 1942, Disney released Bambi which has this line

BAMBI: What happened mother? Why did we all run?

BAMBI'S MOTHER: Man was in the forest

Time Magazine had a long running tradition of naming a Man of the Year (Person of the Year since 1999). While women named were Woman of the Year, when they would name a group of people, that group would be Man of the Year. 1967 featured Baby Boomers

In 1962, the Twilight Zone had one of its most famous episodes, To Serve Man, in which aliens come with vast knowledge, ostensibly to help all mankind. The title, however,

is a bit of a pun. "To Serve Man", a book the aliens have, is not how to help mankind, but how to best cook the people they take back to their planet.