What's the name of the little number or symbol that refers you to a footnote?

The word footnote seems to only refer to the actual text that you are brought to if you follow the little number or symbol (such as '*') to the bottom of the page.

Does the actual superscript character have a name?

Well, reference mark is in the dictionary to refer to the symbols:

any of various written or printed symbols, as an asterisk (*), dagger (†), or superscript number, used to indicate the presence of further information in a footnote, bibliography, or other text.
_{Dictionary.com Unabridged
Based on the Random House Dictionary, © Random House, Inc. 2018.}

Another similar phrase that is used is reference symbol.

The corresponding phrase for a number would be a reference number:

A footnote consists of two linked parts: the footnote reference number that appears in text, and the footnote text that appears at the bottom of the column.
_{Create footnotes in InDesign - Adobe Support}

And then there's footnote number and footnote symbol.