What happens if I genocide my own race?

If you are polymorphed into another race, you will "feel dead inside"; if at any point you revert you will immediately lose.

If you are not polymorphed into another race, you will merely die; this cannot be prevented with life-saving or any other effect.

The outcome of attempting to genocide your own race actually depends on a number of factors, including whether the scroll is blessed, cursed or (as is most often the case) neither.

A blessed scroll of genocide is easy to recognize when used, because the prompt will ask you to specify a class of monsters to genocide.

You are expected to answer this prompt with a single letter; if you type more than one letter, only the first one will count. For example, answering human when prompted to genocide a class of monsters would wipe out all monsters denoted by the letter h (i.e. dwarves, hobbits, mind flayers and other small humanoids). This is fatal if you happen to be playing as a dwarf, but otherwise harmless.

Genociding @ with a blessed scroll, however, is always fatal (unless you're polymorphed; see below), regardless of your race. Apparently, this is because the class @ implicitly includes your role, and genociding your role also automatically kills you.

A cursed scroll of genocide, on the other hand, will give you the exact same prompt as an uncursed one, asking you for a specific monster type to genocide.

However, the effect of a cursed scroll of genocide is quite different from an uncursed one — instead of wiping out all monsters of the specified type as expected, it instead summons a number of them around you. This is generally not fatal in itself, although the summoned (and typically hostile) monsters may well proceed to kill you if you're careless.

Finally, if the scroll is neither blessed nor cursed (i.e. "uncursed"), it will simply ask for a single monster name and wipe out all monsters of that type. If the monster name you enter happens to match your race or your role, you will die instantly.

(In fact, it seems that death by genocide is so sudden that you won't even get the usual You die... message, but will proceed straight to DYWYPI. Also, the level you died on will apparently not be eligible to leave bones.)

Notably, NetHack will let you suicide by genociding yourself even if your race (or role) would not normally be a permitted genocide target. Specifically, even though humans and player role monsters are not genocideable by default, genociding your own race and/or role is always permitted.

Also, beware of reading uncursed scrolls of genocide while confused: the game will not even ask you what you want to do, but will automatically genocide your own role, and to add insult to injury, will mock you for it in the high score list ("killed by genocidal confusion").

The only way to genocide your own race (or role) and survive it is to do it while polymorphed into some other type of monster. Of course, you'll die as soon as the polymorph wears out, so an amulet of unchanging is highly recommended, if you want to sustain this unnatural state for more than a handful of turns. Your reward for pulling off this silly trick (besides the chance to brag about it, if you manage to ascend that way) is an oddball entry in the high score list ("quit while already on Charon's boat") if you #quit while your race/role is genocided.

Here's a sample of what happens if you try to genocide human with each type of scroll (while playing as an unpolymorphed human, unless specified otherwise):

Blessed (human gets parsed as h):

As you read the scroll, it disappears.  You have found a scroll of genocide!--More--
What class of monsters do you wish to genocide? human
Wiped out all hobbits.  Wiped out all dwarves.  Wiped out all bugbears.--More--
Wiped out all dwarf lords.  Wiped out all dwarf kings.--More--
Wiped out all mind flayers.  Wiped out all master mind flayers.

Cursed ("reverse genocide"):

As you read the scroll, it disappears.  You have found a scroll of genocide!--More--
What monster do you want to genocide? [type the name] human
Sent in some humans.
The human misses.  The human misses.  The human hits!

^{(This is one of the few rare ways to encounter a generic human in NetHack.)}

Uncursed (while playing as a human):

As you read the scroll, it disappears.  You have found a scroll of genocide!--More--
What monster do you want to genocide? [type the name] human
Wiped out all humans.--More--
Do you want your possessions identified? [ynq] (n)

Uncursed or cursed (while playing as a non-human):

As you read the scroll, it disappears.  You have found a scroll of genocide!--More--
What monster do you want to genocide? [type the name] human
A thunderous voice booms through the caverns:  "No, mortal!  That will not be done."--More--
What monster do you want to genocide? [type the name]

Uncursed (while playing as a human, polymorphed into a master mind flayer):

As you read the scroll, it disappears.  You have found a scroll of genocide!--More--
What monster do you want to genocide? [type the name] human
Wiped out all humans.  You feel dead inside.
...
As you read the scroll, it disappears.  You have found a scroll of genocide!--More--
What monster do you want to genocide? [type the name] master mind flayer
Wiped out all master mind flayers.  You return to human form!--More--
Do you want your possessions identified? [ynq] (n)

(All testing was carried out in wizard mode. No actual ASCII characters were harmed during testing.)

You die:

If you genocide your base race or role, you will die. Variants of your base race are safe to genocide; for example, as a gnome, you could genocide gnome lords or gnome kings, but choosing "gnome" to genocide is fatal. An amulet of life saving will not save you from death by genocide.

And your name goes on the great list of players who have experienced yet-another-stupid-death.