"Going down the rathole" vs. "Going down the rabbit hole."

Both their origins and their current usage suggest different meaning. I think that they are just sometime misused involuntarily:

Rat-hole:

• also rathole, 1812 in figurative sense of "nasty, messy place;" rat (n.) + hole (n.). As "bottomless hole" (especially one where money goes) from 1961.

(Etymonline)

Rabbit hole:

• a bizarre or difficult state or situation —usually used in the phrase down the rabbit hole.

(origin)

• from the rabbit hole that Alice enters in Lewis Carroll's Alice in Wonderland First Known Use: 1980

(M-W)

Down the rabbit-hole implies going into the unknown, as rabbits tend to disappear quickly.

In a rat-hole describes being in a closed up filthy environment.

See also Spider-Hole. for a scary combination of the two.