What part of speech is "only" in "Fame lights a fuse that leads only to extinguishment"?

My impulse is that it's modifying the verb leads, and is thus an adverb; yet it seems that a case could also be made that it's exerting power on the phrase to extinguishment, a noun, which would make it ... what, exactly? an adjective? I'm confused.

Fame lights a fuse that leads only to extinguishment.

In that sentence, only is an adverb, but to extinguishment isn't a noun. It's a prepositional phrase. It's quite common for adverbs to modify prepositions (e.g., completely under the bed). It's also quite common for adverbs to modify entire noun phrases, (e.g., even a Möbius strip, only the lonely). They rarely modify individual nouns inside noun phrases, though they do occasionally.

The sentence could also be written like this:

Fame lights a fuse that does not lead anywhere, except to extinguishment

But it is longer and maybe not as poetic sounding.

In the original wording, "only" emphasizes that there is nowhere else for fame to lead. Omitting "only" changes the meaning and with it, the sentence could be interpreted as

Fame lights a fuse that will likely lead to extinguishment, but there is some chance that it won't.

'Only' is an adverb modifying the verb by answering the question "leads where?" by narrowing the scope of where it may lead. 'To extinguishment' is a prepositional phrase also acting as an adverb modifying the verb by also answering the question "leads where?"

This adverb and adverbial phrase work together in harmony to bring about a more complex modification of the verb answering the same question "leads where?" than either of them could have accomplished alone.