Why were the N64 analog sticks so inferior to those that came after?

Solution 1:

The N64 analog stick is extremely unusual. It does not use potentiometers, as most other platforms (i.e. playstation controller), but instead used a geared-up digital incremental rotary encoders.

^{This is also why you have to have the control stick centered when the system turns on. Since the stick mechanism has no concept of "centered", the N64 assumes that whatever position the control stick is in when the system is turned on is "center".}

As the gearing wore down, the play of the controller progressively increased.

Wikipedia:

The N64 Analog stick does not use analog potentiometers. It uses light emitting diodes and photo detectors controlled by sensor wheels. The sensor wheels are plastic hubs which produce a shutter effect allowing for an accurate direct digital read. The sensor wheels give direct correlation to the stick position compared to potentiometers which can change resistance values over time. However gameplay function of the stick itself is not on or off digital, but analog in a sense, the more the stick is pushed the faster a game character would walk to run.

The analog stick was prone to some long-term reliability issues. If used excessively, the stick became "loose", which means it will not fully return to center position, which may render gameplay more difficult by giving unintended, non-user input to the system. This loosening can be caused by rotating it intensively - a common practice in games like Mario Party where it offers an advantage for some mini-games.

Solution 2:

I'm not sure how the old N64 analog joysticks worked, but the newer ones are often built using two potentiometers at right-angles with one-another, where the joystick is constrained by two guides that are actually connected to the wiper. Perhaps they use a part like in the following image, which even includes a switch when you press down on the stick (common in controllers nowadays).

^{Source: Digi-Key}

The large sphere you see in newer controllers is simply a sort of bezel that covers that all up, and the radius is fairly large to accommodate the whole joystick package. Potentiometers give an analog voltage which is then converted into a number by the controller. If built well so there is minimal backlash, they can be extremely sensitive, so there is a programmed dead-zone where it will just assume it's at the neutral position (e.g. if the numbers are between 505 and 519 when ideally center is 512) and give no output. Controllers will often calibrate themselves on power-up to determine what's "center".

But I digress. As you point out the N64 joystick has a much smaller 'ball', the volume of the mechanics is probably much smaller, and the mechanical advantage (MA) is much higher. With a higher MA, any slop in the system will be amplified, and the effective dead zone (part of the controller's programming) will be amplified. Other answers have mentioned that it doesn't use pots at all, but those principles still hold.