What are the rules for spawning cars?

For general traffic spawns, there's really no rules.

What you're seeing isnt completely random though. Two things skew the probability. First, the game has to economize on memory space. So it's more likely to show you a car you're driving or one quite like it, since it rendering the same model is easier. Even similar cars share some assets. Drive around in a sports car, you are more likely to see that car in traffic, or other sports cars. This kinda doubles down a bit; any car that's already nearby has the same effect, whether in traffic or parked.

Second, because the game tries to be somewhat realistic, the area you're in influences the traffic content. Limos aren't likely down by the docks, but trucks are. Rich cars show in rich areas, and so on.

None of this is a hard and fast rule, unfortunately. Only certain parked cars are consistent spawns. If you are looking for a very specific car, you need to find a parked one or trust to luck.