Do unused application pools in IIS affect performance?

The only overhead incurred will be the few bytes taken up in applicationHost.config to define the Application Pools and any non-default settings they may have.

If you have a 1000 unused app pools it might improve the time it takes to apply on-the-fly configuration changes to IIS (since there's less XML for the WAS service to parse), but with 5 App Pools I would argue that there is virtually no difference

From what I can see, an application pool with no web sites, does not create a worker process

This is a true statement. See this for reference. (IIS 6, but it applies to 7 and above)

My thoughts are this... It isn't hurting anything with such a small number of unused app pools to just leave them, and you never know when you might need to change a website's pool to use one of the old ones in the event of one of your upgrades going bad.

In terms of memory, cpu, and other IIS resources there is absolutely no tax on them unless a worker process is created, and if the app pools aren't mapped to web sites then they won't create a worker process and take up any available resources. The other great thing is you could just leave them in the stopped state too so that you truly make sure nothing ever gets mapped to them, and even if it does since they'd be in the stopped state you still wouldn't be using resources. In about 3 months time I usually remove old app pools just to keep things clean.

Hope that helps.