Why do games perform better after restarts?

There could be a whole host of things that causes this kind of behaviour, but the one that I would think is the most likely is memory usage. After a couple hours of playing you've probably been to half a dozen maps and played with a few dozen players; depending on how the Source Engine works with your graphics card, you might end up with a bunch of unused data floating around in your RAM or graphics card (models, textures, etc). It could be sitting there so it doesn't have to be loaded later, or because of memory leaks, but either way it's taking up space. Restarting is a surefire way to clean up this space.

Other than restarting, I don't think you have a lot of options. If the unused stuff is sitting around to be used later, then closing the game (and maybe doing something else for a bit) should unload it. If it's a memory leak (which is unlikely, but Valve's quality control on TF2 has never been award-worthy), then I don't know what else could be done.