Native class limiting
Solution 1:
Yes you can, but it might not be the kind of class limiting you're looking for:
sv_highlander 1
...makes it so you can only have one of each class; it's only useful for Highlander 9v9 matches obviously.
Solution 2:
As already answered by badp there is no way to fine-tune class limits without mods.
If you have no problems to install mods on your server, you can check these two:
- TF2 Class Restrictions
- TF2 Class Limit
More generally speaking always take a look to AlliedModders Forum when you need some plugin for TF2, it is one of the best resource out there.