What is the correct way to configure a Tridion UGC spam filter? (.NET)

I think the problem is trying to install the SpamFilter in the Webservice. As is stated in the documentation, the SpamFilter must be configured in the Web Application:

3- Configure the spam filter by opening the UGC Web application's configuration file, web.config, located in the root folder of the Web application

Raul is right....the SpamFilter for UGC needs to be configured on the actual Website and not in the UGC WebService. Note that the SpamFilter is called by the UGC ServerControls before any call to the UGC WS is made. Check the official docs for more details

Hope this helps.