Is there any way to answer a phone call using a dial-up modem?

While not a direct answer to your question, someone has accomplished something similar to this using a Raspberry Pi.

Their full article can be found here but the summary is that they've used a phone to Ethernet adaptor (like the one @Rich Homolka mentions in his answer) wired in to the Pi and a couple of scripts to make sure that the caller is a person or appears on a whitelist before the actual phone is allowed to ring. I'm assuming you'd be able to set it up as a black list so that the three numbers would be told to go away in no uncertain terms or just forever put on hold.

I'm assuming (and hoping) that once the competition is over that he'll make his work available for anyone to use.

I'd like to say, this does not answer your question directly, but I think you're looking at the problem the wrong way. Do you really want a 400 Watt computer running all day just to screen a couple phone calls? I think Both "what do I do with a Pentium I" and "How do I screen calls" are both excellent questions, but IMHO I think you should split these.

Im not sure how the "run to the Pentium, click a script, blow some noise over the modem" is any more effective than saying "I'll never buy your product" and hanging up. Or even better, say "let me get ____" and leave your phone off the hook; these guys live and die by turnover, and dead air may cost them and frustrate them enough for them to not call you.

As far as being rude to the caller, remember; the person on the phone talking to you is not the boss. These are people, whose dire financial straits make the best job available to them sitting making repetitive phone calls that people hate.

I have an ObiTalk 110, hooked into my (spare) Google Voice account. The device has two lines, and one can be an existing house line, like in your situation. The other can be SIP or Google Voice, so you have room for expansion. It's a bit clunky to configure, but the ObiTalk can be used to screen phone numbers. Since you have a small number of numbers to block, this clunkiness may not be an issue.

If you like the ObiTalk, you can eventually use it to talk to Google Voice directly. You could even port your home number to Google Voice if you needed to keep it (though you may need to port to a Mobile number first). Great call screening! Free USA calls! The downside is, you'd essentially be VoIP on your main line, a pain if you ever need to send a fax.

If the ObiTalk is too clean-box and you want a project, I'd suggest Asterisk on Raspberry Pi or something similar. More fun than an old Pentium, and you save some electricity.

Sure you can do this. Hook the modem up to the phone line. You will need a terminal program to control the modem. Granted, I havent seen one in years and MS doesnt include one with Windows anymore, but Im sure if you google it, you can find one.

You might even be able to pipe the command "ATA" to the COM port in command window.

FYI "ATA" is the Hayes standard command to tell a modem to pick up.