How to setup proxy server on XP to test software?

Fiddler is technically a web debugging tool, but it does so by acting as a proxy. It comes with a built-in scripting environment. It would be trivial to implement some arbitrary "proxy rules" in a small script.

Squid is a widely used open-source proxy server with a port for Windows.

Just download it and follow the installation instructions on the website. You shouldn't have to do anything special to configure it for your purpose, just start it and point your software to the server's IP address at port 3128.

I can recommend Proxy+. It is free for 3 users, contains SOCKS-proxy, runs as a service and managed via Web-browser.

If you want to send all traffic through proxy set proxy server in Internet Explorer settings and type proxycfg -p 127.0.0.1:3128 in command prompt. This should work for most of software. For those apps that does not respect system settings you can use something like Free Cap.

To block network traffic for applications individually you can use 3rd-party firewall. I prefer Outpost Firewall Free - it is free and easy to configure.