how to configure HAProxy as a reverse proxy?

Yes, the way you have things configured will essentially load balance all incoming connections between the backend servers.

What you want is basically one backend definition for each real web server. I think the gist of what you want is multiple backend definitions, one for each real web server. Then in your frontend section, you'd want to use various conditionals to use the associated backend for whichever frontend URL was requested. A pseudo-config might look something like this:

frontend localhost
    mode http
    use_backend web1 if { hdr(host) -i abc.com }
    use_backend web1 if { hdr(host) -i def.com }
    use_backend web2 if { hdr(host) -i cba.com }
    use_backend web2 if { hdr(host) -i fed.com }

backend web1
    server web1 web1.foo.com

backend web2
    server web2 web2.foo.com

Note the mode http. I believe that's required to actually do things like make decisions based on HTTP headers.

And when it's time to move what URLs go to what backend, it's a simple matter of updating the config file and doing a soft reload to haproxy.