Homebrew gives a warning: "You have MacPorts or Fink installed"

I just got a new Mac Pro with OS X 10.9. When I run:

brew doctor

I get the warning:

Warning: You have MacPorts or Fink installed:
/opt/local/bin/port

This can cause trouble. You don't have to uninstall them, but you may want to 
temporarily move them out of the way, e.g.
sudo mv /opt/local ~/macports

I didn't install MacPorts or Fink, why do I get the warning?

Note that some 3rd-party packaging tools may sometimes automatically install MacPorts. I recall RVM does that (or at least used to, I don't know the current state), so while you're not aware of actually installing it manually, it might still be there.

If you move /opt/local to a different location that might break whatever pulled in MacPorts in the first place.