What does the Retire button do in Homeworld (remastered)?

Solution 1:

Retiring ships commands them to be deconstructed and credit you with some RUs. It's like recycling them, so the resources can be used to create something else. This is exactly the same behavior as in the original Homeworld.

From the Homeworld one manual:

Bring up the right click menu and select the Retire command. Ships given the Retire order will return to the nearest Mothership or Carrier and be scrapped. A percentage of the construction cost (in RUs) will be recovered. OR: Press the [I] key once you have selected the ship(s) you wish to retire.

From the Homeworld 2 manual:

This is not 100% of the construction cost. I understand it's approximately 80% of the construction cost.

Solution 2:

"Retire" destroys the selected ships. It's usually used if you have obsolete ships that you no longer need, or if there's a ship limit that you're hitting.