What determines when base troops are discarded?

This won't happen, at least not the way you expect.

When you're at maximum capacity, any new recruit will land in the Waiting Room. Mother Base will of course try to assign waiting staff to whichever team they're best suited to. If that team is full, Mother Base will replace any less suited staff with the waiting one. If no less suited staff can be found, Mother Base will try to assign the waiting staff to their next best suited team until all teams have been tried. Naturally, if any of your staff dies on a mission or needs medical attention, a replacement will be found in the Waiting Room.

Any staff that has been replaced will not be reassigned to another team, even if they could replace someone else on that team, at least not immediately. It's likely they will be assigned to a new team the next time Mother Base chooses to assign them, which normally happens when new staff arrives, or leave the Brig or Sick Bay, or old staff die. You will need to manually trigger an Auto Assign if you do not wish to wait.

Some of your staff, Mother Base won't ever reassign. Those are staff members with unique skills. If you have a low-rank Russian interpreter, he will remain in your Support team unless you specifically assign him somewhere else. If you happen to have 2 Russian interpreters, the lower-ranked one will be reassigned, while the other will remain in your Support team. This isn't limited to interpreter skills and also extends to R&D-relevant ones, like Bionics Specialist and more. These people won't Auto Assign, even if you ask for it. You'll have to manually assign them to a specific team.

People who are currently away on missions obviously cannot change their team until they've returned, and some of your staff are locked to specific teams until the story demands otherwise, such as...

Huey (R&D) and Quiet (Brig or Sick Bay)

Finally, people who you have manually assigned to a team will never be auto assigned by Mother Base or replaced by more competent staff, unless you manually assign or trigger an Auto Assign. Staff that are under a Direct Contract can only be assigned manually, and will never automatically change their team, even if you ask for an Auto Assign.

The Waiting Room is quite spacious and offers a lot of space for staff. Making use of that room is useful for when you plan on expanding Mother Base and do not wish to part with your A-ranked staff, just because they happen to be lowest ranked.

However, the Waiting Room does not offer an unlimited amount of space. I am currently unsure how many people fit in it and if the size of Mother Base affects that limit. When I hit it, there were 175 people waiting inside. I am also unsure if that limit can be exceeded by assigning staff to it manually.

What happens when the Waiting Room is full, along with the rest of Mother Base? Some of the people waiting inside will automatically be dismissed. Who exactly is chosen, I don't know. It could be the oldest recruit, or it could be the lowest ranked one, or it could also be random. It is therefore best to occasionally clean up your Waiting Room so you don't lose some of your better staff to E-rank ones, especially if you're about to expand your base.