What are some effective strategies for boarding enemy ships?

Solution 1:

When boarding, I tend to aim for the Oxygen subsystem room - it's typically small, and critical enough that I will quickly attract attention. If they don't respond, then I've pretty much sealed their death quickly.

Meanwhile, I'll target whatever critical system I choose (usually weapons, shields, drones, or engines) with my laser weaponry. Once I've hurt one or more of their crewmembers, I'll target the medbay with missiles or bombs. That way, I can finish off heavily wounded crewmembers while disabling their ability to heal. Fire bombs and subsystem bombs are the best for this sort of attack.

I do believe that firing on the rooms where your boarders are will hurt them as well as the enemy, so I tend to avoid this. Trying to get the timing right for lasers (to reduce shields) + beams (to do damage) is also kind of tricky if you're also monitoring a boarding party.

Also, it helps if you've got a couple of teams of boarders - while you're healing the first squad, send the second squad in to keep the pressure up.

Some ships are particularly bad for trying to board, however. If they've got significant crew, especially a lot of mantises, I may give up trying to win by wiping them out. On the other hand, getting into an engagement and realizing they have no medbay pretty much means I've got a win locked down.

As you've noticed, Rock and Mantis crewmembers are the best to send as boarders. Don't forget that you can pause the game with the space bar, so if you think getting your crew back on board is going to be a close call, pausing and unpausing to slow the game down may help you slice the timing razor-thin.

Possible spoilers if you've never been to sector 8:

It's also been noted in a couple of strategies for fighting the final boss that the weapon systems are in isolated rooms with just a single human guarding each. If you send 2 of almost any race of boarders into these rooms, you can kill the guard and disable the system, thereby making this fight easier. The best weapon to start with is (in my opinion) the missile launcher, which is third from the left.

Solution 2:

I just "finished" (sector 8 but not won) a game where boarding was my only combat strategy.

Take out super shields, if necessary, by firing your fastest weapons.
Take out the medical bay with, ideally, a breach bomb, or a missile if unavailable.
Beam over into the non-functional medical bay. Attack.

Your attackers will take out anyone trying to fix the bay. Enemy medical bays are (almost?) always 2x1, so if you beam in two Rocks or Mantises, or two humans or slugs with combat training, they'll be able to take out almost anything. If things go bad,

Beam out. Keeping your crew alive is the most important thing, especially if they're combat-trained.
Bomb the O₂ unit. This can result in oxygen running out while they're trying to fix the medical bay.
If you've got missiles to spare, re-bomb the medical bay if your own people are not in it.

Obviously this strategy requires getting a transport module, mantises (or rocks), and a breach weapon. If you don't have those, boarding isn't a very good strategy. But that's how roguelikes work - you optimize with what you're given on each run, not a universal best strategy.

Solution 3:

Best boarding strategy is to use a firebomb on the medical bay, and then send a party of rock men into the flaming room. They can't repair the damage and it will quickly be destroyed.

A note here though: Do not kill all the crew on final boss. Spoiler:

If all the crew members are killed, the AI takes over and automatically repairs all systems without crew intervention required. (just like the automated ships seen earlier in the game)

Solution 4:

Couple tactics I like:

If your teleport room is a 2-person, upgrade it all the way. Then send in your 2 crew to a 2x1 room; this way the enemy can only send in 2 defenders max. When your teleport is ready again, teleport another 2 crew into a 2x2 room next to your other 2 guys. Then move the original 2 guys into that 2x2 room. So now they're 4 in one room and can do some damage to defenders since they rarely send in 4 defenders. So far this may sound obvious, but the key to this is that even though your teleport room is a 2-person, you can still teleport all 4 guys back in one shot. The 2 extra guys appear in an adjacent room. (this is totally counter-intuitive to anyone growing up watching Star Trek!)
If you have crystal crewmember, use their lockdown ability to keep defenders out of rooms. Let 1 defender in and block the others = easy kill since you're 2 on 1.
Heal Bombs will make your boarders unstoppable since it bypasses shields and hence almost always heals your boarders (unless the bomb gets dodged which is rare)
I hate it when I've got 4 guys in a room and 2-3 enemy crew just keep rotating in and running back to the medbay when they're low on health. You can kill off those guys by waiting until an enemy is at about 25%, then moving 2 guys to the adjacent room which is on the way to their medbay. This way as they're leaving and are running through that adjacent room to get to the medbay, your 2 outside guys can hit them on the run and kill them.
They prioritize shields and weapons, so if your boarders are attacking another room, and you damage either shields or weapons, the defenders will stop attacking you to go repair. They usually just leave 1 guy against your 2+. They usually will not run out of the shield or weapons rooms unless it's to go to the medbay.

Solution 5:

If the enemy ship has an active medical bay, it's going to be very, very difficult to take over with boarders, as the enemy crew members will run to the medbay when they're injured. At least part of your fighting, then, will be inside their medbay, with a big home advantage. (I haven't run the numbers but I wouldn't be surprised if a human could beat a Mantis under those conditions...)

Priority one, then, is to take out the medical bay. Don't worry about struggling to take their shields down - use a weapon that bypasses shields, i.e. missiles or bombs. These should destroy the medbay in one hit, at which point you can beam your boarding party into it, and fight anyone coming to repair.

Other strategies I've found useful are:

disabling multiple systems, and/or starting fires about the enemy ship. This distracts the enemy crew members and I've often found them taking on my boarding party 1-on-2 (and so losing while causing minimal damage) - even when the ship has 3 or 4 crew members on board.
Starting fires in the areas you're fighting in, if you have a Rock boarding party. As well as the general ability of fire to distract as above, your boarding party is immune to fire while the enemy crew almost certainly isn't. It's the reverse of the medbay "home advantage", with them being constantly damaged over time in addition to the combat.
Targeting crew members specifically with ship weapons in order to "soften them up". This is especially true if there are more crew than your boarding party could destroy at once. Certain bombs can be very good for this, though if the shields are down, laser bombardment to a room containing crew does quite well. (And there's an irony to damaging someone so they run to the medbay on low health, then finishing them off with a heavy laser shot in the medbay itself.)
Upgrading the teleporter lets you beam your party in and out more quickly. This is obviously good for healing them up and returning to the fray, but also means you can take bigger risks with the initial deployment, as it's much more likely that you can beam them back out if they're losing.

Your options will also change depending on whether you consider boarding to be your primary offense, with the rest of the ship's systems supporting that goal - or whether you send boarders in to "mop up" a disabled ship, for the extra resources. This would definitely affect what weapons and systems choices you pick, for example (e.g. heavy breach bomb and defense drones in the former case, vs. attack drones and offensive weaponry in the latter).