How does the fatman deal damage?

According to the Wiki:

The Fat Man also damages multiple opponents per shot.

In that respect, it acts just like the Fat Man in the regular Fallout games.

It even looks like the person who fired it could hurt themselves as well:

Extreme care must be taken when operating the nuclear launcher as the blast radius can easily extend back to the firer if shot too close to one's self.

Edit: I've found a Reddit that talks about your real question of whether or not the damage is spread out based upon the amount of enemies in the room and it does seem that way:

Explosive weapons do have AOE damage in quests, yes. It appears the damage a hit would deal though is just split between the enemies in the room, so you're not actually doing more damage than something like a Dragon's Maw.

It also appears that someone else has made a similar observation that you did in regards to the damage spread:

Based on my observations playing the new quests I have noticed an issue with AOE weapons. I noticed that when I sent in a dweller with a missile launcher or a fatman the damage dealt per attack decreased with the number of enemies in each room. The only conclusion I can come to regarding why this occurs is that AOE damage is not implemented properly. Instead of calculating damage for each enemy within the blast radius the game instead takes the attack damage for a single shot and divides it evenly across all enemies within the room. Now i have played fallout games in the past and this is simply not how AOE damage is supposed to work especially when your tossing mini-nukes around.

As a bonus, I've found another answer to a different question here on Arqade that says that the Fat Man cannot deal friendly fire:

Nope, the Fatman is safe to fire inside your vault. Just like other area weapons (missile launchers, flamethrowers), it does not damage other dwellers or the rooms. In fact, none of the weapons damage other dwellers or the room itself.