What Alliance ships do I need in order to defeat an Imperial Star Destroyer?

No.*

Removing the fighters from the equation, the Mon Calamari Cruiser would defeat the Imperial Star Destroyer (assuming you mean a mk. I, not a mk. II) having both greater firepower (320/240 for turbolasers and ion cannons respectively versus the Star Destroyer's 260/220) and signinificantly better damage control (50 for the cruiser, 30 for the destroyer).

This means that the Mon Calamari Cruiser will resist taking damage to its systems far better than the Star Destroyer. The Star Destroyer's shield regen will fail before the cruiser's, allowing the cruiser to slowly reduce the Star Destroyer's shields and destroy/disable it.

As for the fighters, 3 X-Wing squadrons is not sufficient to destroy the Star Destroyer's compliment of 6 TIE Fighter squadrons. To test this, I ran some quick scenarios that provided the following results:

1 X-Wing Squadron : 1 TIE Fighter Squardon

Victor: X-Wing (8/12 remaining)

3 X-Wing Squadrons : 3 TIE Fighter Squadrons

Victor: X-Wing (23/36 remaining)

3 X-Wing Squadrons : 4 TIE Fighter Squadrons

Victor: X-Wing (13/36 remaining, with interference of Imperial capital ships)

1 X-Wing Squadron : 3 TIE Fighter Squadrons

Victor: TIE Fighter (all squadrons remaining, 2 damaged)

2 X-Wing Squadrons : 4 TIE Fighter Squadrons

Victor: TIE Fighter (2 squadrons remaining, both damaged)

3 X-Wing Squadrons : 6 TIE Fighter Squadrons

Victor: TIE Fighter (4 squadrons remaining, 1-2 damaged)

4 X-Wing Squadrons : 6 TIE Fighter Squadrons

Victor: X-Wing (21/48 remaining)

Because the mechanics of starfighter combat makes all squadrons of one fighter group target only 1 opposing squadron until that squadron is destroyed, X-Wing squadrons are unable to win 1:2 and 1:3 matchups. However, they are able to come out on top in matchups of 1:1 and 2:3.

This means that you will need 4 X-Wing squadrons or greater to defeat 6 TIE Fighter squadrons, or at least additional anti-starfigher capital ship support (1 Corellian Corvette would likely do the trick).

As to the asterisk (*), since the 3 X-Wing squadrons will prove insuffiecent in eliminating all TIE squadrons, leaving behind 3-4, it is possible that with the support of its fighters, the Star Destroyer may be able to deal more damage to the Mon Calamari Cruiser than the cruiser can to it. Since such a scenario would likely be impossible to test in singleplayer without extreme luck, I am unable to say how it would play out exactly, but the presence of 4 TIE squadrons makes me think that the Imperial force would come out on top.

That said, it is possible that the remaining TIE squadrons would not make a difference and that the cruiser's incredibly high damage control rating would prove insurmountable to overcome with only TIE Fighters alone. The Mon Calamari Cruiser may be able to destroy the Star Destroyer and then whittle down the fighters given enough time.

Edit:

I finally managed to create a setup whereby I could play against myself and test this exact situation. I tested the following scenarios:

• Mon Calamari Cruiser (3 X-Wings) vs. Imperial Star Destroyer (6 TIE Fighters), uncontrolled
• Mon Calamari Cruiser (3 X-Wings) vs. Imperial Star Destroyer (6 TIE Fighters), controlled
• Mon Calamari Cruiser (2 X-Wings) vs. Imperial Star Destroyer (6 TIE Fighters)

This latest round of research revealed something I had not previously considered: that the six squadrons of TIE Fighters are not immediately available at the beginning of the fight because they must launch from the hanger of the Star Destroyer. This creates a vulnerability that the Alliance player can exploit. If the X-Wings engage the TIE Fighters without delay, they will encounter only about 2-3 TIE squadrons instead of the complete six. Since X-Wings have no problem winning in 1:1 and 3:2 match ups, with a bit of luck, they can quickly dispatch 1-2 squadrons, essentially preventing the Imperial player from ever amassing enough fighters to achieve space superiority. By doing so, the Alliance player can mop up the rest of the TIE squadrons and switch to attacking the Star Destroyer with his fighters. This results in an Alliance Victory.

However, being able to exploit this vulnerability is only possible if the Imperial player does not take appropriate countermeasures to prevent it (controlled scenario). If the Imperial player is skilled enough, he can, by use of maneuver, prevent or delay engaging the advancing X-Wings and allow his squadrons to launch more completely. By doing so, he will have at least 4 squadrons to combat the Alliance fighters, with more to join. Additionally, by not having the Star Destroyer attack the Mon Calamari Cruiser immediately, the Imperial player gives additional protection to his launching fighters, and the Star Destroyer may even join its firepower to attacking the X-Wings. This allows the Imperial player to maintain his numerical advantage while thwarting any attempt by the Alliance player to exploit the delayed launching. This results in an Imperial Victory.

I also tested whether 2 X-Wing squadrons would be sufficient to eliminate the Imperial fighter screen by exploiting the weakness of the launch delay. The firepower of only two squadrons proved insufficient however to quickly disable the first few TIE squadrons, allowing more to launch and overwhelm the X-Wings. This resulted in an Imperial Victory.

To conclude, the results of testing the actual scenario confirmed my initial hypothesis: a single Mon Calamari Cruiser with three accompanying X-Wing squadrons is not sufficient to defeat an Imperial Star Destroyer and TIE Fighter squadrons. Only with an oblivious Imperial commander could an Alliance player potentially manage to defeat such significant opposition. Therefore, additional fighter or capital ship support is needed to defeat an Imperial Star Destroyer.