How are high scores achieved?

In the vanilla combat challenges (no DLC), there are a limited amount of bad guys per combat challenge. Each combat challenge is comprised of four rounds. Each consecutive round adds more bad guys and, in general, difficulty. At the end of each round, which occurs when all enemies have been dispatched, it tallies up the points that you've earned so far. Variations bonuses and such apply per round, so you may have gotten hit or lost your combo streak in one round, but you can still get bonuses for those in the other rounds. This combat mechanics guide will help.

The fewer enemies that there are, the harder it is to fit in all variations without incapacitating too many to prevent being able to perform all variations.
You'll want to build your combo streak with the weakest hits as possible. Squeeze every hit out of each baddy. I usually say to "juice". Beatdown finisher tenderizing punches means each punch after a stun, in what is called a beatdown, not including the finishing blow. My tests show that the weakest hits come from beatdown finisher tenderizing punches.
Perform finisher variations with increasingly higher modifier-multiplier moves. For example, let's say that you have two enemies left and the beatdown finisher and ground pound variations left to do. Beatdown finisher has a x50 modifier and ground pound has x75. You should do the beatdown finisher first.

Step by step: I made acronyms to help me memorize the best order in which to perform the variations. My latest iteration (I modify this even as I post it), in its full form, is: CBSAGDLMFIUG. I would pronounce this as cob-sag-delem-fee-ug. Counter, Batarang/Bolas, Projectile, Lieutenant, Titan, Strike, Aerial assault, Gadgets, Disarm & destroy, Lesser-lethal special combos, Multi-ground takedown, beat-down Finisher, Instant takedown, Ultra-stun, Ground-pound. The routine is largely the same for Catwoman, who only ignores D.
C & B are for Counter and Batarang/Bolas. These moves are first because the first two hits of a combo don't deal damage, i.e. when the combo count is below x3. Some moves still deal damage regardless of the lowness of the combo streak, like aerial assault and strike, which is why they're not first, despite how much damage that they do. You'll also only want to counter one attack to only get a +x1. Placing it first gives you more control to manipulate the mob for that, because you don't have to worry about dropping a combo that you don't have yet. You may want to evade/redirect after this.
S is for Strike. This is third because, at this point, it can't be a critical strike. Like I said, as little damage as possible. And again, this is a move that deals damage regardless of combo lowness, so there's no extra benefit to doing it first, not to mention that it won't freeflow at sub-x3.
A is for Aerial assault. This is fourth because it has a x25 modifier. Around this point, it's possible to incapacitate a baddy early, unintentionally, so you may want to beware of who you've been hitting. A directed aerial assault is safer the farther you go, and can be performed by moving in a direction and attacking during an aerial assault.
G is for Gadgets. I tried to avoid using the same letters, but I couldn't. Batman: batarangs are already done, batclaw, explosive gel (which must hit something), remote electrical charge (REC), and freeze blast. Catwoman: whip and caltrops. You may want to throw in evades/redirects to make this safer.
D is for special combo Disarm & destroy. This begins the special combo portion. This is a special combo move that only Batman has. Disarming doesn't deal damage.
L is for Lesser lethal special combos. Lesser lethal special combos comprise of batswarm, for Batman, and whiplash ("whip around") and whip trip, for Catwoman. Unlike disarm & destroy, these lesser lethal special combos do a bit of damage, hence "lesser" lethal. So, disarm should be performed before batswarm.
M is for special combo Multi-ground takedown. Make sure you only take down one. This can immediately follow up a strike or batarang.
F is for beatdown Finisher. Batman's starts counting hits for the combo and actually dealing damage after the first four punches. Catwoman's starts after five. This begins the final phase. You'll be doing beatdowns on much of the remaining enemies. You can juice more hits out of an enemy by going for one "actual" hit at a time, then restarting (as by stunning), instead of completing a beatdown finisher in one go. Picture this: three enemies left and these last variations left to go. Tenderize the remaining three in rotation to juice out every last multiplier. Freeze blast one, stun second, stun third, tenderize third, freeze blast third, stun second, tenderize second, freeze blast second, stun first, etc. until one of the thugs inevitably goes down immediately after the beatdown start-up. Then...
I is for special combo Instant takedown. Finally...
U is for Ultra-stun. Ultra-stun the last guy and...
G is for Ground pound. x75 modifier. You can freeze blast a grounded enemy before ground pounding them, for good measure.
Now, there's one variation that I haven't mentioned: Redirection. Unlike what the guide says, it is not Evade but Redirection. I assume that you'll do this anyway.

Optimally, you're only finishing any enemy with special combo takedown, knife evade takedown or ground pound (because those finishing moves have the highest modifier (x100, x100, and x75, respectively)) and you're only finishing them when they have a sliver of health left. However, there aren't always knives to evade and electrified walls to move enemies into, and more importantly, unless you're super mindful, you can't keep track of how much damage each enemy has received or how much health that each has to maximize the points from each enemy. So, rotating beatdowns is the next best thing.

Putting it all together: Now, that acronym is hella long, and I find myself evading/redirecting until I run through the letters in my head. Here's a break-down by group: AGLF. Attack moves, Gadgets, Special combos, Finishers.
A is for Attack moves. Counter, Batarang, Strike, and Aerial assault.
G is for Gadgets.
L is for Lesser lethal special combos. Disarm & destroy and batswarm, or whiplash and whip-trip.
F is for Finishers. Special combo Multi-ground takedown, beatdown Finisher, special combo Instant takedown, and Ground pound.
This is more of a semantic mnemonic. This helps to perform the moves fluently.

Is it ever better to just juice rather than fit in variations? After gathering data and running numbers, I've concluded little except that it's plausible that pure juicing, instead of variations, can be more profitable if your combo exceeds a certain point. I tried to craft a formula for calculating the difference in points earned from the two methods, but it's complicated. If you want to take a crack at the numbers, I post my jumbled analysis here: http://pastebin.com/5GryuHZz

Environmental projectiles: Of the projectiles that are thrown by thugs, gas canisters are set in certain combat challenge stages. Some only become allowed to be picked up by the enemy after a certain round. These can still be blown up at any time via batarang or bolas, even before their "activation" round. This is useful because blowing one up adds +1 to the combo count. Beware: Blowing it up when you're too close to it causes you to get hit by the explosion thus threatening to be a c-c-c-combo breaker.
I don't know if blowing one up via batarang/bolas fulfills the batarang/bolas gadget variation.

I recently learned that there's an indefinite juice loop only with Catwoman and her bolas. Normally, moves don't count for points unless they damage, but the bolas won't deal damage at sub-x3 combo and award points on the hit that goes from x1 to x2.

Opinion: Once I learned that juicing beatdowns was necessary, I was like, "say it ain't so!" I had the same reaction to discovering the Catwoman indefinite. What I mean is, yes, getting a high score by juicing beatdowns is frustrating and the system seems too arbitrary. The case changes for the DLC combat challenges which are based on time or survival.
In a time-based, you'll probably want to go simply crazy with critical strikes.

Update: From research that I conducted here, I've learned that it's also possible to indefinitely juice Titans.

Just adding some info on the highest scores, if you were to use the "beatdown x5" method combined with ground pounds, your combat score on a perfect run will be

$$score=75\times T \times (8.5 \times T + 0.5) $$

Where $T $ is the number of thugs. Now applying this formula to blind justice you would get combat scores of $5,850 (T=3) $ in round 1, $16,125 (T =4) $ in round 2, $31,500 (T=7) $ in round 3, and $64,125 (T=10) $ in round 4. This is a total of $117,600$ for the combat score. You can also use $4$ gadgets without causing damage (all except batarang I think - $1,000$ per round). And you would also get flawless freeflow ($1,000$ per round) and perfect fight bonus ($500$ per round +$5,000$ in last round). This adds $14,000$ to your score. You would only be doing $4$ variations here (ground pound, ultra stun, evade/redirect, quickfire gadget) giving you $1000$ ($250$ per round). In total this is $132,600$. Repeating similar calculations for survival of the fittest and it's over $400,000$ points.

Now, there is an extra $32,000$ variation bonus points not gained here. But, in order to get these variation points, you need to do moves that either cause more damage (so multiplier is reduced for ground pounds), or takedown with less points. As a basic example, suppose in round 1, instead of three ground pounds, you do beat down finisher and special combo takedown. These are worth $50$ points instead of $75$ so your combat score is now $25\times 50 +26\times 50 + $27\times 75=4,575$. So the variation in moves has cost you $1,275$ in combat score. In general it would cost more, the difference being $25\times (16T+3)$. This is over $2,000$ for $T>4$ so its not worth it to go for takedown variation bonus in most levels.

example Here is a youtube video of someone doing all $12$ variations on $T=4$ thugs on blind justice extreme (it also shows the beatdown x5 method). You can see the score is $4,115$ for combat and $7,000$ variation. By reordering the moves (eg have the beatdown finisher at the end and not the start), might be able to eek out another $500$ combat score. So we could say that around $12,000$ is the highest score you can get with $4$ thugs by pursuing variations. The above beatdown plus ground pound method would get $10,350$ with $T=4$. So variation wins right? Yes...but....as this level has a knife...It would actually be better still to replace ground pound with blade dodge takedown, as this is $100$. Putting $100$ instead of $75$ and the score would be $13,800$ (ie 33% bigger).

risk All the above is based on someone who can execute perfectly. The "beatdown 5x + ground pound" method gives you $0$ points until the very end of the fight. So you could have a combo up to say 80x in blind justice round 4 but if you make a mistake after this you basically lose your points for the round - the ground pounds then give only $4,125$ points, not $64,125$! It is also assuming that someone can keep track of which thugs have only 1 hit left to avoid doing a beat down finisher instead - not realistic when there are more than 5 or 6 thugs. The variation method is more incremental - if you manage $11$ variations you only lose $1,000$ of the $7,000$ variation bonus, not all of it. Here is a youtube video of someone doing the strategy - they get very high scores. They achieve $113,950$ for blind justice - not too far off that theoretically best score of $132,600$ (can see most of the points are different in round 4 due to the difficulty of keeping track of hits and also the use of counters).

what about even higher scores? My view - they are using some sort of hack, based on moves that give either "combo" or "points" without doing damage to the thugs. Then it becomes like the iceberg vip lounge - go for as long as you can. One example might be that if the first move in a combo is a counter, it doesn't do damage (I think) but you get $10$ points - so you could keep losing your combo and then doing a single counter.