Can I "shout someone out", or only "give a shout out to" someone (to recognize them)?

You could try it, but it would be non-standard and likely misconstrued.

The order

verb => object => out

as in

shout Karen out

brings up all kinds of extreme associations, since out used in that way usually means to do something completely.

I kicked Jim out.

We threw them out.

Rachel found Jim out, finally. He'd been cheating on her for weeks.

You've got beer? Bust it out!

You two fight it out. I want nothing to do with this.

I thought I had the right word, but in the end I crossed it out.

This is out used in the completive sense of "at or to an end" (NOAD), so I would suggest to you that "to shout someone out" may not be the best alteration of the more familiar "shout out to someone" trope.

In the OED, "shout-out" is only recognized as a noun:

A mention, acknowledgement, or greeting, esp. one made over the radio or during a live performance; a namecheck.

In the United States, esp. among performers or fans of rap music; in the United Kingdom, particularly associated with dance music and club subculture.

1990 Newsday (Nexis) 8 Feb. ii. 15 There were Mardi Gras anthems and a shout out to Africa, and plenty of spare, angular funk.

1991 Source Dec. 36/2 Big fat shout outs and congrats to the Black Rock Coalition on the release of their compilation album.

1993 B. Cross It's not about Salary 16 Playing James Brown, Sly Stone and Rare Earth, doing shout-outs from the mic, and screaming ‘Rock the house’, Herc began to formulate what later became known as hiphop.

1997 M. Collin & J. Godfrey Altered State iv. 128 House music all day and all night long, seven days a week, interspersed with advertisements for raves and ‘shout-outs’ to callers who rang in to the station's mobile phone.

2000 Philadelphia City Paper 27 Apr.–4 May 54/1 On the media tip, special shout-out to KYW-TV's Walt Hunter.

As a verb, it seems like "to shout out" is really a phrasal verb which isn't quite the same as giving someone a shout out:

to say something suddenly in a loud voice

Usually the verb form of the noun, as I've heard it, is just "give someone a shout-out". I understand what someone means if they say "I'm shouting out to all my friends!" but it seems a little off. (For reference, even Urban Dictionary doesn't mark it as a verb.) Since "to shout out" doesn't mean quite the same thing as "giving a shout out", it seems even more off to insert something between the verb and preposition.

Assuming that "to shout out" were a proper phrasal verb with the intended meaning, saying "shout Target out" seems like saying "run over Target to"--the subject is coming in the middle of the verb itself. So I think the original example you gave is incorrect.

I really think you want to stay with to give a shout out to [name] or to shout out to [name].

The other construction, shout [noun/pronoun] out is very similar to

call someone out

or

chew someone out

Both have completely different meanings than the friendly shout out to.... That is a positive action, whereas calling someone out is like a reprimand, and chewing someone out is less formal but similar, along the lines of a tirade of rebuke and displeasure.

I'm in the Southwestern U.S.A., and never heard shout [noun/ pronoun] out used in any context.

The "shout out" usage is very common in New Zealand (and perhaps Australia). As a native Brit I had never heard it before moving to NZ, and clearly some of the answerers here haven't either.

It means "to say hello to" or "to offer recognition to". It might be used, for example, by a sportsman being interviewed on TV who wants to acknowledge all his friends back home: "Can I just give a shout out to all the fellas I train with".

In a slightly different usage, it is almost the norm in NZ to say "Let me shout you a coffee", which means simply an offer to pay for the coffee. (Presupposing the NZer is indeed offering to pay - a fairly rare occurrence not unconnected to a strong Scottish cultural heritage.) A Brit would more likely say "Let me treat you to a coffee".

The concept of "shouting" meaning a freebie is illustrated by a well-known roofing company running a TV advertising campaign known as a "Roof Shout", see http://www.stoppress.co.nz/blog/2012/10/roof-shout-part-deux-return-old-guy

I am not yet well enough tuned in to the local usage to help with its use as verb vs noun, or the separation of "shout" and "out".