Am I losing out on XP if I kill someone in Deus Ex:HR and don't accomplish the objective I set?

If you kill people from range you are potentially missing out on XP, because:

Neutralize an enemy: 10XP
Hand-to-hand takedown: 20XP
Non-lethal: 20XP
Headshot: 10XP

So a non-lethal hand-to-hand takedown is worth 50XP, where just shooting someone is only worth 10XP (or 20XP if it was a headshot) and a lethal hand-to-hand takedown is 30XP.

However, playing non-lethal is more difficult as enemies can be revived and traq darts are slow to act (so non-lethal at range needs careful planning), and only using hand-to-hand takedowns means you're more likely to be spotted or trip an alarm.

Remember that you can get the Ghost (500XP) and Smooth Operator (250XP) bonuses for completing objectives undetected and without alarms (respectively), and I imagine they're usually worth more than the few XP lost by not taking down everyone hand-to-hand, so do whatever you need to do to avoid being detected whenever possible.

You might find What actions give XP in Deus Ex: Human Revolution? helpful.

To answer the titular question and your final statement: No, if you said you wanted non-lethal, it only affects what weapon he gives you, if you start killing people you won't lose any exp (it will change how people talk about you though, like some people praise you if you don't kill anyone etc).

If you want to maximise your exp, the best way to do so is either: Get to the end of the level unseen (without moving on to the next level) so you get the Ghost bonus, and then go back and take everyone down. Or if to get the ghost bonus you need to move onto the next level, take everyone down without being seen first, then move on. Keep in mind that the Ghost bonus is the equivalent of 10 non-lethal takedowns, so if by taking people down you're going to get spotted, the best option (usually) is to just finish the level unseen.