Rules about prepositions and duplicating 'to'
While your sentence
I think we really ought to leave that up to the people we're assigning the label to to judge.
is not grammatically incorrect, it would be considered inelegant by many writers, especially because it involves a duplicate preposition. (It is not uncommon to find "...had had..." in formal contexts and this is not usually frowned upon.) So, you are right your construction is not stylistically preferred. Also, there should not be a comma between the duplicates.
Here are possible revisions in increasing order of elegance:
I think we really ought to leave that up to the people to which we're assigning the label to judge.
I think we really ought to leave that judgment up to the people to which we're assigning the label.
I think we really ought to leave that judgment up to the people whom we're labeling.
I think we really ought to leave that up to the people we're labeling to judge.
I think we really ought to leave that judgment up to the people we're labeling.
I have seen duplicate words in print frequently, but usually with words like "had" rather than "to". For example: "If Joe had had a better instructor, he would have passed the test." Such sentences can be a little hard to read, but in general there is no rule against it; the construct is grammatical. Often a reader will have the same reaction as an automatic spell checker -- "This must be a typo" -- only to re-read and see that it actually does make sense.
There seems to be a case for the notion that "had had" is more acceptable than "to to". In print, it's probably best to avoid "to to", but when spoken, I really don't see a difference regarding the level of awkwardness or clarity. For example:
Person A: I'm really not in the mood to run the mile right now.
Person B: But you have to to make the team!
Spoken with proper tone of voice, I think this usage is perfectly acceptable and understandable.
In general, I would consider rewording the sentence to make it flow better, on a case-by-case basis. Here is an article provided by Grammar Girl discussing usage and correctness in more detail.