“Had not” vs. “have not”?

As other answers have already suggested, B's sentence is in the past perfect tense (or past tense with perfect aspect), which is formed by taking the past tense form of the verb "have" ("had") followed by the past participle of the main verb. The past perfect is used with two different moods:

Past Perfect + Subjunctive Mood

The subjunctive mood in conjunction with the past perfect has a couple specific uses in English, which I will outline below.

1. Conditional sentences. The past perfect is used in conditional sentences to express a hypothetical, counterfactual circumstance. In these constructions the condition clause (directly after "if") is in a past time frame, and the main clause (the consequence) is in a past, present, or future time frame. For example:

   If I had heard the phone ring, I would have answered it.

   If the economy hadn't declined so quickly, he would already have a job now.

   Besides simply placing the word "if" before the conditional clause, the conditional clause can also be formed by inversion of the subject and main verb of the sentence.

   Had I heard the phone ring, I would have answered it.

   Had the economy not declined so quickly, he would already have a job now.

   Both of these constructions is not the same as B's sentence, so it clearly cannot fall under this usage.

2. Expressions of wish. As in conditional sentences, the past perfect is used here to express a counterfactual past state or past action.

   I wish I had not made such a stupid mistake.

   However, this usage only applies after certain verbs or expressions, such as "wish", "if only", "would rather", etc. In B's sentence, there is no such expression, so B's sentence does not correspond with this usage.

There are a few other cases where the subjunctive mood might be used with the past perfect. One example is given here from Wikipedia:

In that case the dogs would find the scent that you had left.

In all of these cases, the past perfect is used to show a counterfactual, hypothetical situation, and is always used with some sort of phrase ("if", "suppose", "wish [that]", "would rather", "in that case", etc.). As far as I know, the subjunctive mood is never used in such a simple sentence like the one B uses:

Yeah, I had not seen it before.

If B were to add "I wish" or "if" (along with a consequence) at the beginning of the sentence, then it would be correct. For example:

Yeah, I wish I had not seen it before.

Yeah, if I had not seen it before, I would be as excited as you.

Past Perfect + Indicative Mood

The past perfect in conjunction with the indicative mood is the most common use of the past perfect and is used to refer "to an event that took place prior to the time frame being considered. This time frame may be stated explicitly, as a stated time or the time of another past action." In this case, since the two have no explicit time frame, the only possible way B's sentence can work is if there is an implicit time frame in mind. For such an implicit time frame, we need context.

If the two have not yet started or are in the middle of the episode, there doesn't seem to be any available implicit reference time frame in the past, in which case B's sentence cannot correspond to the indicative mood usage. However, if the two have already finished the episode, then there is a time frame in mind. This time frame is the implicit time frame of the now past action of them watching the episode. Consequently, if the two have already finished watching the episode, then B's sentence has no problem grammatically.

Now, I will point out that based on A's comment ("I think this is a new episode!"), it seems doubtful that the two have already finished the episode, but my point is that there is no grammatical error if the two have already finished the episode.

Present Perfect

Finally, the present perfect form ("I have not seen it before.") is correct if the two have not yet finished the episode, because B is talking about an experience that he has not had before. (See here for a simple explanation of the uses of the present perfect.)

Conclusion

In conclusion, the past perfect form ("Yeah, I had not seen it before.") is correct if the two have already finished the episode, and the present perfect form ("Yeah, I have not seen it before.") is correct if the two have not yet finished the episode.

In the particular instance raised by the OP, the suitability of B's wording depends on contextual details not provided in the posted question, and on how narrowly we define the word new. If the episode that A and B are watching is being broadcast for the first time on the occasion when they are watching and talking about it, and if new means "never broadcast previously," then B cannot have seen the episode before, and "I have not seen it before" is the normal (and correct) wording.

On the other hand, if the episode that A and B are watching and discussing has only recently become available for viewing—as a streaming file from the Internet, for example—and if new is understood to include the sense "first made available for viewing in the very recent past," then B might have seen the episode earlier on the same day that A and B are watching it (for example), in which case it would be appropriate for B to say "I had not seen it before," where before points to the earlier viewing—that is, where before is short for "before I saw it earlier today."