Why do I get "Illegal generic type for instanceof"?

The o is not a generic type -- it's just a plain Object.

That's not the problem. The problem ... and the root cause of both compilation errors ... is that D is a generic class. And it is generic because it is a non-static nested class in a generic class. Its fully qualified name would be some.pkg.C<T>.D.

FYI: The real D does make use of the generic parameter T.

And the fact that it could make use of T is what makes D a generic class.

The reason that you cannot use instanceof D or (D) is generic type erasure. Basically, the runtime cannot distinguish between the types of (say) C<String>.D and C<Integer>.D. And since it cannot do that, it cannot determine if instanceof D should return true or false or if (D) should succeed or throw ClassCastException.

One solution would be to declare D as static. But that will not work with your "real D" because a static class cannot make use of a generic type parameter from the enclosing class(es). Your "FYI" says that it does that.

Another solution is to instantiate the outer class C passing it the actual type of T as a java.lang.Class<T> instance. Then use this Class instance to implement the runtime type checks and casts as required. This is likely to be messy.

The third solution is to carefully analyze the code and determine if it is safe to a @SuppressWarning annotations to suppress the "unsafe cast" etc warnings.

What type erasure? 'o' is of type Object directly.

Actually Object is the declared type of the o variable. The actual object will most likely have some other type, and it is that type that (if it is a D instance for example) will have been subjected to type erasure.

@StephenC is right in that the problem is that D means C<T>.D, which is a parameterized type. The solution is to use the raw type or the wildcard-parameterized type:

if ( !(o instanceof C.D) )

or

if ( !(o instanceof C<?>.D) )

If you make the inner class static, the code compiles fine.

Ex:

private static class D{...}

Read here for the difference.

You could also try o.getClass() != D.class (after guarding against o being null, of course)

D.class.isInstance(o)

seems to work here.