What is the relationship between component family, component type and renderer type?

The renderer is selected by the component family, not by the component type as you seem to expect.

Let's cite the JSF 2.0 specification:

3.1.2 Component Type

While not a property of UIComponent, the component-type is an important piece of data related to each UIComponent subclass that allows the Application instance to create new instances of UIComponent subclasses with that type. Please see Section 7.1.11 “Object Factories” for more on component-type.

3.1.3 Component Family

Each standard user interface component class has a standard value for the component family, which is used to look up renderers associated with this component. Subclasses of a generic UIComponent class will generally inherit this property from its superclass, so that renderers who only expect the superclass will still be able to process specialized subclasses.

Basically, the component type is mandatory for JSF in order to create the component by Application#createComponent() method.

UIComponent component = context.getApplication().createComponent("study.faces.Div");

This has the advantage that the component study.faces.Div is not a compile time dependency and thus offers runtime polymorphism and pluggability possibilities (if you're familiar with JDBC's Class#forName() mechanism and its factories, then you'll understand that part better).

Each component type belongs to a family which can consist of one or more components. The renderer is selected based on the component family and renderer type by RenderKit#getRenderer()

Renderer renderer = context.getRenderKit().getRenderer(component.getFamily(), component.getRendererType());

The renderer is not selected based on the component type and renderer type. This allows for the reuse of the renderer for multiple component types belonging to a component family. Otherwise you'd need to register a single renderer for every single component even though the components could share the same renderer.

The following faces-config.xml entry

<component>
    <component-type>study.faces.Div</component-type>
    <component-class>javax.faces.component.UIPanel</component-class>
</component>

tells JSF that the Application should create an instance of the given component class whenever a component of the given component type is to be created. The component family is not specified in there, because that's already implicitly known by component.getFamily().

And the following faces-config.xml entry

<render-kit>
    <renderer>
        <component-family>javax.faces.Panel</component-family>
        <renderer-type>study.faces.DivRenderer</renderer-type>
        <renderer-class>com.study.ui.DivRenderer</renderer-class>
    </renderer>
</render-kit>

tells JSF that the RenderKit should return an instance of the given renderer class whenever a renderer of the given component family and renderer type is been requested.

The following .taglib.xml entry

<tag>
    <tag-name>div</tag-name>
    <component>
        <component-type>study.faces.Div</component-type>
        <renderer-type>study.faces.DivRenderer</renderer-type>
    </component>
</tag>

tells JSF (well, Facelets) that the given tag should create a component of the given component type in the view root (whose class is been definied in faces-config.xml) and that its renderer type should be set to the given renderer type. Note that the component type is not used to select the renderer, instead it is used to create a component in the view root. Also note that the renderer type entry is optional. Otherwise component's own predefinied renderer type will be used. This allows for reusing existing component types with a different renderer type.

The javadoc says of getFamily():

Return the identifier of the component family to which this component belongs. This identifier, in conjunction with the value of the rendererType property, may be used to select the appropriate Renderer for this component instance.

The maintainers' rationale is covered in the JSF specification:

Component Family

Each standard user interface component class has a standard value for the component family, which is used to look up renderers associated with this component. Subclasses of a generic UIComponent class will generally inherit this property from its superclass, so that renderers who only expect the superclass will still be able to process specialized subclasses.

I interpret this as providing some ability for additional type checking beyond the Java class type (since subclasses will always be subclasses.) I'm not sure how useful this really is.