Category objects

A category can be regarded as a set with certain algebraic operations. Hence we can define a category object in a category just like a group object. Is this notion useful? Is there any research on them?

I wouldn't say a category can be regarded as a set; in fact, it certainly can't unless it's small. But that's not what you're asking. I think the closest thing to what you're asking about is the notion of an internal category, which exists in all categories with pullbacks.

If $\mathcal{C}$ has pullbacks then an internal category in $\mathcal{C}$ is given by:

• two objects $C_0, C_1$, which can be thought of as the 'object of objects' and 'object of arrows, respectively;
• arrows $c, d : C_1 \rightrightarrows C_0$, which can be thought of as the 'codomain arrow' and 'domain arrow' respectively;
• an arrow $i : C_0 \to C_1$, which can be thought of as the 'identity arrow arrow'; and
• $m : C_1 \times_{C_0} C_1 \to C_1$, which can be thought of as the 'composition arrow'.

These arrows in $\mathcal{C}$ then interact according to how you'd expect them to according to the category axioms.

For more, see nCatLab.

You can define category objects in a category with enough pullbacks. This is very useful and is called "internal category theory". The basics are explained in http://ncatlab.org/nlab/show/internal+category. More generally, one can define categories internal to a monoidal category by suitably replacing pullbacks by cotensors. This is explained in http://ncatlab.org/nlab/show/internal+category+in+a+monoidal+category

Even more generally, but not written anywhere I know of, is the ability to define categories internal to operads with contensors. More general still, one can define operads internal to operads with cotensors.

People have already mentioned that this is called an internal category and defined in the obvious way, so let me explain a couple of applications.

Theorem. Let $\mathcal{E}$ and $\mathcal{S}$ be elementary toposes and suppose $p : \mathcal{E} \to \mathcal{S}$ is a bounded geometric morphism. Then there exists an internal category $\mathbb{C}$ in $\mathcal{S}$ and a factorisation of $p$ as $$\mathcal{E} \longrightarrow [\mathbb{C}^\textrm{op}, \mathcal{S}] \longrightarrow \mathcal{S}$$ where $[\mathbb{C}^\textrm{op}, \mathcal{S}]$ denotes the topos of internal presheaves on $\mathbb{C}$ in $\mathcal{S}$, where the geometric morphism $\mathcal{E} \to [\mathbb{C}^\textrm{op}, \mathcal{S}]$ is a geometric inclusion.

Proof. See §B3.3 in Sketches of an elephant.

In other words, Giraud's theorem characterising Grothendieck toposes can be relativised to arbitrary base toposes – and in the strongest possible way, because we do not need to assume that $\mathcal{S}$ is cocomplete or boolean or anything like $\textbf{Set}$ at all.

Another application can be found in Janelidze's categorical Galois theory: under good conditions, the category of "split algebras" in one category will be (contravariantly) equivalent to the category of internal presheaves on an internal groupoid in another category. For example,

Theorem. Let $\sigma : R \to S$ be a Galois descent morphism of rings, and let $\textrm{Gal}(\sigma)$ be the corresponding Galois groupoid in the category of profinite spaces. Then the category of $R$-algebras split by $\sigma$ is contravariantly equivalent to the category of internal presheaves on $\textrm{Gal}(\sigma)$ in the category of profinite spaces.

Proof. See Theorem 4.7.15 in [Categorical Galois theory].

Also, I'll mention that Bénabou worked out how to define the notion of a locally internal category, which is rather more complicated. This can be found in §B2.2 of Sketches of an elephant.