Integral using group theory

I am taking a first course on representation theory and trying to solve a problem where I'm asked to use group theory to integrate something.

Given the polynomial: $$ P(x,y) = ax^2 +bxy +cy^2 $$ consider the integral $$ \int_\Delta P(x,y)dxdy $$ where $\Delta$ is an equilateral triangle with its centroid at the origin and one of the vertices lying on the $y$-axis. The question asks to "decompose $P(x,y)$ into irreducible representations of the symmetry group of the triangle and identify which vanish under the integral and which do not".

The idea of this question is to integrate the polynomial using symmetry, this is fine, but what is not clear to me is what is meant by "decomposing $P$ into irreducible representations" since I don't see how $P$ is a representation. What I have done is breaking $P$ into parts which are symmetric or anti-symmetric with respect to some symmetry of the domain and using invariance of the integral under change of variable. My question is:

What do they mean by that decomposition of the polynomial?

PS: please don't give solutions to the problem itself!

Solution 1:

Write $V \cong \mathbb{R}^2$ for the plane. $V$ contains an equilateral triangle, and this triangle has some finite symmetry group $G \leq \operatorname{GL}(V)$. For example, one element of $G$ will be clockwise rotation by $120^\circ$. The group $G$ acts on the vector space $V$ by definition.

Now, let $f(x, y)$ be any polynomial in two variables. Since in particular we can treat $f$ as a (nonlinear) function $f: V \to \mathbb{R}$, this actually defines an action of the group $G$ on $f$, via the formula $(g \cdot f)(v) := f(g^{-1} v)$. You can check that this action indeed gives a representation of $G$ on the vector space of all two-variable polynomials. Furthermore, this action preserves the degree of a homogeneous polynomial.

Let $X$ denote the vector space of all degree-2 polynomials. Since $X$ is a finite-dimensional representation of $G$, it must decompose into isotypic components: $X = X_a \oplus X_b \oplus X_c$, where $a, b, c$ are indexing the isomorphism classes of irreducible representations of $G$. (For example, $X_a$ will be the sum of all subrepresentations isomorphic to the trivial representation). Since your $P$ is an element of $X$, it must also break apart as $P = P_a + P_b + P_c$.