Distinguishable painted prisms with six colors (repetition allowed)

We can almost factor the problem into the separate problems of determining the number $n_s$ of colourings on the square faces distinguishable under swapping and the number $n_r$ of colourings on the rectangular faces distinguishable under rotations about the prism axis. The only interaction between these two subproblems arises because if the square faces have the same colour, the colourings of the rectangular faces will behave differently depending on whether they are chiral, i.e. distinguishable from their mirror image. The product $n_sn_r$ counts a chiral colouring of the rectangles and its mirror image twice, even though they can be transformed into each other by swapping the square faces if those have the same colour, so we have to count the chiral and achiral colourings of the rectangles separately and compensate for the double-counting.

For $(4)$ identical colours on the rectangles, there is $1$ achiral pattern with $6$ colour choices.

For $(3,1)$ identical colours on the rectangles, there is $1$ achiral pattern with $6\cdot5$ colour choices.

For $(2,2)$ identical colours on the rectangles, there are $2$ achiral patterns with $\binom62$ colour choices each.

For $(2,1,1)$ identical colours on the rectangles, there is $1$ achiral pattern with $6\binom52$ colour choices and $1$ chiral pattern with $6\cdot5\cdot4$ colour choices.

For $(1,1,1,1)$ identical colours on the rectangles, there is $1$ chiral pattern with $6\cdot5\cdot4\cdot3/4$ colour choices.

Each of the achiral colourings of the rectangles can be combined with $\binom62+6$ colourings of the square faces, whereas for the chiral colourings of the rectangles the $6$ colourings of the square faces with identical colours are double-counted, so to compensate we should count only $\binom62+3$ colourings of the square faces per chiral colouring of the rectangles.

Putting it all together, we have $6+6\cdot5+2\binom62+6\binom52=126$ achiral colourings of the rectangles with $\binom62+6=21$ colourings of the square faces and $6\cdot5\cdot4+6\cdot5\cdot4\cdot3/4=210$ chiral colourings of the rectangles with $\binom62+3=18$ colourings of the square faces, for a total of $126\cdot21+210\cdot18=6426$ distinguishable colourings of the prism.

Now comes the weird part. The answer given in the book leads to $6246$, with just two digits swapped. But if you correct the error you spotted, you only get $6381$. The reason is yet another error -- the number for "swap ends, as above, rotate $90°$ or $270°$" that Gerry corrected from $6^4$ to $6^2$ should in fact be $6^3$, since this operation swaps two pairs of adjacent rectangles into each other. That makes the total come out right to $6426$.

P.S.: Oddly enough, your typo made the number in your original post come out right, since the overall result of the three errors was just that the numbers in the last two lines were swapped :-). What's also odd is that the "Instructor's Solutions Manual" for the 7th edition that I found online contains the incorrect solution you quote, whereas the Spanish edition that you can also find online, which seems to be from 1988 and thus older than the 7th English edition, doesn't give a detailed solution, but gives the correct number $6426$ in the solutions section.

Your $6^4$ is correct. However, $6^2$ for the third end-swapping case is wrong: it really is $6^3$, as you originally had it. If the rotation is $\pi/2$ clockwise, the ends swap, the top and right side swap, and the bottom and left side swap, so you get to choose three colors, not two.

Here’s an elementary enumeration along the lines suggested by André Nicolas. Set the prism on end. Suppose first that the top and bottom faces get the same color; we’ll count the number of distinguishable ways to color the four sides.

Using just one color: $6$ ways.

Using two colors, $3$ sides of one color and $1$ of the other: There are $6$ ways to choose the color for the single face and then $5$ ways to choose the other color, for a total of $30$ ways. (Running total: $36$)

Using two colors, $2$ sides of each color: There are ${6 \choose 2} = 15$ ways to choose the two colors. Once the colors are chosen, they can be applied in just $2$ distinguishable ways: either opposite sides are the same color, or they are different colors. The total number of ways is therefore again $30$. (Running total: $66$)

Using three colors: In this case one color must appear twice, the other two once each. There are $6$ ways to choose the color that appears twice, and there are then ${5 \choose 2} = 10$ ways to choose the other two colors. The two faces of the same color may be either adjacent or opposite. In either case it makes no difference how the remaining two colors are applied to the other two sides, since we can swap ends, so we get a total of $120$ ways in this case, $60$ from each subcase. (Running total: $186$)

Using four colors: There are ${6\choose 4} = 15$ ways to choose the colors. Arbitrarily choose one of the colors to paint one side. There are $3$ ways to color the opposite side, and since we can swap ends, it makes no difference which way we apply the remaining two colors to the other two sides. This case yields a total of $15\cdot 3 = 45$ ways, for a grand total of $231$.

For each of these $231$ colorings of the four sides, there are $6$ ways to color the top and bottom faces, for a total of $1386$ colorings in which the top and bottom faces get the same color.

The analysis when the top and bottom faces get different colors is fairly similar. This time, however, it’s easier to account for the coloring of the top and bottom faces as we go along, noting that there are ${6\choose 2} = 15$ pairs of colors for the top and bottom faces.

Using just one color for the four sides: no change, so $6$ ways to color the sides and $6\cdot 15 = 90$ colorings altogether. (Swapping ends doesn’t affect the sides.)

Using two colors: no change, so $60$ ways and $60\cdot 15 = 900$ colorings; swapping ends affects the sides, but they can be restored by a rotation. (Running total: $990$)

Using three colors: As before, there are $6$ ways to choose the color that appears twice and then $10$ ways to choose the other two colors, and the two faces of the same color may be either adjacent or opposite. If they are opposite, it makes no difference how the remaining two colors are applied to the other two sides, so that sub-case yields $60$ ways and $60\cdot 15 = 900$ colorings. If they are adjacent, however, the order in which the other two colors are applied to the other two sides does matter, and we get $120$ ways. Since these $120$ ways of coloring the sides are not preserved by swapping ends, each apparently extends to $30$ different ways of coloring the prism, for a total of $120\cdot 30 = 3600$ colorings. However, this actually counts each coloring twice. Say that the two singleton side colors are $a$ and $b$, and that the top and bottom colors are $c$ and $d$. The coloring in which $b$ immediately follows $a$ in clockwise order and $c$ is on top is the same as the coloring in which $a$ immediately follows $b$ and $d$ is on top. Each of these colorings is therefore counted twice in the $3600$, once with the side colors in the order $ab$ and once with them in the order $ba$. This subcase therefore produces only $1800$ colorings. (Running total: $3690$)

Using four colors: There are still $15$ ways to choose the colors. As before, arbitrarily choose one of the colors to paint one side. There are still $3$ ways to color the opposite side, but we can no longer swap ends, so the order in which we apply the remaining two colors to the other two sides now matters. This case yields a total of $15\cdot 6 = 90$ ways. A little thought shows that it’s like the second three-color subcase, so it gives rise to $90\cdot 15 = 1350$ colorings and a grand total of $5040$ colorings with distinct top and bottom colors.

Thus, there are altogether $1386+5040 = 6426$ distinguishable colorings of the prism. As a check, the Polya Enumeration Theorem yields the same result.