The action on the dihedral group on the hexagon is illustrated below:

Dihedral group on the hexagon

The number of assignments of $2$ colors to the vertices that are preserved by a group element $\alpha$ is $$2^{\text{Number of vertex orbits under } \langle \alpha \rangle}$$ since each vertex orbit can be assigned any color, and every vertex in any orbit must be colored the same.

The vertex orbits are highlighted below corresponding to the group elements above (vertices in the same orbit are assigned the same color):

Vertex orbits

Inputting this into Burnside's Lemma gives the number of assignments of $2$ colors (inequivalent under rotations and reflections) as $$\tfrac{1}{12}(2^6 + 2^1 + 2^2 + 2^3 + 2^2 + 2^1 + 2^3 + 2^4 + 2^3 + 2^4 + 2^3 + 2^4)=13.$$

Precisely two of these inequivalent assignments of $2$ colors have all colours the same: when they're all white, and when they're all black. That leaves $11$ inequivalent assignments of $2$ colors to the vertices where both colors are used.


I upvoted the first answer but I would like to show how to compute the cycle index $Z(D_6)$ of the dihedral group $D_6$ and apply the Polya Enumeration Theorem to this problem.

We need to enumerate and factor the twelve permutations that contribute to $Z(D_6).$

There is the idenity, which contributes $$a_1^6.$$ The two rotations by a distance of one and five contribute $$2 a_6.$$ The two rotations by a distance of two or four contribute $$2 a_3^2.$$ Finally the rotation by a distance of three contributes $$a_2^3.$$

There are three reflections about an axis passing through opposite vertices, giving $$3 a_1^2 a_2^2$$

and three reflections about an axis passing through the midpoints of opposite edges, giving $$3 a_2^3.$$

This finally yields the cycle index $$Z(D_6) = \frac{1}{12} \left(a_1^6 + 2a_6 + 2a_3^2 + 3a_1^2 a_2^2 + 4a_2^3\right).$$

Coloring the hexagon with at most two colors we get $$Z(D_6)(A+B)_{A=1, B=1}.$$

This yields $$\frac{1}{12} \left(2^6 + 2\times 2 + 2\times 2^2 + 3\times 2^2 \times 2^2 + 4 \times 2^3\right)$$ which evaluates to $$13.$$

There is a list of similar computations at MSE Meta.