Finding the Center of Mass of a disk when a part of it is cut out.

The center of mass can probably be computed by using difficult integrals, but it can also be computed in a very simple way.

Let's call $\vec{OG}$ the center of mass, $m_{big}$ and $m_{small}$ the mass of the big and small disk respectively. By using the center of mass definition, we have,

$$ \vec{OG} = \frac{1}{m_{big} + m_{small}}(m_{big} \vec{OO} + m_{small} \vec{OO'}) $$

Since the small disk is cut out and not added we should not use a positive mass, but a negative mass for $m_{small}$, therefore, since the surface of the hole is 4 times smaller than the total surface,

\begin{align} m_{small} &= -\frac{1}{4} m_{big} \end{align}

By substituting this value into the first equation, we obtain,

$$\vec{OG} = - \frac{1}{3}\vec{OO'}$$

Given the symmetry you noted, it seems not worth it to do the integral in polar coordinates when you know the center of mass will only have an $x$. Let's rotate your picture upside down. Notice that computing the integral will give us

$$\iint\limits_{\text{shaded region}}^{} x\:dA = \iint\limits_{\text{small circle on right side}}^{} x\:dA$$

by symmetry because $x$ is an odd function. Next, you want integrate w.r.t. $x$ first because the square roots will cancel

$$\bar{x} =\frac{4}{3\pi R^2}\int_{-\frac{R}{2}}^\frac{R}{2} \int_{\frac{R}{2}-\sqrt{\frac{R^2}{4}-y^2}}^{\frac{R}{2}+\sqrt{\frac{R^2}{4}-y^2}} x\:dx \:dy = \frac{4}{3\pi R}\int_{-\frac{R}{2}}^\frac{R}{2} \sqrt{\frac{R^2}{4}-y^2}\:dy = \frac{R}{6}$$

I am assuming the mass per unit area is one for simplicity.

Using symmetry there is no need to actually compute an integral other than to determine masses.

In general, if you have two essentially disjoint sets $A,B$ then $\int_{A \cup B} f = \int_A f + \int_B f$.

If we let $\bar{x}_A = {1 \over \int_A dm } \int_A x dm$, then $\bar{x}_{A \cup B} = { \int_A dm \over \int_{A \cup B} dm } \bar{x}_A + { \int_B dm \over \int_{A \cup B} dm } \bar{x}_B$.

Furthermore, if we translate an object by $d$ we have $\bar{x}_{A +\{d\}} = \bar{x}_A + d$.

Let $A$ be the big disc with the little disk cut out and let $B$ be the little disk.

If we take the centre of the big disk at the origin, we have $\bar{x}_{A \cup B} = 0$, $\bar{x}_B = ({R \over 2},0)$, $\int_{A \cup B} dm = \pi R^2$, $\int_{B} dm = {1 \over 4} \pi R^2$ (and $\int_{A} dm = {3 \over 4} \pi R^2$, of course).

Solving the above for $\bar{x}_A$ gives $\bar{x}_A = -({R \over 6},0)$.

There is a simple solution, showing where the mass center of two bodies is located, knowing their position and their masses. (Well, on object has negative mass in our case.)

This answer addresses the question in the OP, it is closer to it (instead of the above physics approach) since it deals with the mentioned integral. We will "compute" it. (But not via polar transformation. But rather in a way fitting to the above intuition. However, it is a mathematical way to deal, since we use the linearity of the integral and proceed blindly.)

I will introduce some notations. Let $D$ be the (bigger) disc / the ball $B(0,R)$ centered in $0\in\Bbb C$ and with radius $R$, $D'$ be the (smaller) disk $B(R/2,R)$ centered in $R/2\in\Bbb C$ and with radius $R/2$, and $D''=D-D'$ the difference set.

(We have positioned the three centers in this way without loss of generality.)

We denote by $m,m',m''$ the total mass of the three sets. For symmetry reasons, the mass center of $D''$ is on the $Ox$ axis. Let $a=0$, $a'=R/2$, $a''=?$ be the three abscissa of the mass centers.

Then, in the spirit of the OP, where integrals are involved: $$ \begin{aligned} a'' &= \frac 1{m''}\int_{D''}x\; dx\; dy\\ &= \frac 1{m''}\int_{D}x\; dx\; dy - \frac 1{m''}\int_{D'}x\; dx\; dy\\ &= \frac 1{m''}\cdot ma - \frac 1{m''}\cdot m'a' \\ &=\frac m{m''}a-\frac {m'}{m''}a' \\ &=\frac {1}{3/4}\cdot 0-\frac {1/4}{3/4}\cdot(R/2) \\ &=\frac 16R\ . \end{aligned} $$

Here is another "physics oriented" way:

Because of symmetry the center of mass lies on the line connecting $O$ and $O'$. The mass at each point $-R \leq r \leq R$ on this line lies symmetric above and below which corresponds to a linear density

$$\rho(r) = 2\sqrt{R^2-r^2} \text{ for } -R\leq r\leq 0$$ and taking out the hole $$\rho(r) = 2\left(\sqrt{R^2-r^2} - \sqrt{\left(\frac R2\right)^2-\left(r-\frac R2\right)^2}\right) \text{ for } 0\leq r\leq R$$

Now, noting that the mass of the disk with hole is $\frac 34 \pi R^2$ we get

$$r_M = \frac 4{3\pi R^2}\int_{-R}^Rr\rho(r) dr = - \frac 4{3\pi R^2}\int_0^Rr\sqrt{\left(\frac R2\right)^2-\left(r-\frac R2\right)^2}dr $$ $$= - \frac 4{3\pi R^2}\cdot \frac{\pi R^3}{8}=-\frac R6$$