Integrating around a dog bone contour

There is a conceptual error in the OP.

Note that if $w=z(1-z)$, then the condition $\arg(w)\in[0,2\pi)$ restricts the domain of the complex $z$ plane to a half space.

To see this, we write $z=|z|e^{i\arg(z)}$ and $1-z=|1-z|e^{i\arg(1-z)}$ so that $w=|z||1-z|e^{i\left(\arg(z)+\arg(1-z)\right)}$.

Inasmuch as $\arg(z)+\arg(1-z)$ spans a range of $4\pi$ in the complex $z$-plane, then $\arg(w)$ does likewise.

To address the concerns in the OP in detail, we begin with a short primer.

PRIMER:

The complex logarithm $\log(z)$ is defined for $z\ne 0$ as

$$\log(z)=\log(|z|)+i\arg(z) \tag 1$$

It is easy to show that the complex logarithm satisfies

$$\log(z_1z_2)=\log(z_1)+\log(z_2) \tag 2$$

which means that any value of $\log(z_1z_2)$ can be expressed as the sum of some value of $\log(z_1)$ and some value of $\log(z_2)$.

To see that $(2)$ is true, we simply note that $\log(|z_1||z_2|)=\log(|z_1|)+\log(z_2)$ and $\arg(z_1z_2)=\arg(z_1)+\arg(z_2)$.

NOTE: The relationship in $(2)$ is not generally satisfied when the logarithm is restricted on, say, the principal branch for which $\arg(z)=\text{Arg}(z)$, where $-\pi<\text{Arg}\le \pi$.

Using $(2)$, we can write for $z\ne0$, $z\ne 1$

$$\begin{align} f(z)&=\sqrt{z(1-z)}\\\\ &=e^{\frac12 \log(z(1-z))}\\\\ &=e^{\frac12 \left(\log(z)+\log(1-z)\right)}\\\\ &=e^{\frac12\log(z)}e^{\frac12\log(1-z)}\\\\ &=\sqrt{z}\sqrt{1-z} \end{align}$$

SELECTING A BRANCH OF $\displaystyle \sqrt{z(1-z)}$

To obtain a specific branch of $\sqrt{z(1-z)}$, we can use a branch of $\sqrt{z}$ and another branch of $\sqrt{1-z}$.

If we select the branches for $\sqrt{z}$ and $\sqrt{1-z}$ to be such that $-\pi<\arg(z)\le \pi$ and $0<\arg(1-z)\le 2\pi$, then the branch of $\sqrt{z(1-z)}$ is such that

$$\sqrt{z(1-z)}=\sqrt{|z||1-z|}e^{i\frac12 (\arg(z)+\arg(1-z))}$$

with $-\pi<\arg(z)+\arg(1-z)\le 3\pi$.

With this choice, it is straightforward to show that $\sqrt{z(1-z)}$ is analytic on $\mathbb{C}\setminus [0,1]$.

EVALUATING THE INTEGRAL

Then, we can write

$$\begin{align} \oint_C \sqrt{z(1-z)}\,dz&=\int_0^1 \sqrt{x(1-x)}e^{i(0+2\pi)/2}\,dx+\int_1^0\sqrt{x(1-x)}e^{i(0+0)}\,dx\\\\ &=-2\int_0^1 \sqrt{x(1-x)}\,dx \tag 3 \end{align}$$

NOTE: We bypassed consideration of the contributions to the integral from the circular deformations around the branch points since their contributions vansish in the limit as the radii go to zero.

Using Cauchy's Integral Theorem, the value of the integral $\oint_C \sqrt{z(1-z)}\,dz$ is unaltered by deforming $C$ into a circular contour, centered at the origin, of radius, $R>1$. Hence, exploiting the analyticity of $\sqrt{z(1-z)}$ for $R>1$, we have

$$\begin{align} \oint_C \sqrt{z(1-z)}\,dz&=\oint_{R>1}\sqrt{z(1-z)}\,dz\\\\ &=\int_{-\pi}^{\pi}\sqrt{Re^{i\phi}(1-Re^{i\phi})}\,iRe^{i\phi}\,d\phi\\\\ &=-\int_{-\pi}^{\pi} \left(iRe^{i\phi}\right)^2 \left(1-\frac{1}{Re^{i\phi}}\right)^{1/2}\,d\phi\\\\ &=-\int_{-\pi}^{\pi} \left(iRe^{i\phi}\right)^2 \left(1-\frac{1/2}{Re^{i\phi}}-\frac{1/8}{(Re^{i\phi})^2}-\frac{1/16}{(Re^{i\phi})^3}+O\left(\frac1{(Re^{i\phi})^4}\right)\right)\,d\phi\\\\ &\to -\frac{\pi}{4}\,\,\text{as}\,\,R\to \infty \tag 4 \end{align}$$

Finally, putting together $(3)$ and $(4)$ yields

$$\int_0^1 \sqrt{x(1-x)}\,dx=\frac{\pi}{8}$$

as expected.

NOTE: The expansion leading to $(4)$ is correct given the chosen branches of $\sqrt{z}$ and $\sqrt{1-z}$. Then,

$$\begin{align} \sqrt{z(1-z)}&=\sqrt{-z^2\left(1-\frac1z\right)}\\\\ &=e^{\frac12\log(-z^2)+\frac12\log\left(1-\frac1z\right)}\\\\ &=iz \sqrt{1-\frac1z} \end{align}$$

where we used $\log(-1)=i\pi$ and $\log(z^2)=2\log(z)$. Then, upon expanding $\sqrt{1-\frac1z}$ in its Laurent series in the annulus $1<z<\infty$, and setting $z=Re^{i\phi}$, we obtain the expansion used to arrive at $(4)$.

We may use the method from the following MSE link for the residue at infinity. The introduction and the continuity argument can be copied verbatim and will not be repeated here. We use

$$f(z) = \exp(1/2\times \mathrm{LogA}(z)) \exp(1/2\times \mathrm{LogB}(1-z))$$

with the two logarithms defined as in the linked-to post. Note that this is not the choice Mathematica or Maple makes, but consistency is sufficient here and we can re-use a vetted computation.

The difference is that we have $f(z)\sim z$ at infinity so we need to determine the coefficients on $z,$ the constant coefficient and the one on $1/z.$ There was less work at the linked-to computation because the function was $\sim 1/z$ at infinity.

We use $f(z)/z^2$ for the first one and put $z = R\exp(i\theta).$ Now the modulus of $\mathrm{LogA}(z)$ is $\log R.$ We get for the modulus of $\mathrm{LogB}(1-z)$

$$\log\sqrt{(1-R\cos(\theta))^2 + R^2\sin(\theta)^2} \\ = \log\sqrt{1-2R\cos(\theta) + R^2}.$$

We are manipulating logarithms of positive real numbers using the real logarithm and may continue with

$$\log R + \log\sqrt{1-2\cos(\theta)/R+1/R^2}.$$

Using the leading term and the method from the link we immediately obtain $f(z) \sim -iz.$ For the constant coefficient we use $(f(z)+iz)/z.$ We have

$$\exp\left(\frac{1}{2}\log\sqrt{1-2\cos(\theta)/R+1/R^2}\right) = \sqrt[4]{1-2\cos(\theta)/R+1/R^2} \\ = 1 - \frac{1}{4} (2\cos(\theta)/R-1/R^2) - \frac{3}{32} (2\cos(\theta)/R-1/R^2)^2 -\cdots \\ = 1 - \frac{1}{2}\cos(\theta)\frac{1}{R} + \left(1/4 - \frac{3}{8}\cos(\theta)^2\right)\frac{1}{R^2} - \cdots$$

We also have

$$\arctan\left(\frac{-R\sin(\theta)}{1-R\cos(\theta)}\right) = \arctan\left(\frac{-\sin(\theta)}{1/R-\cos(\theta)}\right) \\ = \theta + \sin(\theta)\frac{1}{R} + \sin(\theta)\cos(\theta)\frac{1}{R^2} + \cdots$$

so that (this actually holds everywhere as long as $\theta$ matches the range of $\arg\mathrm{LogB}$)

$$\exp\left(\frac{1}{2}i\arg\mathrm{LogB(1-z)}\right) \\ = \exp\left(\frac{1}{2}i\theta\right) \left(1 + \frac{1}{2}i\sin(\theta)\frac{1}{R} + \frac{1}{2}i\sin(\theta)\cos(\theta)\frac{1}{R^2} - \frac{1}{8} \sin(\theta)^2\frac{1}{R^2} + \cdots\right).$$

Adding in $iz$ cancels the contribution from the first terms of these two expansions. We get from the second term and collecting the contributions from the lower and upper half plane

$$\int_0^{2\pi} \frac{1}{R\exp(i\theta)} \exp(\log R) \exp(i\theta+\pi i/2) \\ \times \left(\frac{1}{2}i\sin(\theta)\frac{1}{R} -\frac{1}{2}\cos(\theta)\frac{1}{R}\right) Ri\exp(i\theta) \; d\theta \\ = \int_0^{2\pi} \exp(\log R) \exp(i\theta+\pi i/2) i \left(-\frac{1}{2}\exp(-i\theta)\right) \frac{1}{R} \; d\theta \\ = \frac{1}{2} \int_0^{2\pi} \exp(i\theta)\exp(-i\theta) \; d\theta = \pi.$$

for a residue of $-(\pi)/(2\pi i) = i/2.$ This establishes $f(z) \sim -iz + i/2.$ We integrate $f(z) + iz - i/2$ to get the coefficient on $1/z$, obtaining

$$\int_0^{2\pi} \exp(\log R) \exp(i\theta+\pi i/2) \\ \times \frac{1}{R^2} \left(\frac{1}{4}-\frac{3}{8}\cos(\theta)^2 + \frac{1}{2} i\sin(\theta)\cos(\theta) - \frac{1}{8} \sin(\theta)^2 -\frac{1}{4}i\sin(\theta)\cos(\theta)\right) \\ \times Ri\exp(i\theta) \; d\theta$$

The inner term is

$$\frac{1}{8} - \frac{1}{4}\cos(\theta)^2 + \frac{1}{8}i\sin(2\theta) = - \frac{1}{8}\cos(2\theta) + \frac{1}{8}i\sin(2\theta) = - \frac{1}{8}\exp(-2i\theta).$$

which leaves for the integral

$$\frac{1}{8} \int_0^{2\pi} \exp(2i\theta)\exp(-2i\theta) \; d\theta = \frac{\pi}{4}.$$

We get the residue $-(\pi/4)/(2\pi i) = i/8.$ We have established that at infinity,

$$\bbox[5px,border:2px solid #00A000]{ f(z) \sim -iz + \frac{1}{2} i + \frac{1}{8} i \frac{1}{z}}$$

and hence $\mathrm{Res}_{z=\infty} f(z) = \frac{1}{8} i.$ Taking into account that the contour used with this branch produces twice the value of the integral we obtain

$$\frac{1}{2} \times -2\pi i \times \frac{1}{8} i$$

which is

$$\bbox[5px,border:2px solid #00A000]{ \frac{\pi}{8}.}$$