Let $R$ be a ring. Define a circle composition ◦ in R by $a ◦ b =a+b-ab$, $a, b ∈ R$.

For (b) the neutral element is $0$ since $a◦0 = a + 0-a0 = 0$ and $0◦a = 0 + a-0a = 0$, then element is invertible for all a in S must be $a◦ 1 = a + 1-a1 = 1$ and $1◦a = 1 + a-1a = 1$. I have not been able to prove the closure law of ◦, could someone help me? For Part (C), let $\phi :U(R)\rightarrow S$. $\phi$ is given by $a \mapsto a+x-ax$ ?

Hint $\ $ The calculations become simpler and more intuitive if you employ the following observation. Any set bijection $\,h\,:\,R'\to R\,$ serves to transport the ring structure of $\,(R,+,*,0,1),$ to $\,(R',\oplus,\otimes,0',1') \,$ by defining the operations in $\,R'\,$ to make $\,h\,$ be a ring isomorphism $$\begin{align} &a \oplus b\, =\ h^{-1}(h(a) + h(b)),\quad 0' = h^{-1}(0)\\ &a \otimes b\, =\, h^{-1}(h(a)\, * h(b)),\quad\, 1' = h^{-1}(1)\end{align}\qquad \qquad$$

Yours is special case $\ h(x) = 1\!-\!x\ $ so $\,\ a\otimes b = 1-(1\!-\!a)(1\!-\!b) = a+b-ab$

b). You are wrong on first line. Since $$ a\circ 0= a+0-a 0=a $$ $$ 0\circ a= 0+a-0a=a $$ $0$ is the identity element of $(R, \circ)$.

To prove associativity, note $$ (a\circ b)\circ c=(a+b-ab)+c-(a+b-ab)c=a+b+c-ab-ac-bc+abc $$ $$ a\circ (b\circ c)=a+(b+c-bc)-a(b+c-bc)=a+b+c-ab-ac-bc+abc $$ So associativity holds. Thus $(R, \circ)$ is monoid with $0$ as identity element.

For any $a\in S$, since it is quasi-regular, there is a $x\in S$ that $a\circ x=x\circ a =0$. This proves that any element of $S$ has an inverse in $S$.

For closure of $S$, we need to prove that for $a,b\in S, \:a\circ b\in S$. Let $a,b\in S$. By definition of right quasi-regular, there are $x,y\in S$ that $a\circ x=b\circ y=0$. By associativity of $S$ $$ (a\circ b)\circ (y\circ x)=(a\circ (b\circ y))\circ x=(a\circ 0)\circ x=a\circ x=0 $$ This means $a\circ b$ is right quasi-regular. Likewise $a\circ b$ can be proved left quasi-regular. Hence $\:a\circ b\in S$.

Hence $(S, \circ)$ is group.

Edit: Proof of c) was added after @Bill Dubuque's post.

c). Define map $\varphi:(R,\circ)\to (R,\cdot), \: \varphi(a)=1-a$. For any $a,b\in R$ $$ \varphi(a)\varphi(b)=(1-a)(1-b)=1-(a+b-ab)=\varphi(a\circ b) $$ Clearly it is bijective and so is monoid isomorphism. For any $a\in R$, if $a\circ x=0$, then $\:\varphi(a\circ x)=(1-a)(1-x)=1$. This means that if $a$ is right quasi-regular, then $1-a$ is (left) unit in $(R,\cdot)$. Likewise if $a$ is left quasi-regular, then $1-a$ is (right) unit in $(R,\cdot)$. So unit in $(R,\cdot)$ is isomorphic to quasi-regular element, i.e. $$ U(R,\cdot)\cong S(R,\circ) $$