Find all $f: \mathbb{R} \to \mathbb{R}$ such that $(f(x)+y)(f(x-y)+1)=f(f(xf(x+1))-yf(y-1)), \forall x,y \in \mathbb{R}$

Solution 1:

The only solution is $f(x)=x$, which clearly works. We now show it is the only solution.

Let the given property be $P(x,y)$. By $P(x,-f(x))$, there exists some $a\in\mathbb R$ for which $f(a)=0$. Call $K$ the set of such $a$. We first see from $P(0,1)$ that $$\tag{1}(f(0)+1)(f(-1)+1)=f(0).$$ Now, $P(x,0)$ implies $$\tag{2}f(x)(f(x)+1)=f(f(xf(x+1))),$$ and, for $a\in K$, $P(a,1+a)$ implies $$\tag{3}(f(a)+1+a)(f(-1)+1)=f(f(af(1+a))-(1+a)f(a))=f(f(af(1+a)).$$ So, (2) and (3) together imply $$\tag{4}0=f(a)(f(a)+1)=(f(-1)+1)(f(a)+1+a)=(1+a)(f(-1)+1).$$ for $a\in K$. If $f(-1)\neq -1$, then $1+a$ must be $0$ for any $a\in K$, which means $K=\{-1\}$; in particular, since $K$ is nonempty, $f(-1)=0$. However, then (1) implies $f(0)=f(0)+1$, a contradiction. So, $$\tag{5}f(-1)=-1\implies f(0)=0,$$ where the implication comes from (1). Now, $P(x,1)$ gives $$(f(x)+1)(f(x-1)+1)=f(f(xf(x+1)))=f(x)(f(x)+1),$$ so, for any $x$, $$\tag{6}f(x)=-1\text{ or }f(x-1)=f(x)-1.$$ For $a\in K$, $P(a-1,a+1)$ gives $$\tag{7}(f(a-1)+a+1)(f(-2)+1)=0.$$ Since $f(a)=0$, $f(a-1)$ must be $-1$, and so this gives $a(f(-2)+1)=0$. This means that either $K=\{0\}$ or $f(-2)=-1$. If $f(-2)=-1$, then $P(0,-1)$ gives $$-(f(1)+1)=f(f(-2))=f(-1)=-1,$$ so $f(1)=0$. However, this would imply by (6) that $f(0)=-1$, a contradiction. This means that $K=\{0\}$.

Say $b\in\mathbb R$ such that $f(b)=-1$. Then $P(x,x-b)$ gives $$0=f(f(xf(x+1))-(x-b)f(x-b-1)),$$ which implies $$\tag{8}f(xf(x+1))=(x-b)f(x-b-1).$$ Since $b=-1$ satisfies this, (8) applied to $b=-1$ gives $$\tag{9}(x-b)f(x-b-1)=(x+1)f(x).$$ At $x=b+1$, (9) gives $0=(b+2)f(b+1)$. However, $f(b+1)=0$ implies $b+1=0$, so $b\in\{-2,-1\}$. Since we have shown $f(-2)\neq -1$, this means $$\tag{10}f(x)=-1\implies x=-1.$$ This means that, for $x\neq -1$, (6) reduces to $$\tag{11}f(x-1)=f(x)-1.$$ We will now show $f(-2)=-2$, which will give (11) for all $x$. Indeed, $P(x,-f(x))$ gives $$f(f(xf(x+1))+f(x)f(-f(x)-1))=0,$$ which (using that $0$ is the only $a$ for which $f(a)=0$) when combined with (8) at $b=-1$ gives $$f(-f(x)-1)=-x-1$$ whenever $x\neq 0$. At $x=-2$, this means $f(-f(-2)-1)=1$, which means by (11) that $f(-f(-2)-2)=0$. This implies $f(-2)=-2$, as desired. So, (11) holds for all $x$.

Now, by induction on $n$, $f(n)=n$ for all integers $n$. Apply $P(n,-x)$ for $n\in\mathbb N$ to get $$(n-x)(f(n+x)+1)=f(n(n+1)+xf(-x-1)).$$ Using (11) repeatedly to simplify, this gives $$(n-x)(n+f(x)+1)=n(n+1)+f(xf(-x-1))$$ for all integers $n$. So, this must hold as a polynomial identity, which implies (by equating coefficients of $n$) that $f(x)=x$, as desired.