$f^2+(1+f')^2\leq 1 \implies f=0$

The equation is equivalent to $$ f^2+2f'+f'^2\le0\tag{1} $$ Since $f^2+2f'\le0$, where $f\ne0$, we have $$ (1/f)'\ge\color{#C00000}{1/2}\tag{2} $$ If $f(x_0)=a\gt0$, then $\dfrac1f(x_0)=\dfrac1a\gt0$ and $(2)$ says that $$ \frac1f\left(x_0-\frac3a\right)\le\frac1f(x_0)-\color{#C00000}{\frac12}\frac3a\lt0\tag{3} $$ as long as $\dfrac1f$ doesn't pass to $-\infty$ in $\left[x_0-\frac3a,x_0\right]$.

In any case, on $\left[x_0-\frac3a,x_0\right]$, $\dfrac1f$ must pass through $0$, which is impossible because $f\in C^1(\mathbb{R})$.

If $f(x_0)=a\lt0$, then $\dfrac1f(x_0)=\dfrac1a\lt0$ and $(2)$ says that $$ \frac1f\left(x_0-\frac3a\right)\ge\frac1f(x_0)-\color{#C00000}{\frac12}\frac3a\gt0\tag{4} $$ as long as $\dfrac1f$ doesn't pass to $\infty$ in $\left[x_0,x_0-\frac3a\right]$.

In any case, on $\left[x_0,x_0-\frac3a\right]$, $\dfrac1f$ must pass through $0$, which is impossible because $f\in C^1(\mathbb{R})$.

Therefore, $f(x)=0$ for all $x\in\mathbb{R}$.

Since $f(x)$ is bounded and decreasing both $\lim_{x \rightarrow \infty} f(x)$ and $\lim_{x \rightarrow -\infty} f(x) $ exist. If $f(x)$ were not identically zero, then at least one of these limits is nonzero. Say it is the first one, and call the limit $L$.

By the mean value theorem, $f(n+1) - f(n) = f'(x_n)$ for some $x_n$ between $n$ and $n + 1$. The left-hand side of this equation converges to $L - L = 0$ as $n$ goes to infinity, so we have $$\lim_{n \rightarrow \infty} f'(x_n) = 0$$ But we also have $$\lim_{n \rightarrow \infty} f(x_n) = L$$ Plugging $x_n$ into $f(x)^2 + (1 + f'(x))^2 \leq 1$ and taking limits as $n$ goes to infinity gives $L^2 \leq 0$, a contradiction.

A similar argument works if $\lim_{x \rightarrow -\infty} f(x) \neq 0$.

Hints: As you mentioned, $f$ is decreasing and bounded. Think about $\lim_{n \to \infty} f(n)$. Must this limit exist? What does this imply for the limit of the derivative $f'$?

Full Solution. The function $f(x)$ is decreasing and bounded, so $\lim_{x \to \infty} f(x)=L$ for some $L \in [-1,1]$. For the sake of contradiction, we suppose $|L|>0$. To set up the contradiction, we relate $|f(x)|$ and $f'(x)$: Let $\epsilon\in (0,1]$, and suppose that we have $0 \geq f'(x) \geq -\epsilon$ for some $x \in \mathbb{R}$. Then \begin{align*} f^2(x) & \leq 1-(1+f'(x))^2\\ &\leq -2f'(x) - (f'(x))^2 \\ &\leq -2f'(x) \\ & \leq 2\epsilon. \end{align*} Thus $|f(x)| \leq \sqrt{2\epsilon}$. Therefore we know that if $|f(x)| > \sqrt{2\epsilon}$, then $f'(x) <-\epsilon$. For sufficiently large $x$, we must have $|f(x)| > |L|/2=\sqrt{2(|L|^2/8)}$, hence $f'(x) <-|L|^2/8$. This contradicts the fact that $f(x)$ is bounded below. An entirely analogous argument shows that $\lim_{x \to -\infty} f(x)=0$. Monotonicity implies $f=0$.QED