Define $f(x)$ to be the distance from x to the nearest integer. What are the critical points of f?

Your first graph is the correct one for $f(x)$, and $f$ is well-defined by the definition in your title--although I might rather say "...to a nearest integer" since more than one could be "nearest."

You are correct that all integers are critical points of $f$. Those are the bottom corners in your graph. However, you missed some critical points, the ones halfway between two consecutive integers--i.e. the values $n+\frac 12$ where $n$ is an integer. Those are the top corners in your graph.

All other points have a derivative of $1$ or $-1$, so we have found all the critical points of $f$.