A classical problem about limit of continuous function at infinity and its connection with Baire Category Theorem
Solution 1:
Since I see you asked this a while ago, I'll answer one of your questions now; I'll give you a solution to your first problem that relates it to the Baire Category Theorem. I have some ideas about your second question, and if no one else pops in to answer it in the meantime, I'll try to come up with something later. (Added: See below.)
If $f:\mathbb R^+\to\mathbb R^+$ is continuous and $\lim_{n\to\infty}{f(nx)}=0$ for every $x\in\mathbb R^+$, we want to prove that $\lim_{x\to\infty}{f(x)}=0$.
Fix some $\epsilon>0$. The sets $E_N=\{x: n \geq N \implies f(nx)\leq \epsilon\}$ are closed (write $E_N$ as the intersection over $n\geq N$ of the sets $\{x:f(nx)\leq\epsilon\}$, which are closed by the continuity of $x\mapsto f(nx)$). On the other hand, the assumption $f(nx)\to0$ that is made for every $x>0$ ensures that you can write $\mathbb R^{+}$ as the union of the $E_N$. The Baire Category Theorem says that at least one of them, say $E_{N}$, contains an open segment $(a,b)$. Thus if $n\geq N$ and $t\in(na,nb)$, then $f(t)\leq \epsilon$. But for some positive integer $M\geq N$ (any $M$ bigger than $a/(b-a)$ will suffice), \begin{equation*} (Ma,\infty)=\bigcup_{n\geq M}(na,nb). \end{equation*} So if $t>Ma$, then $f(t)\leq\epsilon$, which proves that $f(t)\to0$ as $t\to\infty$.
Here are some ideas for your second question, for now—like I said I'll update later if no one else finishes it off. Let
$$
P=\left\{x>0:\lim_{n\to\infty}{f(nx)}\text{ exists}\right\}.
$$
If $P=\mathbb R^+$, the above argument can be tinkered with to ensure that $\lim_{x\to\infty}{f(x)}$ exists, or you can use the argument in the book you linked to. You want to assume only that $P$ is closed with no isolated point, i.e., that $P$ is perfect. In that case, I believe I can modify the BCT argument to show that $\lim_{x\to\infty\atop x\in P}{f(x)}$ exists. (Edit: After thinking about this a bit I don't believe it's as simple as it initially looked to show that $\lim_{x\to\infty\atop x\in P}{f(x)}$ exists; in fact, I'm not even completely certain it's necessarily true.)
Here are a couple of easy observations right off the bat (though I'm not sure they lead in the right direction...):
If $x\in P$, then $nx\in P$ for $n=1,2,3,\dots$
If $P$ contains an interval, then $P=\mathbb R^+$. Hence if $P$ is anywhere dense, then $P=\mathbb R^+$. If $P$ contains points arbitrarily close to zero, then $P$ is dense by 1., so $P=\mathbb R^+$. So we can assume that $P\subset[\epsilon,\infty)$ for some $\epsilon>0$, and that $P$ is nowhere dense.
Added: After speaking to several more experienced people about your second problem (as I've phrased it above), it seems that it is likely false, but that constructing a counterexample may be difficult. Here's the gist of the difficulty. If you take a totally disconnected perfect subset $E$ of $[1,2]$ and let $P=\bigcup nE$, then $P$ is a totally disconnected perfect subset of $\mathbb R^+$ satisfying 1. above. You can take $g$ to be a function which is constant on $P$, and extend $g$ to a continuous function $f$ on $[0,\infty)$ such that $\lim_{x\to\infty}{f(x)}$ does not exist. If $G$ is the complement of $P$, the difficulty lies in answering the question of whether $\lim_{n\to\infty}{f(nx)}$ necessarily exists for some $x\in G$. I'm not exactly sure how to do that, but if the answer is no, then we have a counterexample.
Finally, I'm not quite certain the original problem was asking for all of this. The wording in the book you cite is as follows:
Prove that if $f\in C([0,+\infty))$ and the limit $\lim_{n\to\infty}{f(nx)}$ exists for any $x\geq0$, then the limit $\lim_{x\to\infty}{f(x)}$ exists. Prove this if the limit $\lim_{n\to\infty}{f(nx)}$ exists only for points $x$ in some nonempty closed set without isolated points.
To take this quite literally, note that $\lim_{n\to\infty}{f(n\cdot0)}$ certainly exists, so the set on which $\lim_{n\to\infty}{f(nx)}$ exists is a perfect set $E$ containing zero with the property 1. above, i.e., $x\in E\implies nx\in E$ for all positive integers $n$. From this it follows that $E=[0,\infty)$, and the problem reduces to the initial question. It was the word "nonempty" in the hypotheses that initially led me to believe that this was not the authors' intended interpretation, but the more I think about it, the more likely it seems that it was (especially considering the fact that the book is translated).