Prove continuity on a function at every irrational point and discontinuity at every rational point.
Solution 1:
This function is known as Thomae's function and is an excellent example of a function that is continuous at uncountably infinite number of points while discontinuous at countably infinite number of points. Note that the reverse result is not possible, i.e. one cannot have a function that is discontinuous on an uncountably infinite dense set and continuous on a countably infinite dense set.
As for the proof , the discontinuity part follows from the fact that irrationals are dense in $ R $ as you rightly thought. The continuity part is a little bit more involved and it would help to employ the Archimedean principle in tandem with the $ \epsilon$-$\delta $ definition.
Crudely, look at $ x=\sqrt3 $ , see that using the non-terminating decimal expansion i.e. $ \sqrt 3=1.732148\ldots $ , one can forge a sequence of rationals $ \ \ 1 , 17/10 ,173/100 ,\ldots \ $ that converges to $ x $, while at the same time your $ f(x_n) $ is $ \ \ 1/10,1/100,1/1000,\ldots \ $ which converges to $ 0 $ which is nothing but $ f(x) $. This is merely an illustration of the continuity and not a rigorous proof.
Solution 2:
You cannot do this using simple rules about limits, like de l?Hôpital's rule or the continuity of "analytic expressions", etc. You have to go back to the actual definition of continuity when solving this problem. Here are a few hints:
In the first place you want to prove different things about a given $x\in{\mathbb R}$, depending on whether $x$ is rational or irrational.
If $x$ is rational then $f(x)$ is a certain positive number. It is claimed that $f$ is not continuous at $x$. Assume to the contrary that $f$ is continuous at $x$. Then there is a neighborhood $U:=U_\delta(x)$ such that $f(t)>{f(x)\over2}$ for all $t\in U$. Hm?
If $x$ is irrational then $f(x)=0$. It is claimed that $f$ is continuous at $x$. This means that for any $\epsilon>0$ we should be able to produce a neighborhood $U:=U_\delta(x)$ such that $f(t)<\epsilon$ for all $t\in U$. Which points $t\in{\mathbb R}$ could violate the condition $f(t)<\epsilon\ $?