Extending continuous and uniformly continuous functions
If $x_n$ is a Cauchy sequence, then uniform continuity of $f$ allows us to conclude that the sequence $f(x_n)$ is also Cauchy.
The sequences $a_n=a+\frac{1}{n}$ and $b_n =b-\frac{1}{n}$ are Cauchy, hence so are the sequences $f(a_n)$, $f(b_n)$. Since $\mathbb{R}$ is complete, these sequences converge to some numbers $f_a, f_b$ respectively. Define the function $\overline{f}:[a,b] \to \mathbb{R}$ by $\overline{f}(a) = f_a$, $\overline{f}(b) = f_b$ and $\overline{f}(x) = f(x)$ for $x \in (a,b)$. Clearly $\overline{f}$ is continuous in $(a,b)$, it only remains to show continuity at $a,b$.
Suppose $x_n\in [a,b]$, and $x_n \to a$. We have $|\overline{f}(x_n) - \overline{f}_a| \leq |\overline{f}(x_n) - \overline{f}(a_n)| + | \overline{f}(a_n) - \overline{f}_a|$. Let $\epsilon >0$, then by uniform continuity, there exists $\delta>0$ such that if $|x-y|< \delta$, with $x,y \in (a,b)$, then $|f(x)-f(y)| < \epsilon$. Choose $n$ large enough such that $|x_n-a_n| < \delta$, and $| f(a_n) - f_a| < \epsilon$. If $x_n = a$, then $|\overline{f}(x_n) - \overline{f}_a| = 0$, otherwise we have $|\overline{f}(x_n) - \overline{f}_a| \leq |f(x_n) - f(a_n)| + | f(a_n) - f_a| < 2 \epsilon$. Consequently $\overline{f}(x_n) \to \overline{f}_a$, hence $\overline{f}$ is continuous at $a$. Similarly for $b$.
For the second case, take $f(x) = \frac{1}{x-a}$. Then $f$ is continuous on $(a,b)$, but the domain cannot be extended to $[a,b]$ while keeping $f$ continuous, and $\mathbb{R}$ valued. To prove this, take the sequence $a_n$ above, then $f(a_n) = n$, and clearly $\lim_n f(a_n) = \infty$. If $f$ could be continuously extended to $\overline{f}$, then $\overline{f}(a) \in \mathbb{R}$, which would be a contradiction.
HINT
- For the second one, consider something like the function $f(x) = \dfrac{x^2}{(x+1)(x-1)}$ as a function $f:(-1,1) \to \mathbb{R}$
- For the first one, it might just work out. Alternatively, show that if $f$ is uniformly continuous and $x_n$ is a Cauchy sequence in the domain of $f$, then $f(x_n)$ is a Cauchy sequence as well. Then pick a Cauchy sequence in the domain of $f$ that goes to $a$, for example.