Determine the $\sup(A)$ if $A = \{ \frac{(-1)^n}{n}: n \in \mathbb{N} \}$ and prove your result.

There is no need to separate the cases. No matter what $\epsilon$ you take, it is always true that you can choose $\frac12\in A$ (i.e., the element you get if you choose $n=2$).

Because $\frac12\in A$, it is clear that whatever $\epsilon>0$ you are given, you have

$$\frac12 > \frac12-\epsilon$$

and this is already enough.

In fact, this is a good starting point to prove a more general result:

If $m$ is an upper bound for $A$, and $m\in A$, then $m=\sup(A)$.

You can prove this statement more or less exactly from your lemma.

Since $\frac12$ is an upper bound of $A$ and since $\frac12\in A$, $\frac12=\sup A$.

If you want to use the lemma, you can do it this way: if $\varepsilon>0$, then, $\frac12-\varepsilon<\frac12$. Therefore, $\frac12=\sup A$.