What is $-i$ exactly?

The vector $(1,-2)$ is also neither positive nor negative, yet there is value in having the operation $-(1,-2)=(-1,2)$.

Note: there is no natural meaning for "positive" for vectors, this is just an analogy.

You must take care, even though $2^2 = (-2)^2 = 4$ this doesn't mean that $2 = -2$, indeed the problem here (thinking about reals for a while), is that if you consider negative numbers, then the function $f: \mathbb{R}\to\mathbb{R}$ given by $f(x) = x^2$ is not injective.

Since you have created $i$ to be $i=\sqrt{-1}$, then you are obviously allowed to create a new number $-i = -\sqrt{-1}$. Also, if you think on the function $f: \mathbb{C} \to \mathbb{R}^2$ given by $f(a+bi) = (a,b)$, this function identifies $\mathbb{C}$ with $\mathbb{R}^2$, and in this sense, a complex number is a point in the plane. Note that we have $f(i) = (0,1)$ and that $f(-i) = (0,-1)$, so there are two things to note: every number that can be written $ai$ for some $a \in \mathbb{R}$ can be identified with a point in the $y$ axis, and introducing $-i$ we have not just the upper $y$ axis, but the complete one.

EDIT: Because of the comments I thought it would be good to add this edit. Above I've provided a geometrical view of what happens when we introduce $-i$, now I'm just going to say a little about the algebraic point of view. Since every complex number $z \in \mathbb{C}$ is $z=a+ib$ for some $a,b \in \mathbb{R}$ if we do not allow things like $-i$ we would have some issues. For instance, consider quadratic equations with complex conjugate roots: one of them would be a well defined element of $\mathbb{C}$, but the other one wouldn't! Also, it wouldn't be possible to present for each $z$ some $-z$ with the nice property $z + (-z) = 0$ and it turns out that $\mathbb{C}$ wouldn't be a field. Also, if you use the standard operations of sum between complexes and you have $z = a+ib$ and you want the additive inverse $-z$ you can prove rather easily that you must have $-z = (-a) + i(-b)$, so that $-i$ would just really be $i(-1)$, so in practice, the element $-i$ (and any other negative complex in general) appears naturally when we define the complexes as we do and when we introduce the operations as we do.

The imaginary number $i$ was discovered/invented/revealed as a solution the equation $$ x^2+1=0\tag{$\ast$} $$ That equation, and that equation alone, is the link between $i$ and the reals. However, there is another solution to $(\ast)$: $-i=-1\times i$ also satisfies $(\ast)$. Basically, this is because $(-1)^2=1$.

Since the link between $i$ and the reals is the same equation that is satisfied by $-i$, one can easily wonder, "which is which?" When we choose a solution to $(\ast)$, do we get $i$ or $-i$? In a sense, it doesn't matter; both satisfy $(\ast)$ and $i^2=-1$ is the property we use to do math in $\mathbb{C}$.

On the other hand, complex conjugation, swapping $i\leftrightarrow-i$, is an important isomorphism of $\mathbb{C}$. For instance, any polynomial with real coefficients that has $x+iy$ as a root, must also have $x-iy$ as a root. There are many ways to show this, but one of the most basic is by swapping $i\leftrightarrow-i$. This isomorphism does not affect the real coefficients of the polynomial, but it swaps $x+iy\leftrightarrow x-iy$.

In the end, we call one root of $(\ast)$, $i$, and the other, $-i$; it really doesn't matter which. They both satisfy $(\ast)$ and that is what is important.

I quickly scanned all the answers and didn't see anyone mention rotations, so I thought I would -- the simplest answer to your question is that $-i$ just represents a rotation of $\frac{\pi}{2}$ clockwise, whereas $i$ represents a rotation of $\frac{\pi}{2}$ counterclockwise. In other words, $i$ and $-i$ are inverses in the group $SO(2)\cong\left\{e^{i\theta}\,|\,\theta\in[0,2\pi)\right\}$ of rotations of the plane. From this perspective, $i$ and $-i$ are about as emphatically "different" as anyone could imagine.

EDIT: I thought I'd add that yet another way of saying what I've already said is that complex conjugation is just a way of swapping the orientation of the plane. (In particular, it is an orientation-reversing isometry.) The seemingly "sign-like" difference between $i$ and $-i$ is really just a difference in the choice of orientation.