Defining $|x|=-1$

Solution 1:

First of all, you can define $|\cdot|$ to mean whatever you want in any given context, as long as you're clear and upfront about it.

That being said, one usually wants $|\cdot|$ to be a norm, which means it fulfills a certain list of criteria. Among them is $|x|\geq 0$. If you break these rules, does your operation really deserve to be called "absolute value"? Does your operation deserve to be written using $|\cdot |$? Personally, I would say it doesn't, which means that using that symbol wouldn't be wrong, per se, but it would make it more difficult for your readers to understand what's going on, simply because of what they expect from that notation.

One notable exception, as pointed out in the comments, is the determinant of square matrices. And real / complex numbers are square matrices (of dimension $1\times1$), so in that context we really have $|-1|=-1$. But that's a different operation.

Solution 2:

The absolute value is quite a different thing than a square. A square simply comes from multiplication and nothing else. Especially, a square does not need an order on the underlying structure. However, the absolute value can only be defined after an order in defined by setting $$ |x| = \begin{cases} x & x\geq 0\\ -x & x < 0\end{cases}. $$ So, it is indeed defined to be non-negative. It is not that you may have some algebraic structure with an absolute value and then ask yourself "What if $|x|$ is negative?" in the same way you ask about squares… Put differently:

You can't deduce form the field axioms that $x^2 = -1$ has no solutions, but you can deduce from the axioms of the ordering that $|x|=-1$ has no solutions.

To answer the actual question: I haven't seen variant of absolute values (or norms, or metrics) to take negative values and doubt that such a thing has been studied.