Cross product and pseudovector confusion.

In three dimensions, pseudovectors are a simple way to treat bivectors, oriented planar subspaces. True vectors are oriented linear subspaces with a weight (their magnitudes); bivectors are planar instead of linear. The normal vectors to these oriented subspaces are what we usually call pseudovectors, and it is for this reason that various operations (like reflections or inversions through the origin) produce "wrong" results.

Notationally, we deal directly with a bivector by forming a wedge product of vectors. That is, the bivector formed by vectors $a,b$ is $a \wedge b$. Given a linear operator $\underline T$, we define the action of the linear operator on a bivector by the following law:

$$\underline T(a \wedge b) \equiv \underline T(a) \wedge \underline T(b)$$

Let us consider the simple case of $\underline T(a) = -a$ for any $a$. Then the associated bivector transforms as $\underline T(a \wedge b) = -a \wedge -b = a \wedge b$, as you observe. Doing it this way--by defining the action of a linear operator on a bivector--makes it sensible, rather than saying simply that pseudovectors transform differently from regular vectors. Here, you build the operator according to a specific rule, and the result is deterministic.

Note that we can continue to build things with wedges that traditional formulations of vector algebra and calculus tend to gloss over. We can define the action of a linear operator on three vectors wedged together.

$$\underline T(a \wedge b \wedge c) = \underline T(a) \wedge \underline T(b) \wedge \underline T(c)$$

The quantity $a \wedge b \wedge c$ is called a trivector or pseudoscalar. In three dimensions, there is only one linearly independent unit trivector, $\hat x \wedge \hat y \wedge \hat z$. The action of $\underline T$ on this object is very interesting. It happens that

$$\underline T(\hat x \wedge \hat y \wedge \hat z) = \hat x \wedge \hat y \wedge \hat z \, \det \underline T$$

This can be taken as a definition of the determinant, defined in a wholly geometric way.

Ultimately, though, yes, linear operators should act individually on the vectors that make them up--they should preserve wedge products. Cross products are related to wedges, however, and so most of the time, applying a linear operator to preserve crosses is sensible, but there are some times (inversion and reflections being among them) that it is not.

Edit: about the relations between operators on duals and duals of operators. The Hodge star is much, much better treated in geometric algebra as multiplication by the pseudoscalar. We define $i \equiv \hat x \wedge \hat y \wedge \hat z$ and make sense of expressions like $\star a = i a$ and $\star (a \wedge b) = -i (a \wedge b)$ through the geometric product. Here are the canonical properties of the geometric product:

• $\hat u \hat u = 1$ for some unit vector $\hat u$
• $\hat u \hat v = - \hat v \hat u$ for two orthogonal unit vectors $\hat u, \hat v$
• $(ab)c = a(bc)$--that is, associativity--for three vectors $a, b, c$

You should be able to show then that $i = \hat x \hat y \hat z$ and that $\star a = i a$ captures the Hodge star operation on a vector.

Now, why bother with this stuff? Because it makes formulas that would be ugly and clumsy with the Hodge star very simple. There exists a simple formula relating the adjoint (in Euclidean space, the transpose) of an operator with the inverse. That is,

$$\overline T^{-1}(a) = [\underline T(i)]^{-1} \underline T(ia)$$

for any multivector $a$, where $\overline T$ is the adjoint operator to $\underline T$. Written with Hodge stars, we would need a term of $(-1)^k$ that would alternate based on grade, and it would all be a royal mess. This formula, however, written in geometric algebra, is entirely simple.

Now then, rotations and reflections all belong to the group of orthogonal linear operators, obeying $\overline T^{-1} = \underline T$, so for rotations and reflections we get instead,

$$\underline T(a) = \frac{1}{\det \underline T} i^{-1} \underline T(ia)$$

or, more simply,

$$(\det \underline T) i \underline T(a) = \underline T(ia)$$

In Hodge star notation, for any vector $a$,

$$(\det \underline T) \star[\underline T(a)] = \underline T(\star a)$$

For a rotation, the determinant is $+1$, and as such, the $i$ just pulls out. Rotating the vector and then finding the dual is the same as rotating the dual. For an inversion, the determinant is $-1$, and you can see how the inversion of the vector gets canceled by the determinant's factor.