Derivation of the polarization identities?

For a real (or complex) inner product space $V$, the inner product can be expressed in terms of the norm as either $$ \langle x,y\rangle=\frac{1}{4}(\|x+y\|^2-\|x-y\|^2) $$ or $$ \langle x,y\rangle=\frac{1}{4}(\|x+y\|^2-\|x-y\|^2+i\|x+iy\|^2-i\|x-iy\|^2) $$ respectively.

These identities are not hard to verify directly once they are given. I would like to know, is there a systematic way one would derive these from scratch without knowing them beforehand, or are both polarization identities just clever observations?

I explained in the comments to the question how the polarization identity for real inner products can be discovered by looking at the associated quadratic form at $x+y$ and $x-y$, which in their separate ways lead to a sum of terms that involves a single term $B(x,y)$. Here I'll discuss the complex case.

Let $H(x,y)$ be a Hermitian form, so $\overline{H(x,y)} = H(y,x)$ and the single-variable function $Q(x) = H(x,x)$ satisfies $Q(cx) = H(cx,cx) = c\overline{c}H(x,x) = |c|^2Q(x)$ for $c \in {\mathbf C}$. Let's play the same game as in the real case by looking at $Q(x+y)$ and $Q(x-y)$: \begin{eqnarray*} Q(x+y) & = & H(x+y,x+y) \\ & = & H(x,x) + H(x,y) + H(y,x) + H(y,y) \\ & = & Q(x) + 2{\rm Re}(H(x,y)) + Q(y) \end{eqnarray*} and \begin{eqnarray*} Q(x-y) & = & H(x-y,x-y) \\ & = & H(x,x) - H(x,y) - H(y,x) + H(y,y) \\ & = & Q(x) - 2{\rm Re}(H(x,y)) + Q(y). \end{eqnarray*} Therefore we can solve for the real part of $H(x,y)$ by subtracting the second result from the first: $$ Q(x+y) - Q(x-y) = 4{\rm Re}(H(x,y)) \Longrightarrow {\rm Re}(H(x,y)) = \frac{1}{4}(Q(x+y) - Q(x-y)). $$ That is the first part of the complex polarization formula. To get a formula for the imaginary part, note that ${\rm Im}(H(x,y)) = {\rm Re}(-iH(x,y)) = {\rm Re}(H(x,iy))$, so if we run through the above work with $iy$ in place of $y$ then we get $$ {\rm Im}(H(x,y)) = {\rm Re}(H(x,iy)) = \frac{1}{4}(Q(x+iy) - Q(x-iy)). $$ If we write $H(x,y)$ as ${\rm Re}(H(x,y)) + i{\rm Im}(H(x,y))$ and feed the formulas for the real and imaginary parts of $H(x,y)$ into this, the complex polarization formula appears.

We don't have to use $Q(x+y)$ and $Q(x-y)$, or $Q(x+iy)$ and $Q(x-iy)$; just one of each would suffice: since $Q(x+y) = Q(x) + 2{\rm Re}(H(x,y)) + Q(y)$, we have $$ {\rm Re}(H(x,y)) = \frac{1}{2}(Q(x+y) - Q(x) - Q(y)), $$ and $$ {\rm Im}(H(x,y)) = {\rm Re}(H(x,iy)) = \frac{1}{2}(Q(x+iy) - Q(x) - Q(iy)), $$ and $Q(iy) = Q(y)$, so \begin{eqnarray*} H(x,y) & = & {\rm Re}(H(x,y)) + i{\rm Im}(H(x,y)) \\ & = & \frac{1}{2}(Q(x+y) - Q(x) - Q(y) + i(Q(x+iy) - Q(x) - Q(y))) \\ & = & \frac{1}{2}(Q(x+y) + iQ(x+iy) - (1+i)Q(x) - (1+i)Q(y)). \end{eqnarray*}

Finally, what happens for a Hermitian form on a vector space over an arbitrary field? Let $V$ be a vector space over a field $F$ and $\sigma$ be an automorphism of $F$ with order $2$. If $H \colon V \times V \rightarrow F$ is $\sigma$-Hermitian, so $H(w,v) = \sigma(H(v,w))$, then is $H$ determined by the function $Q(v) = H(v,v)$? For $a \in F$, $$ Q(av+w) = ({\rm N}a)Q(v) + Q(w) + {\rm Tr}(aH(v,w)), $$ where ${\rm N}$ and ${\rm Tr}$ are the norm and trace maps $F \rightarrow F^\sigma$ (where $F^\sigma$ is the fixed field of $\sigma$ on $F$, so the extension $F/F^\sigma$ has degree 2 and is Galois). From the nondegeneracy of the trace pairing on separable extensions, such as $F/F^\sigma$, the above formula as $a$ varies in $F$ shows $Q$-values determine $H$-values, even if $F$ has characteristic 2, when the usual polarization formula over $\mathbf C$, with division by 4, makes no sense.

Let me give a different sort of explanation as to where the formula comes from, at least in the real case.

Say we have a normed vector space $V$ whose norm is known to come from some (unknown) inner product and we want to compute $\langle x, y \rangle$ for two vectors $x, y \in V$. Let $W$ be the subspace spanned by $x$ and $y$. (Assume $x$ and $y$ are linearly independent for the moment.)

If we can indeed reconstruct the inner product from the norm, then $W$ may as well be $\mathbb{R}^2$ with the usual dot product. Drawing $x$ and $y$, we have a picture like this:

It should be obvious from the picture that $t$, and hence the inner product, is uniquely determined. Of course

$$\langle x, y \rangle = ||x|| \, ||y|| \cos(t),$$

and by the law of cosines we have

$$||x+y||^2 = ||x||^2 + ||y||^2 - 2 ||x|| \, ||y|| \cos (\pi - t),$$

$$||x-y||^2 = ||x||^2 + ||y||^2 - 2 ||x|| \, ||y|| \cos (t).$$

Subtracting and substituting, we obtain

$$||x+y||^2 - ||x-y||^2 = 4\langle x, y \rangle.$$

Hopefully that gives a bit more intuition for where the equations are coming from.