Exponential family representation of multi-variate Gaussians

I'm a bit stumped by the exponential family representation of a multi-variate Gaussian distribution. Basically, the exponential form is a generic form for a large class of probability distributions. The standard form is

$$f_X(x) = \exp[\theta' T(x) + F(\theta)]$$

where $\theta$ is a set of parameters (based on $\mu$ and $\Sigma$), $T(x)$ is a vector of sufficient statistics, and $F$ is a function of the parameters that ensures the distribution is a pdf, i.e., sums to one. For more information on this form, see http://www.cs.columbia.edu/~jebara/4771/tutorials/lecture12.pdf, http://en.wikipedia.org/wiki/Exponential_family, etc.

The "conversion" for a multi-variate Gaussian distribution to exponential family form is listed as

$$\theta = [\Sigma^{-1}\mu, -\frac{1}{2}\Sigma^{-1}]'$$ $$T(x) = [x, x x']'$$

but this is confusing because the outer product $x x'$ is a matrix and $-\frac{1}{2}\Sigma^{-1}$ is also a matrix. Thus, it seems the product between $\theta$ and $T(x)$ should result in a scalar "entry" and a matrix "entry". Obviously, this expression needs to evaluate to a scalar.

The inner product works fine in the scalar case, and I understand this conversion is computed by manipulating to the quadratic form $x'Ax + b'x$. Still, it seems that I am completely missing something here. Thanks for your help.

Solution 1:

It has been such a long time since you asked but I just want to give a proper answer with full equations.

$\begin{align*} p(x) &= \frac{1}{\sqrt{\left|2\pi \Sigma \right|}} \exp \left\{-\frac{1}{2}\left(x-\mu\right)'\Sigma^{-1}\left(x-\mu \right) \right\}\\ &= \exp \left\{-\frac{1}{2}\log\left(\left|2\pi\Sigma \right| \right) \right\}\exp \left\{-\frac{1}{2}\left(x-\mu\right)'\Sigma^{-1}\left(x-\mu \right) \right\}\\ &= \exp \left\{-\frac{1}{2}\left[\underbrace{x'\Sigma^{-1}x - 2\mu'\Sigma^{-1}x}_{\theta'T\left(x\right)} + \mu'\Sigma^{-1}\mu + \log \left(\left|2\pi\Sigma\right| \right)\right] \right\} \end{align*}$

To rearrange the original equation into the form of an exponential family, we need to use the relationship between Frobenius product and vectorizing operator.

$\begin{align*} x'\Sigma^{-1}x &= \Sigma^{-1}:xx'\\ &= \operatorname{vec}\left(\Sigma^{-1}\right)' \,\operatorname{vec}\left(xx' \right) \\ \mu' \Sigma^{-1} x &= \left(\Sigma^{-1}\mu \right)'x\end{align*}$

$\therefore x'\Sigma^{-1}x - 2\mu'\Sigma^{-1}x = \begin{bmatrix}\operatorname{vec}\left(\Sigma^{-1}\right) \\ -2\Sigma^{-1}\mu \end{bmatrix}'\begin{bmatrix}\operatorname{vec}\left(xx'\right) \\ x \end{bmatrix}$