What is the square root of a Fourier transform?

The Fourier transform can be diagonalised. Let $h_n$ be the Hermite functions, then we have $$ \mathcal F[h_n] = (-i)^n h_n. $$ So a good definition for a square root of the Fourier transform would be $$ \mathcal G[f] = \sum_{n=0}^\infty \langle f, h_n \rangle \mathcal G[h_n ] = \sum_{n=0}^\infty \langle f, h_n \rangle (-i)^{n/2} h_n. $$ A closed form expression can be found in the other answers.

You have an operator $\mathcal{F}$ that is unitary and satisfies $\mathcal{F}^4=1$. Let $r=e^{2\pi i/4}=i$. Define polynomials $$ p_k(z) = \frac{\prod_{l=0,l\ne k}^{3}(z-r^l)}{\prod_{l=0,l\ne k}^3(r^k-r^l)}. $$ Then $p_k(r^l)=\delta_{k,l}$ for $0 \le k,l \le 3$. Therefore, $$ p_1+p_2+p_3+p_4 = 1. $$ (This is because this third order polynomial sum is $1$ at the four points $r,r^2,r^3,r^4$.) Likewise, $$ p_1(\mathcal{F})+p_2(\mathcal{F})+p_3(\mathcal{F})+p_4(\mathcal{F})=I, \\ p_k(\mathcal{F})p_l(\mathcal{F})=0,\;\; k\ne l. \\ \mathcal{F}p_k(\mathcal{F})=r^{k-1}p_k(\mathcal{F}). $$ Consequently, $$ \mathcal{F} = p_1(\mathcal{F})+rp_2(\mathcal{F})+r^2p_3(\mathcal{F})+r^3p_3(\mathcal{F}). $$ Therefore, $$ S = p_1(\mathcal{F})+r^{1/2}p_2(\mathcal{F})+rp_3(\mathcal{F})+r^{3/2}p_4(\mathcal{F}) $$ satisfies $$ S^2 = p_1(\mathcal{F})+rp_2(\mathcal{F})+r^2p_3(\mathcal{F})+r^3p_4(\mathcal{F})=\mathcal{F}. $$ So there is a third-order polynomial in $\mathcal{F}$ that is a square root of $\mathcal{F}$.