The "inverse" of $\nabla\times$ operator
Solution 1:
First, note that we expect such an inverse will be nonlocal, since the inverse of the standard differential operator $d/dx$ is the integral $\int_a^x \, dx'$.
Suppose in the simplest case we have the simply connected domain $\mathbb{R}^3$ (with enough decay at infinity that whatever integrals we write down will converge), and we are trying to solve $$ \nabla \times A = B $$ for $A$. Taking another curl gives $$ \nabla \times (\nabla \times A) = \nabla \times B, $$ and it looks like I've made things worse. But we have $$ \nabla \times (\nabla \times A) = \nabla(\nabla \cdot A)-\nabla^2 A, $$ where the last term is the vector Laplacian. Now, if we can say that $\nabla \cdot A=0$ (which it's admittedly not clear is possible; let's come back to that), then we need to solve the equation $$ -\nabla^2 A = \nabla \times B. $$ But if we use Cartesian coordinates, the vector Laplacian acts like the ordinary Laplacian on each component of $A$; therefore, we can invert it using the Green's function for the Laplacian (if you haven't met this, it's given by the solution to $-\nabla_x^2 G(x-y) = \delta(x-y)$ satisfying the right boundary conditions), which in this case is $-1/(4\pi |x-y|)$. Then we define $$ A_B(x) = \int_{\mathbb{R}^3} G(x-y) (\nabla \times B)(y) \, dy=\int B(y) \times [\nabla G(x-y)] \, dy, $$ integrating by parts. Does this work? Well, $$ \nabla \times (X \times a) = (a\cdot \nabla)X-(\nabla \cdot X)a, $$ so we have $$ \nabla \times A_B = \int (B(y) \cdot \nabla)\nabla G(x-y) \, dy + \int B(y) (-\nabla^2 G(x-y)) \, dy; $$ the former term is zero because if we integrate it by parts, we get a $\nabla \cdot B$, which is zero since $B$ is supposed to be the curl of something. The second term is just $B(x)$ by the definition of the Green's function!
(Some more care is needed in the above: deciding how to actually turn the $\nabla_x$ into a $\nabla_y$ and so on, but that's the right idea.)
Okay, that works. Now let's tidy up. We firstly want to show that we can take $\nabla \cdot A=0$. Suppose we define $\Lambda$ so that $-\nabla^2\Lambda=\nabla \cdot A$ (easy enough, using the Green's function). But then $A_{\Lambda}=A+\nabla \Lambda$ also solves $\nabla \times A_{\Lambda} = B$, and has zero divergence. This also tells us how to get from our $A_B$ to a more general $A$ that does not have $\nabla \cdot A=0$: add on a gradient of something.
Solution 2:
You have the Helmholtz decomposition in physics:
$$ {\bf F} = -\nabla \Phi + \nabla \times {\bf A}$$
which say that the differential parts of a vector fields can be decomposed as the sum of a rotation-free (scalar potential) part and a rotational part ( the curly one ). Therefore it should be impossible to "invert", as the curl only captures part of the vectors which is not part of the scalar potential. While an inverse therefore is impossible we can probably find a suitable generalized inverse or pseudoinverse, usually assuming that the missing components are 0.
Solution 3:
There are some special cases. Here's one from Electro Magnetic Wave Guides. Here you can invert a curl by taking the cross product of a curl with a part of which it might be composed.
Let $\vec{v}=\psi\vec{A}$
$\nabla \times \vec{v}=\nabla\psi\times\vec{A}+\psi\nabla\times\vec{A}$
$\nabla \psi\times(\nabla \times \vec{v})=\nabla\psi(\nabla \psi \cdot \vec{A})-\vec{A}(\nabla \psi)^2+\psi\nabla\psi\times(\nabla \times \vec{A})$
$\vec{A}(\nabla\psi)^2=\nabla\psi(\nabla \psi\cdot \vec{A})-\nabla\psi\times(\nabla \times \vec{v})+\psi\nabla\psi\times(\nabla \times \vec{A})$
$\psi\vec{A}(\nabla \psi)^2=\psi\nabla\psi(\nabla \psi \cdot \vec{A})-\psi\nabla\psi\times(\nabla \times \vec{v})+\psi^2\nabla\psi\times(\nabla \times \vec{A})$
$\vec{v}=\frac{\psi}{(\nabla \psi)^2}[\nabla\psi(\nabla \psi \cdot \vec{A})-\nabla\psi\times(\nabla \times \vec{v})+\psi\nabla\psi\times(\nabla \times \vec{A})]$
If $\vec{A}$ is irrotational, then $\nabla \times \vec{A}=0$.
If $\nabla \psi$ is orthogonal to $\vec{A}$, then $\nabla \psi \cdot \vec{A}=0$
So if those conditions hold, we have :
$\vec{v}=\frac{-\psi}{(\nabla\psi)^2}\nabla\psi\times(\nabla \times \vec{v})$
In a wave guide problem, $\vec{A}$ is usually chosen to represent direction of propagation, often then a vector function of $z$ only and having only a $z$ component. So it's irrotational.
The scalar $\psi$ is chosen to represent some properties of the waves which typically oscillate perpendicularly to the direction of propogation. It is usually just a function of $x$, and $y$ guaranteeing it's gradient is orthogonal to $\vec{A}$.
$\psi$ can be expressed in generic terms, say, requiring it to be a function of x and y. It can be further determined by solving the boundary conditions implied by Maxwell's Laws and the geometry of the wave guide.
The derivatives of $\vec{v}$ are sometimes easier to work with than $\vec{v}$ itself. If determined before hand this inversion process can be used to determine $\vec{v}$.