How to solve this system of ODEs
Solution 1:
REVISED.
Assume the equation $$ f(x)=\exp\left(-i\int^{x}_{0}\cos(a(y))b'(y)dy\right)[a'(x)+i\sin(a(x))b'(x)],\tag 1 $$ where $f:\textbf{R}\rightarrow\textbf{C}$ and $a(x),b(x)$ are $\textbf{R}\rightarrow\textbf{R}$.
Seting $a(y)=t$ in (1) we get $$ f(x)=\exp\left(-i\int^{a(x)}_{a(0)}\cos(t) b'\left(a^{(-1)}(t)\right)a^{(-1)}{'}(t)dt\right)[a'(x)+i\sin(a(x))b'(x)]\Leftrightarrow $$ $$ f\left(a^{(-1)}(x)\right)=\exp\left(-i\int^{x}_{a(0)}\cos(t)\left(b\left(a^{(-1)}(t)\right)\right)'dt\right)\times $$ $$ \times [\frac{1}{a^{(-1)}{'}(x)}+i\sin(x)b'\left(a^{(-1)}(x)\right)]\Leftrightarrow $$ $$ a^{(-1)}{'}(x)f\left(a^{(-1)}(x)\right)=\exp\left(-i\int^{x}_{c}\cos(t)g'(t)dt\right)[1+i\sin(x)g'(x)],\tag 2 $$ where $c_0=a(0)$ and $g(x)=b\left(a^{(-1)}(x)\right)$. Hence (2) becomes $$ \frac{d}{dx}\int^{a^{(-1)}(x)}_{c_1}f(t)dt=e^{-iY}-e^{-iY}(-i\cos(x)g'(x))\tan(x),\tag 3 $$ where $$ Y=Y(x):=\int^{x}_{c_0}\cos(t)g'(t)dt. $$ Hence if we also set $$ H(x):=\int^{a^{(-1)}(x)}_{c_1}f(t)dt\tag 4 $$ and $f(t)=f_1(t)+if_2(t)$, with $f_1,f_2$ real, then $$ H_{j}(x)=\int^{a^{(-1)}(x)}_{c_0}f_j(t)dt\in\textbf{R}\textrm{, } j=1,2.\tag{4.1} $$ Hence $$ H(x)=H_1(x)+iH_2(x). $$ But then $$ H'(x)=e^{-iY}-\frac{d\left(e^{-iY}\right)}{dx}\tan(x)\tag 5 $$ Setting lastly $e^{-iY}=w(x)$, we get $$ H'(x)=w(x)-w'(x)\tan(x)\Leftrightarrow $$ $$ w(x)=-\sin(x)\int^{x}_{c_2}\frac{\cos(x)}{\sin^2(x)}H'(x)dx\Leftrightarrow $$ $$ \exp\left(-i\int^{x}_{c_0}\cos(t)g'(t)dt\right)=-\sin(x)\int^{x}_{c_2}\frac{\cos(t)}{\sin^2(t)}H'(t)dt.\tag 6 $$ Hence $$ \cos\left(\int^{x}_{c_0}\cos(t)g'(t)dt\right)=-\sin(x)\int^{x}_{c_2}\frac{\cos(t)}{\sin^2(t)}H_1'(t)dt\tag 7 $$ and $$ \sin\left(\int^{x}_{c_0}\cos(t)g'(t)dt\right)=\sin(x)\int^{x}_{c_2}\frac{\cos(t)}{\sin^2(t)}H_2'(t)dt\tag 8 $$
Solving (8) with respect to $g'(x)$, we get $$ \int^{x}_{c_0}\cos(t)g'(t)dt=\sin^{(-1)}\left(\sin(x)\int^{x}_{c_2}\frac{\cos(t)}{\sin^2(t)}H_2'(t)dt\right)\Rightarrow $$ $$ g(x)=b\left(a^{(-1)}(x)\right)= $$ $$ =\int^{x}_{c_3}\frac{1}{\cos(u)}\frac{d}{du}\left[\sin^{(-1)}\left(\sin(u)\int^{u}_{c_2}\frac{\cos(t)}{\sin^2(t)}H_2'(t)dt\right)\right]du. $$ Hence $$ b(x)=\int^{a(x)}_{c_3}\frac{\csc^2(u)H_2'(u)}{\sqrt{1-\left(\int^{u}_{c_2}\cot(t)\csc(t)H_2'(t)dt\right)^2}}du,\tag 9 $$ Now from (7) and (8), we have $$ \left(\int^{x}_{c_2}\frac{\cos(t)}{\sin^2(t)}H_1'(t)dt\right)^2+\left(\int^{x}_{c_2}\frac{\cos(t)}{\sin^2(t)}H_2'(t)dt\right)^2=\frac{1}{\sin^2(x)} $$ Hence we can assume that exists real function $\phi(x)$ with inverse $\theta(x)$ such that $$ \sin(x)\int^{x}_{c_2}\frac{\cos(t)}{\sin^2(t)}H_1'(t)dt=\cos(\phi(x))\tag{10} $$ and $$ \sin(x)\int^{x}_{c_2}\frac{\cos(t)}{\sin^2(t)}H_2'(t)dt=\sin(\phi(x))\tag{11} $$ Hence $$ \int^{\theta(x)}_{c_2}\frac{\cos(t)}{\sin^2(t)}H_1'(t)dt=\frac{\cos(x)}{\sin(\theta(x))} $$ and $$ \int^{\theta(x)}_{c_2}\frac{\cos(t)}{\sin^2(t)}H_2'(t)dt=\frac{\sin(x)}{\sin(\theta(x))} $$ Differentiating these formulas we get $$ \sin(x)+\cot(\theta(x))[\cos(x)+H_1'(\theta(x))\theta'(x)]=0\tag{12} $$ and $$ \cos(x)-\cot(\theta(x))[\sin(x)+H_2'(\theta(x))\theta'(x)]=0\tag{13} $$ Solving (12) with respect to $\theta'(x)$, we get $$ \theta'(x)=-\frac{\sin(x)\tan(\theta(x))}{\cos(x)+H_1'(\theta(x))}.\tag{14} $$ Setting this value of $\theta'(x)$ in (13) we get $$ H_2'\left(\theta(x)\right)=-\csc(x)-\cot(x)H_1'(\theta(x)).\tag{15} $$ Now using (4),(4.1) we get $$ a^{(-1)}{'}\left(\theta(x)\right)=\frac{-1}{\cos(x)f_1\left(a^{(-1)}\left(\theta(x)\right)\right)+\sin(x)f_2\left(a^{(-1)}\left(\theta(x)\right)\right)}\tag{15.1} $$ and $$ \theta'(x)=\tan(\theta(x))\frac{\cos(x)f_1\left(a^{(-1)}\left(\theta(x)\right)\right)+\sin(x)f_2\left(a^{(-1)}\left(\theta(x)\right)\right)}{\sin(x)f_1\left(a^{(-1)}\left(\theta(x)\right)\right)-\cos(x)f_2\left(a^{(-1)}\left(\theta(x)\right)\right)}.\tag{15.2} $$ Notes.
$f'(g(x))$ is not $(f(g(x)))'=f'(g(x))g'(x)$.
In general if $h(x)$ is the inverse of $f(x)$, then holds $f(h(x))=h(f(x))=x$ and $f'(h(x))=\frac{1}{h'(x)}$ and $h'(f(x))=\frac{1}{f'(x)}$.
Hence equivalently if $A(x)$ denotes the inverse of $a(x)$ and $\phi(x)$ the inverse of $\theta(x)$: $$ A'(x)=\frac{-1}{\cos(\phi(x))f_1\left(A\left(x\right)\right)+\sin(\phi(x))f_2\left(A\left(x\right)\right)}\tag{16} $$ $$ \phi'(x)=\cot(x)\frac{\sin(\phi(x))f_1\left(A\left(x\right)\right)-\cos(\phi(x))f_2\left(A\left(x\right)\right)}{\cos(\phi(x))f_1\left(A\left(x\right)\right)+\sin(\phi(x))f_2\left(A\left(x\right)\right)}\tag{17} $$
For given $f=f_1+if_2$ the equations (9),(16),(17) are determine the problem in the case of $a(x),b(x)$ real functions. The DE are not linear and quite involved. Hence very hard to solve.
REMARK.
If we assume that $b(x)$ not necesary real, then (with notation independent of that above):
Assume the equation $$ f(x)=\exp\left(-i\int^{x}_{0}\cos(a(y))b'(y)dy\right)[a'(x)+i\sin(a(x))b'(x)].\tag{R1} $$ Seting $a(y)=t$, we get $$ f(x)=\exp\left(-i\int^{a(x)}_{a(0)}\cos(t) b'\left(a^{(-1)}(t)\right)a^{(-1)}{'}(t)dt\right)[a'(x)+i\sin(a(x))b'(x)]\Leftrightarrow $$ $$ f\left(a^{(-1)}(x)\right)=\exp\left(-i\int^{x}_{a(0)}\cos(t)\left(b\left(a^{(-1)}(t)\right)\right)'dt\right)\times $$ $$ \times[\frac{1}{a^{(-1)}{'}(x)}+i\sin(x)b'\left(a^{(-1)}(x)\right)]\Leftrightarrow $$ $$ a^{(-1)}{'}(x)f\left(a^{(-1)}(x)\right)=\exp\left(-i\int^{x}_{c}\cos(t)g'(t)dt\right)[1+i\sin(x)g'(x)],\tag{R2} $$ where $c_0=a(0)$ and $g(x)=b\left(a^{(-1)}(x)\right)$. Hence (R2) becomes $$ \frac{d}{dx}\int^{a^{(-1)}(x)}_{c_1}f(t)dt=e^{-iY}-e^{-iY}(-i\cos(x)g'(x))\tan(x),\tag{R3} $$ where $$ Y=Y(x):=\int^{x}_{c_0}\cos(t)g'(t)dt. $$ Hence if we also set $$ H(x):=\int^{a^{(-1)}(x)}_{c_1}f(t)dt,\tag{R4} $$ then from (R3): $$ H'(x)=e^{-iY}-\frac{d\left(e^{-iY}\right)}{dx}\tan(x) $$ Setting lastly $e^{-iY}=w(x)$, we get $$ H'(x)=w(x)-w'(x)\tan(x)\Leftrightarrow $$ $$ w(x)=\left(C_1-\int \frac{\cos(x)}{\sin^2(x)}H'(x)dx\right)\sin(x)\Leftrightarrow $$ $$ \exp\left(-i\int^{x}_{c_0}\cos(t)g'(t)dt\right)=\sin(x)\left(C_1-\int^{x}_{c_2}\frac{\cos(t)}{\sin^2(t)}H'(t)dt\right),\tag{R5} $$ where $H(x)=\int^{a^{(-1)}(x)}_{c_0}f(t)dt$. Solving (R4) with respect to $g'(x)$, we get $$ -i\int^{x}_{c_0}\cos(t)g'(t)dt=\log\left(C_1\sin(x)-\sin(x)\int^{x}_{c_2}\frac{\cos(t)}{\sin^2(t)}H'(t)dt\right)\Rightarrow $$ $$ g(x)=b\left(a^{(-1)}(x)\right)= $$ $$ =i\int^{x}_{c}\frac{1}{\cos(u)}\frac{d}{du}\left[\log\left(\sin(u)\left(C_1-\int^{u}_{c_2}\frac{\cos(t)}{\sin^2(t)}H'(t)dt\right)\right)\right]du. $$ Hence $$ b\left(a^{(-1)}(x)\right)=i\log\left(\tan\left(\frac{x}{2}\right)\right)-i\int^{x}_{c_3}\frac{\csc^2(u)H'(u)}{C_1-\int^{u}_{c_2}\cot(t)\csc(t)H'(t)dt}du.\tag{R6} $$ Hence if $a(x)$ is any (real invertible and smooth) function, then $b(x)$ is given from (R6).