Solution 1:

First, we can try integrating directly (summarizing comments). This yields

$$ f(x)=f(a)\exp \left(\int\limits_a^x dx' \frac{\lambda-B(x')}{A(x')} \right) $$

The requirement that $f(a)=f(b)$ sets $\int\limits_a^b dx \frac{\lambda-B(x)}{A(x)}=2\pi in$ with $n \in \mathbb{Z}$. If everything is real, and all the integrals exist, then $\lambda=\frac{\int dx \ B/A}{\int dx \ 1/A}$ so that there is just one eigenfunction with this eigenvalue.

To study the problem more generally, let us ask: "when is there a complete set of eigenfunctions?". This will be true if the operator is (truly) self-adjoint. Let

$$ L=A(x)\partial_x+B(x) $$

Define an inner product with real weight $w(x)>0$

$$ \left<v|u\right>=\int\limits_a^b dx \ v^*(x)u(x)w(x) $$

By direct substitution, we find that for real $A$, there is no weight such that $L$ is even formally self-adjoint. This is a bit of a surprise compared to the second order operators. Let $A$, $B$, and $f$ be complex. We will find conditions on $A$ and $B$ as well as $w$ such that $L$ is self-adjoint.

$$ \left<v|Lu\right>=\int\limits_a^b dx v^*Au'w+\int\limits_a^bv^*Buw $$

Integrating by parts

$$ \left<v|Lu\right>=v^*Auw \bigg|_a^b+\int\limits_a^b dx \ uw \left[(vB^*)^*-(v'A^*)^*- (vA{^*}')^*-(vA^*w'/w)^*\right] $$

From the terms in $[...]$ we find the formal adjoint

$$ L^\dagger = B^*-A^*\partial_x-A{^*}'-A^*w'/w \stackrel{?}{=} A \partial_x +B $$

So we require $-A^*=A$, which means $A$ must be pure imaginary. With this, the other terms read

$$ B^*+A'+Aw'/w=B $$

This may be solved for $w$

$$ w(x)=K\exp\left(\int\limits^x dx' \ \frac{B(x')-B^*(x')-A'(x')}{A(x')} \right) $$

With the integration constant $K$ which we may choose. In the special case $B \in \mathbb{R}$, we have $w(x)=\pm i/A(x)$, with sign chosen so $w>0$, which is only possible if $iA$ does not change sign over $a<x<b$. One may object to division by a possibly vanishing $A$ here: but now we can see that if the integral does not exist, or if $K$ cannot be chosen such that $w>0$, then there will be no $w$ such that $L$ is self-adjoint. Now the boundary terms. We require

$$ v^*Auw \bigg|_a^b=0 $$

For $u(a)=u(b)$ and those $v$ such that $v^*(a)=v^*(b)$.

$$ v^*(a)A(a)u(a)w(a)=v^*(b)A(b)u(b)w(b) $$

For these self-adjoint boundary conditions, none of the terms above may vanish, and we require

$$ A(a)w(a)=A(b)w(b) $$

This is automatically satisfied in the case $B \in \mathbb{R}$, $w=\pm i/A$, so long as $A$ does not vanish at either $a$ or $b$. I think this is as far as we can go in full generality: we have found conditions on $A$ and $B$ (and consequently $w$) for $L$ to possess a complete set of eigenfunctions.