Lebesgue Spaces and Integration by parts

Integration by parts from the point of view of Lebesgue integration is more appropriately understood in terms of functions of local finite variation. Here is a version and then I comment more on your particular problem. I hope this is helpful to you.

Theorem: Let $F$, $G$ be right--continuous functions of locally finite variation on an interval $I$ (bound or unbounded) Let $\mu_F$ and $\mu_G$ the Stieltjes-Lebesgue measures generated by $F$ and $G$ respectively. For any compact $[a,b]\subset I$ $$ \int_{(a,b]}F(t)\mu_G(dt)=F(b)G(b)-F(a)G(a)-\int_{(a,b]}G(t-)\mu_F(dt) $$ where $G(t-)=\lim_{s\nearrow t}G(s)$.

A proof can be obtained using Fubini's theorem \begin{aligned} F(b)-F(a))(G(b)-G(a))&=\int_{(a,b]\times(a,b]}\mu_F\otimes\mu_G(dt,ds)\\ &=\int_{(a,b]}\Big(\int_{(a,s]}\mu_F(dt)\Big)\mu_G(s) +\int_{(a,b]}\Big(\int_{(s,b]}\mu_F(dt)\Big)\mu_G(ds)\\ &=\int_{(a,b]}\Big(\int_{(a,s]}\mu_F(dt)\Big)\mu_G(s) +\int_{(a,b]}\Big(\int_{(a,t)}\mu_G(ds)\Big)\mu_F(dt)\\ &=\int_{(a,b]} F(s)-F(a)\mu_G(s) +\int_{(a,b]}G(t-)-G(a)\mu_F(dt)\\ \end{aligned}

In the setting that you have in mind, you may have $\phi$ and $\psi$ that are absolutely continuous in an interval $[c,d]$. Then you $f=\phi'$ and $g=\phi'$ exists a.s. (with respect the Lebesgue measure in $\mathbb{R}$) and \begin{aligned} \phi(x)&=\phi(c)+\int^x_cf\\ \psi(x)&=\psi(c)+\int^x_cg \end{aligned} Then you may consider $F(x)=\phi(x)$ and $G(x)=\psi(x)$ in which case \begin{aligned} \mu_F((dt) &= \phi(c)\delta_c(dt) + f\mathbb{1}_{(c,d]}(t)\,dt\\ \mu_G(dt) &=\psi(c)\delta_c(dt) + g\mathbb{1}_{(c,d]}(t)\,dt \end{aligned} Applying the theorem above, you recover the usual integration by parts formula for $[a,b]\subset[c,d]$.

As for the boundary condition $\phi(t)\psi(t)|^b_a=0$, there many instances in which that happens. The important thing is to notice that $\phi$ and $\psi$ should be absolutely continuous to begin with.

For infinite intervals one has to use dominated convergence (integrability conditions on the derivatives $\phi'=f$ and $\psi'=g$) type of arguments. Typical situations are when both $|\mu_F|$ and $|\mu_F|$ are finite measures.

It is not true in general that $\phi \in L_1(\mathbb{R})$ implies $\lim_{x \to \infty} \phi (x) = 0$. The limit may not exist. For example, let $\phi (x) = 1$ for $x \in [n, n+2^{-n})$ for each natural number $n$ (including zero), and let $\phi \equiv 0$ otherwise. Then, we have

\begin{equation*} \int_{-\infty}^\infty |\phi| = \sum_{n=0}^\infty 2^{-n} = 2 < \infty, \end{equation*} but $\lim_{x \to \infty} \phi (x)$ does not exist. In fact, we can modify this example to construct an unbounded function which has no limit at infinity but belongs to $L^p$ for every finite $p$.

It is not clear to me what your question is. Typically the situations in which boundary terms for integration by parts vanish are when we know the functions are periodic (e.g. they are solutions to a PDE on which we impose periodic boundary conditions) or when one of the functions is compactly supported on the domain of integration.