References about Iterating integration, $\int_{a_0}^{\int_{a_1}^\vdots I_1dx}I_0\,dx$
Are there any references that discuss Iterating integration in general, $\int_{a_0}^{\int_{a_1}^\vdots I_1dx}I_0\,dx$, conditions in which they converge, some special values, some special tricks to compute them, for example $$\Large\int_0^{\displaystyle\int_0^{\displaystyle\int_0^{\vdots}(1-x)^3dx}(1-x)^2dx}(1-x)^1dx$$
Solution 1:
I have not seen this before. Instead of direct references, here are some thoughts and pointers to something else.
If you write $F_k(x)=\int_0^{x} I_k(t) dt$ then your question asks for what $F_0(F_1(F_2(F_3(\dots))))$ is, the limit of the sequence $$F_0,\, F_0\circ F_1,\, F_0\circ F_1\circ F_2,\, \dots$$ Clearly the limit (if it exists) is a function, not a number. If it is a constant function, it is the zero function. For any differentiable $F_k$, you can represent the iteration by integrals.
In the simplest case where all of your integrands are the same, you are looking at functional iteration. If you iterate analytic functions, you should expect nice convergence on a "Fatou set" and chaotic behaviour elsewhere.
There is a formal similarity between your iterated integrals and power towers $$a_0^{a_1^{a_2^{a_3^{\dots}}}}$$ so you could try similar approaches. For nice references, see this MSE question. Basically, there you can consider the functions $$a_0^x,\quad a_0^{a_1^x}, \quad a_0^{a_1^{a_2^x}},\dots$$ and then evaluate that limit at $x=1$.
Repeating what I said above, in your integral question, you can consider $$ \int_0^{x} f_0(t)dt,\quad \int_0^{\large\int_0^x f(x_1) dx_1} f_0(x_0)dx_0, \quad \int_0^{\large\int_0^{\int_0^x f_2(x_2)dx_2} f_1(x_1) dx_1} f_0(x_0)dx_0 $$ but it is less obvious what value of $x$ to use. You could consider for which values of $x$ your tower of integrals converges, though. $x=0$ is an obvious trivial choice (that you may want to avoid), where your tower of integrals collapses to zero.
You could alternatively try to find reasonable limits in a different way. Consider in your concrete example \begin{align}J_1&=\int_0^{J_2} (1-x) dx = \frac12(1- (1-J_2)^2)\\ J_2&=\int_0^{J_3} (1-x)^2 dx = \frac13(1-(1-J_3)^3)\\ &\dots \\ J_{k}&=\int_0^{J_{k+1}} (1-x)^k dx = \frac1{k+1} (1-(1-J_{k+1})^{k+1}) \end{align}
You could ask for which values of $J_1$ a sequence $(J_k)$ exists that satisfies this recursion relation. $J_1=0$ is an obvious trivial solution. Are there others?