Notation for repeated composition of functions
Solution 1:
Basically you can do anything. Notice, however, that the classical operators $\sum$ and $\prod$ do not coincide with the symbols they represent. And old books used to have $\sum$ instead of $\bigcup$ and $\prod$ instead of $\bigcap$.
I personally understand the notation $\mathop{\circ}_{n=1}^N$, but it doesn't look appealing. By analogy, why don't you define $$\mathop{\rm C}\limits_{n=1}^N $$ or $$\mathop{\rm K}\limits_{n=1}^N ?$$ Anyway, I don't like them either...
Solution 2:
Iterated summation and product are represented by the Greek uppercase letters sigma and pi. As we can see, there is a clear phoneticity here.
The only phonetic Greek equivalent of 'c' in 'composition' is kappa.
But as uppercase kappa resembles 'K', and is already used for continued fractions in some texts, we might use the lowercase kappa '$\kappa$' for our purpose:
$$\underset{i=1}{\overset{n}{\Huge{\kappa}}} f_i \left( x \right) \equiv f_n \left( f_{n-1} \left( \dots f_2 \left( f_1 \left( x \right) \right) \dots \right) \right)$$
Let's see an example where we can use this notation. Suppose we were to find the derivative of the composition of $n$ functions w.r.t. the variable they act on, say $x$ :
$\frac{d}{dx} f_n \left( f_{n-1} \left( \dots f_2 \left( f_1 \left( x \right) \right) \dots \right) \right)$
The derivative as computed using the chain rule recursively turns out to be,
$f_n^\prime \left( f_{n-1} \left( f_{n-2} \left( \dots f_2 \left( f_1 \left( x \right) \right) \dots \right) \right) \right) \cdot f_{n-1}^\prime \left( f_{n-2} \left( f_{n-3} \left( \dots f_2 \left( f_1 \left( x \right) \right) \dots \right) \right) \right) \dotsm f_2^\prime \left( f_1 \left( x \right) \right) \cdot f_1^\prime \left( x \right)$
But using the notations for iterative product and composition using $\prod$ and $\huge{\kappa}$ respectively, the expressions become much more compact:
$$\displaystyle{ \frac{d}{dx} \: \underset{i=1}{\overset{n}{\Huge{\kappa}}} f_i \left( x \right) = \prod_{i=1}^n f_i^\prime \left( \underset{j=1}{\overset{i-1}{\Huge{\kappa}}} f_j \left( x \right) \right) }$$
Solution 3:
If I were writing something in which I had to do a large number of these, the following is probably not quite what I would do:
$$
\overset{n}{\underset{k=0}\bigcirc}\ f_k \quad \text{ or } \quad \overset{0}{\underset{k=n}\bigcirc} f_k \ .
$$
Instead, I'd go over to tex.stackexchange.com and ask how to make this thing look respectable instead of like a workaround. I'd probably want it to be comparable in size and boldness to something like $\displaystyle\bigcap$ in $\displaystyle\bigcap_{k=0}^n A_k$ or to $\displaystyle\bigoplus$. Before the \begin{document}
I'd put \newcommand{\Circ}{blah blah blah}
(with a capital "C" distinguishing it from \circ
.
Solution 4:
Let $f_n(x)$ be a sequence of functions indexed by $n$.
Define a new sequence of functions $F_k(x)$ indexed by $k$:
$$ F_k(x) = \begin{cases} f_0(x) &: k=0\\ (f_k\circ F_{k-1})(x)&: k\gt 0 \end{cases} $$
So you have, for example $$ \begin{align} F_0(x) &= f_0(x)\\ F_1(x) &= (f_1 \circ f_0)(x)\\ F_2(x) &= (f_2 \circ F_1)(x) = (f_2 \circ f_1 \circ f_0)(x)\\ &\,\,\,\vdots\\ F_{n}(x) &= (f_n\circ f_{n-1} \circ \cdots \circ f_0)(x)\\ \end{align} $$
Then for your composition $f_n\circ f_{n-1} \circ \cdots \circ f_0$ of $n$ terms you can simply write $F_n$.
Of course you can define a new sequence $G_n$ for the ascending direction, i.e., $G_2 = f_0 \circ f_1 \circ f_2$.