Composition of permutations left to right or right to left?

Unfortunately, both conventions are in use: some people evaluate from left to right, some from right to left. You simply have to know which convention is in use. In the exam problem you can actually infer the convention from the question. You need to evaluate $\gamma(i)$; depending on which convention is in force, this means evaluating either $\tau(i)$ first or $\tau^{-1}(i)$ first. The information given in the problem doesn’t tell you what $\tau^{-1}(i)$ is, but it does tell you that $\tau(i)=1$. Thus, you can be pretty sure that the convention in force here is left-to-right composition: perform $\tau$ first, then $o$, and finally $\tau^{-1}$. And of course when you do, everything works out just right: $\tau(i)=1$, $o(1)=1$, and then $\tau^{-1}(1)=i$, so $\gamma(i)=i$.

(By the way, you seem to be reversing left and right: in your $AB$ example you’re actually doing right-to-left evaluation, since you’re applying $B$ first.)