Why Composition and Dihedral Group have reverse order of operation?

There is absolutely no reason at all. It is entirely up to the author of the textbook which way these things go, and different textbooks will use different conventions for both the symmetric group and the dihedral group.

Whenever you define a group of transformations or operations, you have the choice as to whether $gh$ should mean "do $g$ first, and then $h$", or whether it should mean "do $h$ first and then $g$". These two choices correspond to two possible binary operations on the group, and you have to pick one of these operations.

Now, you might think it's obvious that $gh$ should mean "do $g$ first, and then $h$". However, that's not how composition of functions works. Recall that if $f$ and $g$ are functions, the composition $f\circ g$ is defined by the formula $$ (f\circ g)(x) = f(g(x)). $$ As you can see, in the composition $f\circ g$, the function $g$ is applied to $x$ first, and then $f$ is applied to the result.

So when you're defining a group, you have to decide between using the "obvious" order, and using the order that agrees with function composition. There isn't a good choice here, so some books use the "obvious" order, some books use the "backwards" order, and some books (such as yours, apparently) use different orders for different groups.

Incidentally, the reason that function composition is "backwards" is that everybody writes functions to the left of the input, which is a little strange when you think about it. Everything would be so much simpler if people wrote $(x)f$ instead of $f(x)$, for then function composition could be defined as $$ (x)(f\circ g) = ((x)f)g, $$ which would mean that $f$ would go first. But no one does this, and everyone is so used to functions being on the left that equations like this are actually a little hard to read.

I assume you're using the original labeling here. I think you made some mistake in your calculation, or are using inconsistent conventions. At the top of your post, you indicate that $r^2 s$ corresponds to (15). Now (15) is what you get by doing $s$ first, then $r^2$. But then later in your post, you seem to think that $r^2 s$ means "do $r^2$ first, then $s$".

If you use a consistent convention, then everything works out.

I define $r$ to be one rotation clockwise, and s to be reflection on the 'horizontal' line (see the figure).

As an example, I want to evaluate $rsr^2$. Without lose of generality, I choose right-to-left as order of action that is first $r$ then $s$ and $r^2$ acting on id (first configuration) on the figure. Pictorially:

So the final equivalence is $(12)(36)(45)$.

I am going to do the same 'direction of action' but now on permutation, that is first $r$ then $s$ and $r^2$ acting on id ($(1)$) on the number arrangements or permutation. Mathematically:

$\tau_1=r\circ\text{id}=(123456)(1)=(123456)\rightarrow \tau_2=s\circ\tau_1=(26)(35)(123456)=(16)(25)(34)\rightarrow\tau_3=r^2\circ\tau_2=(135)(246)(16)(25)(34)=(12)(36)(45)$.

Conclusion: The contradiction came from labeling one of 'outside and inside' of vertices not two of them. And, there is no revered order of actions comparing permutation and dihegral group.