Finding a fixed subfield of $\mathbb{Q}(t)$
Solution 1:
Let $F$ be the fixed field.
Since the symmetry group is finite, there is a trace map $\mathbf{Q}(t) \to F$:
$$ \operatorname{Tr}_{\mathbf{Q}(t) / F}(f(t)) = \sum_{\sigma \in S_3} f(\sigma(t)) $$
In fact, we can see that $f(t) \in F$ if and only if $f(t) = \frac{1}{6} \operatorname{Tr}_{\mathbf{Q}(t) / F}(f(t))$.
Usually, for problems like these, I just compute the trace of a few simple elements, and stop as soon as I've used them to generate a field where it's easy to see that $[\mathbf{Q}(t) : F] = 6$.
The norm map works too (as do any of the other elementary symmetric polynomials in the conjugates), but I usually use the trace map for this purpose.
Although now that I've thought about it some more, the norm map might be the better choice when working with a rational function field. Every automorphism of $K(t)$ is a permutation on the set of rational functions whose numerator and denominator are both degree 1 or less. Thus, the norm of such a rational function down to $K(t)^G$ will be a rational function whose numerator and denominator are of degree $d$ or less, where $d$ is the size of $G$.
Thus, if I pick some $f(t)$ whose numerator and denominator are degree 1 or less and set
$$ u := \frac{a(t)}{b(t)} := \operatorname{Nm}_{K(t) / K(t)^G}(f(t)) $$
then $b(t) u - a(t)$ is a polynomial of degree $|G|$ or less over the field $K(u)$ that has $t$ as a root.
If this polynomial is nonzero (i.e. if $u$ is not constant), then $t$ is a root of the polynomial, and we have
$$ [ K(t) : K(u) ] \leq |G| $$ $$ [ K(t) : K(t)^G ] = |G| $$ $$ K(u) \subseteq K(t)^G $$ and thus $$ [K(t)^G : K(u)] \leq 1 $$
and thus we see that $K(u) = K(t)^G$.
As an example, using $f(t) = t$, the algorithm above reproduces the fact that the fixed field of $t \mapsto 1-t$ is indeed $K(t(1-t))$ .
For your problem, $f(t) = t$ doesn't work, but $f(t) = 1+t$ does, and gives:
$$ u = -\left(\frac{(t+1)(t-2)(2t-1)}{t(t-1)} \right)^2 $$
Solution 2:
Consider the following rational function $$ w=\frac{(t^3-3t^2+1)(t^3-3t+1)}{t^2(t-1)^2}. $$ It is relatively easy to check that $w$ is invariant under both $\sigma:t\mapsto 1-t$ and $\tau:t\mapsto 1/t$. The automorphism $\sigma$ maps the factors in the numerator to the negatives of each other (check this), and clearly does the same to the factors of the denominator. Invariance under $\tau$ follows essentially from the observation that the factors in the numerator are reciprocal polynomialss of each other. The degrees match nicely to cancel the effect of $\tau$ to the denominator.
Clearly $t$ is a zero of the sextic polynomial $$ p(x)=(x^3-3x^2+1)(x^3-3x+1)-x^2(x-1)^2w\in\Bbb{Q}(w)[x], $$ so $[\Bbb{Q}(t):\Bbb{Q}(w)]\le6$. OTOH, as pointed out by Hurkyl, the fixed field $K$ of $G=\langle\sigma,\tau\rangle$ ($\subset Aut(\Bbb{Q}(t))$) satisfies $[\Bbb{Q}(t):K]=6$. We just saw that $\Bbb{Q}(w)\subseteq K$, so we can conclude that $$ \operatorname{Inv}(G)=\Bbb{Q}(w). $$
A `funny' thing happend in that $\operatorname{Inv}(G)\simeq\Bbb{Q}(t)$ (send $w$ to $t$). This is not a coincidence. Lüroth's theorem says that all the intermediate fields properly between $\Bbb{Q}$ and $\Bbb{Q}(t)$ are isomorphic to $\Bbb{Q}(t)$.
How did I find $w$? In this earlier question I used the trace to find a non-trivial element $$u=t+\frac1{1-t}+\frac{t-1}t=\frac{t^3-3t+1}{t(t-1)}$$ of the fixed field for the index two subgroup $\langle \sigma\tau\rangle\le G$. Here $w=-u\sigma(u)$ and that is clear the invariant of both $\sigma$ and $\sigma\tau$, hence all of $G$.