Conjugate subgroup strictly contained in the initial subgroup?

Let $G$ be a group, $H\subseteq G$ a subgroup and $a\in G$ an element of the group. Is it possible that $aHa^{-1} \subset H$, but $aHa^{-1} \neq H$?

If $H$ has finite index or finite order, this is not possible.

Consider the group of matrices $$G=\left\{ \begin{bmatrix} x & y \\ 0 & 1 \end{bmatrix} : x \in \mathbb{Q}^{\times},y \in \mathbb{Q} \right\} = \operatorname{AGL}(1,\mathbb{Q}) $$ and its subgroup $$H=\left\{ \begin{bmatrix} 1 & y \\ 0 & 1 \end{bmatrix} : y \in \mathbb{Z} \right\} \cong \mathbb{Z}$$ and of course the single element $$a=\begin{bmatrix} 2 & 0 \\ 0 & 1 \end{bmatrix}$$ A direct calculation gives $$aHa^{-1} = \left\{ \begin{bmatrix} 1 & 2y \\ 0 & 1 \end{bmatrix} : y \in \mathbb{Z} \right\} < H$$ is a proper subgroup of H.

Similar issues showed up in this question.

First of all, the group $G$ has to be non-abelian. Otherwise any subgroup is normal and $a^{-1}Ha = H$ for each $a \in G$. You also need the subgroup $H$ (and thus $G$) to be infinite as you mentioned.

I'll give a different example. Let $G = S_{\mathbb{Z}}$, the group of bijections from $\mathbb{Z}$ to $\mathbb{Z}$.

For the subgroup, let $H = \{f \in G \mid f(x) = x \text{ for each } x \leq 0 \}$. Define the map $\sigma \in G$ by $\sigma(x) = x + 1$ for every $x \in \mathbb{Z}$.

It is not difficult to show that $\sigma H\sigma^{-1} \subseteq H$. Since $\sigma f \sigma^{-1}(1) = \sigma(f(0)) = \sigma(0) = 1$ for any $f \in H$, we notice that $\sigma H \sigma^{-1}$ contains only maps that fix $1$. Thus $\sigma H \sigma^{-1}$ cannot be all of $H$.

Let $G=\langle a,t;t^{-1}at=a^2\rangle$ and take $H=\langle a\rangle$. Noting that $\langle a^2\rangle\lneq\langle a\rangle$, clearly $t^{-1}H t\lneq H$. So this works.

If you are not comfortable with presentations, then here $G$ is actually just linear functions $f: \mathbb{R}\rightarrow\mathbb{R}$ generated by the function $x\mapsto x+1$ (which corresponds to the generator $a$) and the function $x\mapsto 2\cdot x$. The elements of $G$ then consist of all functions of the form $f(x)=2n\cdot x+m/2^k$. Then, $$t^{-1}at(x)=t^{-1}a(2x)=t^{-1}(2(x+1))=t^{-1}(2x+2)=x+2=a^2(x)$$ as required.

The group above is what is called a Baumslag-Solitar group. In general these groups are quite nasty, but this one is actually very nice (for example, it is metabelian and residually finite). In fact, we can replace the $2$ in the presentation with any non-zero integer and we get the same nice properties, and the same stuff happens which I have just outlined. Indeed, if the $2$ is replaced with an $i$, then you get a similar pair of linear functions. If I recall correctly, the function corresponding to $a$ is still $x\mapsto x+1$, but now $t$ becomes $x\mapsto i\cdot x$. If you want more details, then look up the book Groups, Graphs and Trees by John Meier.

Now, Baumslag-Solitar groups are what are called HNN-extensions of the infinite cyclic group $\mathbb{Z}$. If $H$ is some group which contains two isomorphic subgroups, $A$ and $B$ say with isomorphism $\phi: A\mapsto B$, then you can create a new group $G$ by using a stable letter to induce the isomorphism $\phi$. That is, $$G=\langle H, t; t^{-1}At=A\phi\rangle.$$ It is a theorem that $H$ embeds into $G$ in the natural way. My point? Well, if $H$ contains a pair of isomorphic subgroups $A$ and $B$ such that $A<B$ then you can use an HNN-extension to construct a group with the property we are discussing. Indeed, if $A<B=H$ (so $H$ contains a proper subgroup isomorphic to itself) then you can get a nice example of what is going on here. If $B=H$ then this is called an ascending HNN-extension. The Baumslag-Solitar examples above are ascending HNN-extensions of the infinite cyclic group.

Another example is as follows: all subgroups of a free group are free, so the group $$\langle a, b, t; a^t=ab, b^t=ba\rangle$$ is an example of what we are talking about.

HNN-extensions have a geometric visualisation. Take the infinite tree where every vertex has incoming edges labelled by the cosets $H/A=\{hA: h\in H\}$ and outgoing edges labelled by the cosets $H/B$, where $B=A\phi$. Then the group $G=\langle H, t; A^t=A\phi\rangle$ acts on this tree where $H$ fixes some vertex and acts on the incoming and outgoing edges by acting on the labels as left multiplication, while $t$ moves every vertex along the outgoing edge with label $H(=1H)$, the edge labelled by the coset containing the identity. This means that these examples have a similar flavour to an answer of Marc van Leeuwen to an identical question.

Moral: There are lots of examples of groups with subgroups having this property!