The set of all $x$ such that $xHx^{-1}\subseteq H$ is a subgroup, when $H\leq G$

The reason you are having trouble proving it is that it is not true as stated.

For a heavy-handed example, let $G$ be the free group on $x$ and $y$, and let $H$ be the subgroup generated by all elements of the form $x^nyx^{-n}$ with $n\gt 0$.

Then for any $a\in H$ we have $xax^{-1}\in H$. However, $x^{-1}yx\notin H$, because any element of $H$ is a word that starts with a nonnegative power of $x$.

To fix the problem, you would need to require $xHx^{-1}=H$, rather than $xHx^{-1}\subseteq H$. Then your argument would go through in the infinite case as well.

Okay, here's an example you can get your hands on (courtesy my Math 257 notes with T.Y. Lam, Spring 97):

Let $G$ be the group of all invertible $2\times 2$ matrices with coefficients in $\mathbb{Q}$, $G=\mathrm{GL}_2(\mathbb{Q})$.

Let $H$ be the subgroup given by $$ H = \left\{\left.\left(\begin{array}{cc} 1 & m\\ 0 & 1\end{array}\right)\in G\ \right|\ m\in\mathbb{Z}\right\}.$$ Let $$x = \left(\begin{array}{cc} 2 & 0\\ 0 & 1\end{array}\right), \qquad x^{-1} = \left(\begin{array}{cc} \textstyle\frac{1}{2} & 0 \\ 0 & 1 \end{array}\right).$$ Then for every $m\in\mathbb{Z}$ we have: $$\begin{align*} \left(\begin{array}{cc} 2 & 0\\ 0 & 1\end{array}\right)\left(\begin{array}{cc}1 & m\\0 & 1\end{array}\right) \left(\begin{array}{cc}\textstyle\frac{1}{2}&0\\0&1\end{array}\right) &= \left(\begin{array}{cc} 2 & 2m\\0 & 1\end{array}\right)\left(\begin{array}{cc} \textstyle\frac{1}{2}&0\\0 & 1\end{array}\right)\\ &= \left(\begin{array}{cc} 1 & 2m\\ 0 & 1\end{array}\right)\in H. \end{align*}$$ So $xHx^{-1}\subseteq H$. However, even though $$ \left(\begin{array}{cc}1&1\\0&1\end{array}\right)\in H,$$ we have $$\begin{align*} x^{-1}\left(\begin{array}{cc}1&1\\0&1\end{array}\right)x &= \left(\begin{array}{cc} \textstyle\frac{1}{2} & 0\\0 &1\end{array}\right)\left(\begin{array}{cc}1 & 1\\ 0 & 1\end{array}\right)\left(\begin{array}{cc}2 & 0\\0&1\end{array}\right)\\ &= \left(\begin{array}{cc} \textstyle\frac{1}{2}&\textstyle\frac{1}{2}\\0&1\end{array}\right)\left(\begin{array}{cc}2&0\\0&1 \end{array}\right)\\ &= \left(\begin{array}{cc} 1 & \textstyle\frac{1}{2}\\0&1\end{array}\right)\notin H. \end{align*}$$ So $x\in \{g\in G\mid ghg^{-1}\in H\text{ for all }h\in H\}$, but $x^{-1}\notin\{g\in G\mid ghg^{-1}\in H\text{ for all }h\in H\}$. So the set need not be closed under inverses.