Compact metric space group $Iso(X,d)$ is also compact

Without the precise definition of the metric $\rho$, I'm not sure how to answer this definitively, but let me outline an approach that you might be able to take. For this approach, I take as a hidden assumption that the Principle of Dependent Choices holds, so to show that $Y:=\text{Iso}(X,d)$ is compact (considered as a metric space with metric $\rho$), it suffices to show that it is complete and totally bounded.

To show that $\langle Y,\rho\rangle$ is a complete metric space, suppose that we have a $\rho$-Cauchy sequence of functions $h_n\in Y.$ Show that for each $x\in X,$ we have that $h_n(x)$ is a $d$-Cauchy sequence of points of $X$. Since $X$ is a compact space under the $d$-metric topology, then it is complete, so $h_n(x)$ converges in $X$. Define $h:X\to X$ by $h(x)=\lim_{n\to\infty}h_n(x).$ Show that $h\in Y,$ and that $\lim_{n\to\infty}\rho(h_n,h)=0,$ so $\langle Y,\rho\rangle$ is a complete metric space.

To show that $\langle Y,\rho\rangle$ is a totally bounded metric space, we must show that for all $\epsilon>0$ there exist $f_1,...,f_n\in Y$ such that $Y$ is covered by the open $\rho$-balls about $f_1,...,f_n$ of radius $\epsilon$. Without a definition for $\rho,$ I'm afraid I have no suggestions for how to proceed with this.

Edit: If julien is correct in his comment above, that you're simply considering $Y$ in the topology of pointwise convergence (which, in retrospect, it seems that you are), then I'm not certain that it's necessarily induced by a metric. Fortunately, we can proceed without worrying about such a metric.

Let's proceed as follows, instead. Denote the set of all functions $X\to X$ by $X^X$. Given a point $x\in X$ and an open set $U$ in $X,$ we define $$P(x,U):=\{f\in X^X:f(x)\in U\},$$ and let $\mathcal B$ be the set of all sets of the form $$P(x_1,B_1)\cap\cdots\cap P(x_k,B_k)$$ for some positive integer $k$, with the $x_j\in X$ and the $B_j$ open balls in $X$. You can show that $\mathcal B$ is a basis for a topology on $X^X$, say $\mathcal T$, which is the set of unions of $\mathcal B$-sets. We call $\mathcal T$ the topology of pointwise convergence, because of the following

Lemma: If $h_n$ is a sequence of functions $X\to X$ and $h:X\to X$, then $h_n\to h$ with respect to the topology $\mathcal T$ if and only if $h_n(x)\to h(x)$ for each $x\in X$.

Proof: Suppose $h_n(x)\to h(x)$ for all $x\in X$, and take any $\mathcal B$-basic neighborhood of $h$, which will be of the form $$P(x_1,B_1)\cap\cdots\cap P(x_k,B_k)$$ for some $x_j\in X$ and some open balls $B_j$ with $h(x_j)\in B_j$. Since $h_n(x_j)\to h(x_j)$, then there exists $n_j$ such that $h_n(x_j)\in B_j$ for $n\ge n_j$. Putting $N=\max\{n_1,...,n_k\}$, we therefore have for $n\ge N$ that $$h_n\in P(x_1,B_1)\cap\cdots\cap P(x_k,B_k).$$ Thus, $h_n\to h$ with respect to $\mathcal T$.

On the other hand, suppose that $h_n\to h$ with respect to $\mathcal T$. Take any $x\in X$ and any open ball $B$ in $X$ such that $h(x)\in B$. Since $P(x,B)$ is a $\mathcal T$-neighborhood of $h$ and $h_n\to h$, then there is some $N$ such that $h_n\in P(x,B)$ for $n\ge N$, meaning $h_n(x)\in B$ for all $n\ge N$. Thus, $h_n(x)\to h(x)$ for all $x\in X$. $\Box$

Now, julien refers to Tychonoff Theorem (a very useful theorem that requires the Axiom of Choice to prove) in his comment above. We really only need the following special case:

Theorem: $X^X$ is compact in the topology of pointwise convergence if (and only if) $X$ is compact.

The "only if" part is fairly easy to prove, and if you're interested, I can prove the other direction (though it can be a bit of a bear to prove without just using another form of the Tychonoff Theorem).

Hence, all that is left for you to do is to show that $Y$ is a closed subspace of $X^X$, from which the result follows.

Yesterday I was involved in this thread. Hoping to clarify some claims, I have some things to say so I decided to create a new answer instead of writing of big comments.

I shall use the following notations. Put $Y=\operatorname{Iso}(X,d)$. I consider the following three topologies defined on the set $Y$: the topology $\tau_u$ of the uniform convergence, the topology $\tau_p$ of the pointwise convergence, and the topology $\tau_\rho,$ determined by the metric $\rho$.

I should remark that the condition $$(*) \lim _{n \rightarrow \infty} \rho(h_n, h) =0 \iff \forall x \in X: \lim _{n \rightarrow \infty} d(h_n(x), h(x))=0,$$ is not a definition of the metric $\rho$, it is a property of the metric. So there arise a question: "how the topologies $\tau_u$, $\tau_p$, and $\tau_\rho$ on the set $Y$ are related?", which I shall answer in my answer. :-)

@Cameron Buie wrote:

To show that $\langle Y,\rho\rangle$ is a complete metric space, suppose that we have a $\rho$-Cauchy sequence of functions $h_n\in Y.$ Show that for each $x\in X,$ we have that $h_n(x)$ is a $d$-Cauchy sequence of points of $X$. Since $X$ is a compact space under the $d$-metric topology, then it is complete, so $h_n(x)$ converges in $X$. Define $h:X\to X$ by $h(x)=\lim_{n\to\infty}h_n(x).$ Show that $h\in Y,$ and that $\lim_{n\to\infty}\rho(h_n,h)=0,$ so $\langle Y,\rho\rangle$ is a complete metric space.

For me it was easier to show these claims in the opposite order, :-) see my answer here.

@Sandy wrote:

Why is $Iso(X,d)$ closed in $X^X$?

Suppose that a map $h$ is contained in the set $X^X\backslash Y$. I am going to find an open neighborhood $U$ of the map $h$ (in the topology of pointwise convergence on $X^X$), such that the intersection $U\cap Y$ is empty. At first I suppose that the map $h$ is not isometric of the space $(X,d)$. Then there exist points $x,y\in X$ such that $d(h(x),h(y))\not=d(x,y)$. Then it suffices to put $\varepsilon=|d(h(x),h(y))-d(x,y)|$ and $U=\{g\in X^X: d(g(x),h(x))<\varepsilon/2$ and $d(g(y),h(y))<\varepsilon/2\}$. If the map $h$ is isometric, but $h$ is still in $X^X\backslash Y$, then the map $h$ is not surjective. Therefore there exists a point $x\in X\backslash h(X)$. Fix a number $\varepsilon>0$ such that $\varepsilon<d(x,h(X))$. Since $X$ is a metric compact, there exists a finite $\varepsilon/2$-net $F$ of $X$, that is $F$ is a finite subset of $X$ such that for any point $y\in X$ there exists a point $z\in F$ such that $d(x,z)<\varepsilon/2$. Put $U=\{g\in X^X: (\forall z\in F)(d(g(z),h(z))<\varepsilon/2)\}$. Suppose that there exists a map $g\in U\cap Y$. Put $y=g^{-1}(x)$. Since the set $F$ is a $\varepsilon/2$-net $F$ of $X$, there exists a point $z\in F$ such that $d(y,z)<\varepsilon/2$. Since the map $g$ is an isometry of the space $(X,d)$, I have that $d(g(y),g(z))=d(y,z)<\varepsilon/2$. But then $d(x, h(z))=d(g(y),h(z)) \le d(g(y),g(z))+$ $d(g(z),h(z))< \varepsilon/2+\varepsilon/2=\varepsilon$, a contradiction.

Therefore the space $(Y,\tau_p)$ is compact.

Now I show that the topologies $\tau_u$ and $\tau_p$ coincide. The inclusion $\tau_p\subset\tau_u$ is obvious, so it rests to prove the opposite inclusion. Let $h\in Y$ and $\varepsilon>0$ be arbitrary. I am going to find an open neighborhood $U\in\tau_p$ of the map $h$ such that $d(g(x),h(x))<\varepsilon$ for each map $g\in U$ and each point $x\in X$. Since $X$ is a metric compact, there exists a finite $\varepsilon/3$-net $F$ of $X$. Put $U=\{g\in Y: (\forall z\in F)(d(g(z),h(z))<\varepsilon/3)\}$. Let $g\in U$ and $x\in X$ be arbitrary. Since the set $F$ is a $\varepsilon/3$-net $F$ of $X$, there exists a point $z\in F$ such that $d(x,z)<\varepsilon/3$. Since the maps $g$ and $h$ are isometries of the space $(X,d)$, I have that $d(g(x),h(x))\le d(g(x),g(z))+d(g(z),h(z))+$ $d(h(z),h(x))= d(x,z)+d(g(z),h(z))+d(z,x)<\varepsilon/3+\varepsilon/3+\varepsilon/3=\varepsilon.$

Hence, $\tau_u=\tau_p$.

How I show that $\tau_\rho\subset\tau_u$. Suppose the opposite. Then there exist a point $h\in Y$ and a number $\varepsilon>0$ such that for each natural number there is a point $h_n\in Y$ such that $d(h_n(x),h(x))<1/n$ for each $x\in X$, but $\rho(h_n,h)\ge\varepsilon$. Then a sequence $\{h_n\}$ uniformly converges to $h$. Then property (*) implies that $\rho(h_n,h)$ converges to $0$, contradicting the inequality $\rho(h_n,h)\ge\varepsilon$.

Since the space $(Y,\tau_u)$ is compact and $\tau_\rho$ is a Hausdorff topology on $Y$ weaker than $\tau_u$ then $\tau_\rho=\tau_u$, by well-known Theorem 3.1.10 from [Eng].

@julien wrote

it is metric if there exists a compact exhaustion of the space on which the isometries act. Still the case if no such exhaustion exists? If it is separable, I see that. But otherwise...

... there should be a counterexample. Let $X$ be an uncountable set endowed with a metric $d$ such that $d(x,y)=0$ if $x=y$ and $d(x,y)=1$ in the opposite case. Then $\operatorname{Iso(X,d)}$ is a group $S(X)$ of all bijections of the set $X$. The group $S(X)$ endowed with the pointwise topology has a base $\{[A]:A$ is a finite subset of $X\}$, where $[A]=\{g:\in S(X):(\forall x\in A)g(x)=x\}$. Therefore this topological group should be not metrizable (even not a first countable).

At last I allow myself to cite the beginning of a recent paper [BGP] by my scientific consultant Taras Banakh, my scientific adviser Igor Guran, and well known in Ukrainian circles of topological algebra Igor Protasov: "In this paper we answer several problems of Dikran Dikranjan concerning algebraically determined topologies on the group $S(X)$ of permutations of a set $X$.$\dots$ The symmetric group $S(X)$ carries a natural group topology, namely the topology of pointwise convergence $\tau_p$, inherited from the Tychonoff power $X^X$ of the set $X$ endowed with the discrete topology.$\dots$ Answering a question of Ulam [Sco, p.178](cf. [Ula]), Gaughan [Gau] proved that for each set $X$ the topology $\tau_p$ is the coarsest Hausdorff group topology on the symmetric group $G=S(X)$ (cf. [DPS, 1.7.9] and [Dik, 5.2.2])".

References

[BGP] Taras Banakh, Igor Guran, Igor Protasov. Algebraically determined topologies on permutation groups, Topology Appl. 159:9 (2012) 2258-2268

[Dik] D. Dikranjan, Introduction to topological groups, (book in progress).

[DPS] D. Dikranjan, I. Prodanov, L. Stoyanov, Topological groups. Characters, dualities and minimal group topologies, Marcel Dekker, Inc., New York, 1990.

[Eng] Ryszard Engelking. General Topology (Russian version, 1986).

[Gau] E. Gaughan, Group structures of infinite symmetric groups, Proc. Nat. Acad. Sci. U.S.A. 58 (1967), 907-910.

[Sco] D. Mauldin (ed.), The Scottish Book. Mathematics from the Scottish Caf\'e, Birkhauser, Boston, Mass., 1981.

[Ulam] S. Ulam, A Collection of Mathematical Problems, Intersci. Publ., NY, 1960.