Are there two different ways to generalize the orthogonal group?
Question: Is there more than one "appropriate" way to generalize the orthogonal group of $\mathbb{R}^n$?
The two competing ways which come to mind for me are:
The group of all isometries with the origin as a fixed point, for any (finite-dimensional) vector space $V$ with any metric $d$ (not necessarily inducing the vector space topology).
The group of all vector space automorphisms which preserve a given symmetric non-degenerate bilinear form (not necessarily positive-definite).
Are both of these generalizations in use? Do they have different names?
(Unnecessary) Background: There are at least two equivalent ways to define the orthogonal group for $\mathbb{R}^n$: the group of all isometries of $\mathbb{R}^n$ which have the origin as a fixed point (1)(Theorems 19 and 20, pp.137-138, Friedrich, Agricola, Elementary Geometry), or the group of vector space automorphisms of $\mathbb{R}^n$ which preserve the inner product (1)(2)(3).
That these are equivalent is a result of the special properties of the (standard) metric on $\mathbb{R}^n$: not only is it normable, but said norm even satisfies the parallelogram identity and thus is induced by an inner product.
What confuses me is that these two characterizations seem to provide very different directions of generalization.
We can always define a group of isometries for any metric space, irrespective of whether the metric is induced by a norm, much less of whether it is induced by a norm satisfying the parallelogram identity. Even the discrete metric has a well-defined isometry group. In practice it probably makes sense to restrict to those metrics inducing the vector space topology, however.
On the other hand, the other definition of orthogonal group generalizes to vector spaces equipped with any symmetric non-degenerate bilinear form (1)(2), even those which are not positive-definite, and thus don't induce a norm nor induce a metric (I think).
For any $n$-by-$n$ matrix $M$, we can define an inner product on $\mathbb R^n$ given by $$\langle u,v\rangle_{\!_M} := u^{\top}Mv$$ Inner products define "distances" and "angles" by \begin{eqnarray*} \|u\|^2 &=& \langle u,u\rangle_{\!_M} \\ \\ \cos\theta &=& \frac{\langle u,v\rangle_{\!_M}}{\|u\|\, \|v\|} \end{eqnarray*} A linear transformation $T : \mathbb R^n \to \mathbb R^n$ preserves distances and angles if, and only if, $$\langle u,v\rangle_{\!_M} = \langle T u,T v\rangle_{\!_M}$$ for all $u,v \in \mathbb R^n$.
\begin{eqnarray*} \langle u,v\rangle_{\!_M} &=& \langle T u,T v\rangle_{\!_M} \\ \\ u^{\top}Mv &=& \left(Tu\right)^{\top}M\left(Tv\right) \\ \\ &=& u^{\top}\!\left(T^{\top} M T\right) v \end{eqnarray*} We need the set of all $T$ for which $M = T^{\top}MT$.
In the case of the standard Euclidean "dot product", $M=I$ and so we need $T^{\top}T=I$, i.e. the orthogonal matrices.
EDIT: These questions (and their answers) seem strongly related. I could probably piece together an answer from them now if I were less lazy, but for now I will include the links for future reference:
When does the group of isometries of a norm determine the norm?
Isometry group of a norm is always contained in some Isometry group of an inner product?
Characterisation of inner products preserved by an automorphism
Orthogonal Matrices and Symplectic Matrices and Preservation of Forms
Of particular importance is the fact that the isometry group of the metric induced by an inner product determines the inner product up to a scalar multiple. /EDIT
This doesn't actually answer the question (hence the community wiki) but is a collection of some relevant thoughts I have had based on Spivak's Comprehensive Introduction to Differential Geometry Volume 2. Maybe it will help someone else to think of the answer. None of it is necessary to understand the question though, so I thought it might be prudent to separate from the question.
Motivation: Previously, I had been trying to identify a "geometric" criterion for a norm to be induced by an inner product, or for a metric to have a "sensible" notion of angle, and whether this was equivalent to being induced by an inner product. For the simple case of $\mathbb{R}^n$, where the only inner product up to scalings and rotations is the standard one, it seemed to me intuitively that, for norm-induced metrics, what allows one to distinguish those that are induced by inner products is the shape of their unit spheres. Namely, the unit sphere of any norm on $\mathbb{R}^n$ which is induced by an inner product is (I think) a sphere [update: according to Spivak, it is an ellipsoid, not necessarily a perfect sphere]. Spheres are also (the only?) shapes which are invariant under rotations and reflections, i.e. the orthogonal group. Obviously if we must define the orthogonal group in terms of an inner product then this is somewhat circular, but if the orthogonal group has a definition in terms of more elementary properties of the vector space which are independent of any choice of inner product then it might work.
What also came to mind is the definition of angle (in radians) from pre-calc: the ratio of the length of the arc which subtends the angle to the circumference of a subtending circle. In order for this to be well-defined, we need it to be invariant under rotations and reflections, which only occurs if the "generalized circle" (i.e. unit sphere) is invariant under rotations and reflections. Thus it seemed that perhaps a well-defined notion of angle could be reduced to a choice of metric or of norm (I'm not sure) whose unit spheres are invariant under the orthogonal group. Thus it became somewhat important to me to try to understand whether the orthogonal group could only be understood in terms of a choice of an inner product or only in terms of a certain subgroup of isometries.
I believe that this question may be asking the same thing: Is a faithful representation of the orthogonal group on a vector space equivalent to a choice of inner product? but as mentioned before I do not understand it well enough to say for certain.
I also think that the following theorem from p. 208, Volume 2 of Spivak's Comprehensive Introduction to Differential Geometry may be relevant:
Theorem. Let $F: V \to \mathbb{R}$ be a continuous Minkowski metric on an $n$-dimensional vector space $V$. Suppose that for all $p$ and $q$ in the unit sphere $\{v \in V: F(v)=1 \}$, there is a linear transformation $\phi: V \to V$ such that $\phi(p)=q$ and $F(\phi(v))=F(v)$ for all $v \in V$. Then $F$ is the norm determined by some positive definite inner product.
The definition of Minkowski metric used by Spivak does not appear to be standard (in particular, the term is usually reserved for the symmetric bilinear form of relativity). It describes something similar to a seminorm, in that it is a relaxation of a norm, but a Minkowski metric is required to separate points (a seminorm is not) but is not required to satisfy the triangle inequality (a seminorm) is. From p.200 of Volume 2 of Spivak:
A Minkowski metric on a vector space $V$ is a function $F: V \to \mathbb{R}$ such that $F(v)>0$ for all $v \not=0$ and $F(\lambda v)=|\lambda|F(v).$
Clearly $F$ is completely determined by a "unit sphere" $\{v: F(v)=1 \}$; the unit sphere is symmetric with respect to $0 \in V$ [i.e. invariant under reflections because of the absolute homogeneity of $F$], and intersects every ray through $0$ exactly once. Moreover, any such set is clearly the unit sphere for some $F$.
Spivak also goes on to explain that the absolute homogeneity of such a function $F$ allows one to define a notion of length for curves in $V$ which is independent of parametrization.
On p. 203 Spivak shows how the convexity of the unit ball of such an $F$ is equivalent to $F$ satisfying the triangle inequality, and thus being a norm. On pp. 204-205 he goes into the geometric significance of this further:
A Minkowski metric $F$ can be used to define a "distance" function on $V \times V$, by $(v,w) \mapsto F(v,w)$ (however, it is easy to see that this distance function satisfies the triangle inequality, and is consequently a metric, if and only if $F$ is a Banach space norm on $V$). This is just the procedure by which, in analytic geometry, we define the distance between two points in $\mathbb{R}^n$; in this case, we choose $F(x)=(\Sigma(x^i)^2)^{1/2}$, motivated, of course, by the Pythagorean theorem. After the Pythagorean theorem has been incorporated into our definition of distance in this way, it is interesting to ask what content, if any, remains to this theorem. The answer is, that the Pythagorean theorem has been declared true only for right triangles with sides parallel to the axes, but remains true for all right triangles. This is because, when $F(x)=(\Sigma(x^i)^2)^{1/2}$, the isometries of $\mathbb{R}^n$ are transitive on the unit sphere. That is to say, if $p$ and $q$ are in the unit sphere, then there is a linear transformation $\phi: \mathbb{R}^n \to \mathbb{R}^n$ with $\phi(p)=q$ and $F(\phi(x))=F(x)$ for all $x$.
This same transitivity property holds for any $F$ which is the norm $|| \cdot ||$ associated to a positive-definite inner product $\langle , \rangle$ on an $n$-dimensional vector space $V$, since there is an isomorphism $f: \mathbb{R}^n \to V$ such that $f^* \langle , \rangle$ is the usual inner product on $\mathbb{R}^n$ (Theorem I.9-3). It turns out that this property actually characterizes the Minkowski metrics $F$ which arise from inner products.