Cosine similarity / distance and triangle equation
Solution 1:
Neither of these is a metric on $\mathbb R^n$ for several reasons, some of which you've pointed out. In a sense, the main reason, of which the undefined result for the zero vector is a symptom, is that this value depends only on the direction of the vectors and not on their length (and the zero vector has no direction).
If you want to consider these functions as metrics, you need to consider them on the set of directions (i.e. rays from the origin) in $\mathbb R^n$, or equivalently on the unit sphere $S^{n-1}$ in $\mathbb R^n$.
The second function, the "arccos distance", is just the angle between the two directions/vectors, and this is a metric because it's the geodesic distance on the unit sphere.
The first function, the "cosine distance", isn't a metric because for small angles it approximately calculates (half) the square of the angle, $1-\cos \alpha\approx\frac12\alpha^2$, and if you turn by the same angle $\alpha$ twice, the sum of the two individual "distances" will be approximately $\frac12\alpha^2+\frac12\alpha^2=\alpha^2$ whereas the "distance" between the directions $2\alpha$ apart will be approximately $\frac12(2\alpha)^2=2\alpha^2$.
To answer your questions:
I don't know a source, but I hope the above arguments should be immediate enough to convince without a source.
No, since you'd still get a distance of $0$ for any vectors in the same direction, so this could be at most a pseudometric.
Never heard of those.
There is none; the function will necessarily be discontinuous if you extend it to $0$, and the value at $0$ will necessarily be arbitrary because there are vectors in all directions arbitrarily close to $0$.
Solution 2:
A and B are true, but C is not true, because any parallel vectors have length 0. Arccos-dist is just projection onto the sphere followed by the regular metric on the sphere (you can see this because the length of the vector doesn't matter, so you can assume each vector lies in the sphere); so yes, injective maps into the sphere become metric spaces this way, but Euclidean space minus the origin will not.
See this link under "vector version", where we see arccos dist is the spherical metric, and it only depends on the direction.