What exactly did Hermann Weyl mean?
From a Google search, it appears the quote is from Hermann Weyl's Philosophy of Mathematics and Natural Science. I found a copy online here; the relevant passage is on page 90 (search "act of violence"):
The introduction of numbers as coordinates by reference to the particular division scheme of the open one dimensional continuum is an act of violence whose only practical vindication is the special calculatory manageability of the ordinary number continuum with its four basic operations. The topological skeleton determines the connectivity of the manifold in the large.
His meaning appears to turn on the idea of a "division scheme." In a preceding paragraph, he wrote:
In general a coordinate assignment covers only part of a given continuous manifold. The 'coordinate' $(x_1, \ldots, x_n)$ is a symbol consisting of real numbers. The continuum of real numbers can be thought of as created by iterated bipartition. In order to account for the nature of a manifold as a whole, topology had to develop combinatorial schemes of a more general nature.
To paraphrase, I believe he's saying that the topological structure of both the real number line and manifolds more generally is determined by how it breaks into smaller pieces, and how those pieces fit together (in modern terminology, we might look at a space's locale of open sets). To add a coordinate system rooted in arithmetic operations, either to the real line or to a manifold, in his view, is to add too much structure (although it does help with calculations).