Solution 1:

Actually, there is an active theory of algebraic topology for "pathological" spaces that has come a long way in the past two decades: Wild (algebraic/geometric) Topology. In fact, this has become a small field in its own right with a lot of recent momentum.

Descriptions of fundamental groups do become more complicated because in a wild space there may be shrinking sequences of non-trivial loops, which allow you to form various kinds of infinite products in $\pi_1$. Hence wild algebraic topology requires more than just the usual tools from algebraic topology but also is deeply connected to linear order theory, continuum theory, descriptive set theory, and topological algebra.

Here is an example of an astonishing result from this field, which addresses your interest in detecting homotopy type:

Homotopy Classification of 1-Dimensional Peano Continua (K. Eda): Two 1-dimensional Peano continua (e.g. Hawaiian earring, Sierpinski carpet/triangle, Menger cuber) are homotopy equivalent if and only if their fundamental groups are isomorphic.

The combined work of Greg Conner and Curtis Kent announced last year proves the same thing is true for planar Peano continua.

Once you realize how complicated these groups are due to the kinds of infinite products that can occur (although a word calculus of sorts does exist), it is absolutely remarkable that such theorems are true...almost scandalous. Results like the one above are very hard to prove. Eda's result required a lot of ingenuity and machinery that is being used and extended in current work.

Here is a little more pre-2000 history:

1950s - 1960s: There were a few scattered papers by some prominent mathematicians, e.g. Barrat/Milnor, H.B. Griffiths, Curtis/Fort.

1970s: Shape theory was developed to extend homotopy theoretic methods to provide invariants for more general spaces. The idea of space theory is to understand objects as (or at least approximated by) inverse limits of the usual "nice" spaces, applying your invariant to the nice approximating spaces, and call the inverse system of algebraic objects a "pro-invariant" and the inverse limit a "shape-invariant." The book Shape Theory by Segal and Mardesic is, I think, the best book on this topic. However, shape invariants only sometimes help with understanding homotopy type and traditional algebraic invariants of wild spaces.

1980s: Not much happened except for Morgan and Morrison fixing H.B. Griffiths description of the fundamental group of the Hawaiian earring.

1990s: Katsuya Eda, whose background was in logic, discovered that the Fundamental group of the Hawaiian earring behaves like a non-abelian version of the famous Specker group $\prod_{\mathbb{N}}\mathbb{Z}$. Eda was the first to make the key connection to order theory and describe the Hawaiian earring group as a group of reduced linear words $w$ (like a free group) where $w$ has countably many letters and each letter of your alphabet can only appear finitely many times in $w$. This work made the Hawaiian earring group practical to use; it is the key to many recent advancements.

Since Eda's work there has been a great deal done and there is now a huge amount of literature on the subject.

Solution 2:

In a sense this is a philosophical question. As you said, algebraic topology usually focusses on "nice spaces" like polyhedra, CW-complexes or manifolds. For one thing, this has historical reasons. For example, homology theory started with simplicial complexes and only over the time turned towards general spaces. On the other hand, if one is interested in effective computations of , say, singular homology groups, one must restrict to spaces making this possible. And again we come to polyhedra, CW-complexes and manifolds.

Many people have thought hard how to shift the border towards more general spaces, and obtained results like Cech (co)homology and Steenrod homology. See for example my answer to https://math.stackexchange.com/q/2807820.

In my opinion the general context of such approaches is shape theory. See the references in my above-mentioned answer and for example

Mardešic, Sibe, and Jack Segal. Shape theory: the inverse system approach. Vol. 26. Elsevier, 1982.

See also https://en.wikipedia.org/wiki/Shape_theory_(mathematics).

The general philosophy to study non-nice spaces $X$ is to approximate them by nice spaces. There are two "dual" methods: To study $X$ via maps from nice spaces to $X$ or to study $X$ via maps from $X$ to nice spaces. The first approach can be denoted as the singular approach; it ends with CW-substitutes for $X$ (roughly speaking, $X$ and a CW-substitute $X'$ admit the "same" maps living on nice spaces). The second approach leads to approximate $X$ by inverse systems of nice spaces which is the essence of shape theory. I do not say that one approach is superior to the other, these are just different points of view. However, I recommend to have a look at the above references to see that shape theory produced quite a number of interesting results and algebraic invariants.

Solution 3:

One reason is that most algebraic invariants are homotopy invariant, meaning the algebraic invariant takes the form of a functor $F:\mathbf{Top} \to \mathcal A$ and that any weak homotopy equivalence $f:X\to Y$ is mapped to an isomorphism $F(f)$ in $\mathcal A$.

But in $\mathbf{Top}$, every space $X$ can be associated with a a CW-complex $X'$ together with a weak equivalence $X' \to X$. (Explicitly $X'$ can be taken to be the geometric realization of the simplicial set $\mathrm{Sing}\,(X)$ of singular simplices.) So by only studying the value of the invariants on CW-complexes, one gets the invariant for any topological space, granted that one can compute such an $X'$ as before for any $X$.

Of course, it just gives a hint on why things are as they are and not on why people did not try go push algebraic topology to non-homotopy invariant functors. Maybe just because it is not consider algebraic topology anymore: this field is supposed to study algebraic objects computed from the general look/shape of a space, and homotopy-deforming is not supposed to change this look/shape. Or maybe we have gone to far into generality when defining topological spaces, and pathological spaces are non-welcomed artefacts: remember that topological spaces is a tentative modelization of the "real" spaces we have around (in physics and other related fields for example); there is no evidence that topological spaces are the right way to go, and some other notions appeared since, like locales (some kind of generalized sober spaces). This last paragraph is only wild guesses on my behalf and it does not necessarily reflect any general opinion.