Are continuous maps "weaker" than other morphisms?

It is patently false that

In every other morphism type, the marking quality of the morphism is guaranteed for the inverse

This may be true for the ones you know, but it is far from being the norm.

For starters, the notion of "bijection" may not even make sense in the category you're working in. There are plenty of examples.

Look at the category of spans, where objects are sets, but a morphism from $A$ to $B$ is a pair of functions $A \gets C \to B$. What does it mean for such a morphism to be bijective? You could say "both maps are bijective", but that's already a stretch.

Look at the fundamental groupoid of a space $X$. Objects are points of $X$, and morphisms are homotopy classes of paths. What does it mean for a homotopy class of paths to be "bijective"? I have no idea.

Look at the Fukaya category of a symplectic manifold $M$ (see this answer). Objects are Lagrangian submanifolds of $M$, and morphisms are intersection points of Lagrangian. What does it mean for an intersection point to be "bijective"?!

Then there is the issue that even if you can make sense of "bijective", it may be false that "bijective homomorphism = isomorphism". You already have the example of topological spaces and continuous maps. Here are others.

Consider normed vector spaces. Take $L^1 = (C^\infty([0,1]), \|-\|_1)$ and $L^2 = (C^\infty([0,1]), \|-\|_2)$. Thanks to the Cauchy–Schwarz inequality, the identity map $L^2 \to L^1$ is a morphism of normed spaces. The identity is eminently bijective, but $L^1$ is not isomorphic to $L^2$, because there is no inverse morphism.

But you might say, "normed spaces aren't very algebraic". Okay. Here is an example that looks algebraic (some people might disagree). Consider the category of partially ordered sets, where morphisms are order-preserving maps. There isn't much topology here. Let $X = \{x,y,z\}$ with $x<y$ and $x<z$ (but you cannot compare $y$ and $z$), and $A = \{a,b,c\}$ with $a<b<c$. Then there is an order-preserving bijection $f : X \to A$, given by $f(x) = a$, $f(y) = b$, and $f(z) = c$. But $f$ is not an isomorphism, because there is no inverse morphism.

And for good measure, here is a counterexample to the "converse" (found in the Wikipedia article). Consider the category whose objects are CW-complexes (a special kind of topological spaces), and morphisms are homotopy classes of continuous maps. Then it's possible for two CW complexes to be isomorphic in this category without even being in bijection as sets! For example, $\{0\}$ and $[0,1]$ are isomorphic, because they're both contractible, but there is no bijection between $\{0\}$ and $[0,1]$.

If you want an algebraic example of this, consider the derived category of chain complexes of abelian groups. Then $0$ is not in bijection (in each degree) with the chain complex $\dots \to \mathbb{Z} \xrightarrow{=} \mathbb{Z} \xrightarrow{0} \mathbb{Z} \xrightarrow{=} \mathbb{Z} \to 0$, but both are isomorphic in the derived category.

Derek Elkins mentioned in the comments that if your category is equipped with a "forgetful" functor $U : \mathcal{C} \to \mathsf{Set}$ (hence you can make sense of "bijections") which is monadic, then $U$ reflects isomorphisms, i.e. if $U(f)$ is a bijection then $f$ is an isomorphism. But now you see that this is a very special property: your functor $U$ has to satisfy the special condition of "being monadic". This isn't the case for all functors, far from it. Moreover a reasonable functor $U$ may not even exist...

So to conclude, "bijective morphism = isomorphism" is the exception, not the norm. This is very important to remember. To answer your question, "continuous maps" are not "weaker", they are "as usual", and morphisms in algebraic settings are actually nicer than usual. But even something that "looks" algebraic may not satisfy this condition – see the example of posets above.

Yes, it is the case that algebraic categories have this property while topological or geometric ones may not. An elementary explanation is that a function preserves some operation if and only if its inverse does; so any category whose morphisms are defined by preserving certain functional operations will have this property. This ranges from the examples of groups and the like you suggest all the way up to compact Hausdorff spaces, which can be defined in terms of convergence operators for ultrafilters on their underlying sets. What the failure of this property shows, then, for spaces, is that it's impossible to look at a topology on a set in terms of algebraic operations. This failure is common in structures defined via relations. For instance, it fails for partially ordered sets. This can often be repaired by changing the forgetful functor-for instance, by viewing a partially ordered set as a category.