Is a connected sum of manifolds uniquely defined?

It is a standard excercise in differential geometry to prove that a connected sum $M\#N$ of two smooth manifolds $M,N$ of the same dimension is uniquely defined (under some assumptions regarding orientation). In the proof one needs to show that $M\#N$ does not depend on:

the choice of balls (chart maps) around fixed points $(m,n)\in M\times N$ (assuming that maps in comparison induce the same local orientation near $m$ or $n$),
the choice of points.

I can prove the first for smooth manifolds (for two neighbourhoods of $m\in M$, I choose such a small neighbourhood that it looks like a ball in both charts, and I observe that the claim is trivial for balls). The second follows from homogeneity of both: topological and smooth manifolds.

My question is: Prove or disprove the first point for topological manifolds. If necessary, one can assume orientability, but the smooth case gives hope that the claim may be true also for non-orientable manifolds.

Technical clarifications: For the sake of this question "the" definition of $M\#N$ is as follows. Take $(m,n)\in M\times N$ and chart maps establishing homeo/diffeo of neighbourhoods of $m,n$ with open balls. Then canonically identify the punctured balls with cylinders and glue the cylinders reversing the vertical coordinate: $$B\setminus \{m\} \simeq S^{n-1}\times (0,1)\owns(s,t) \mapsto (s,1-t) \in S^{n-1}\times (0,1) \simeq B'\setminus \{n\}.$$ To avoid problems we can assume that the chart maps can be extended to bigger balls.

P.S. ~~Some related doubts~~ Update: My earlier doubts resolved thanks to discussion in the comments:

why wikipedia needs (note that the definition of connected sum in wiki is slightly different than mine):
- oriented manifolds (in both cases if I understand correctly). Resolved: Thanks to George Lowther's comment I already know that the connected sum depends on the orientation of the spheres/cylinders glued (I missed that in my proof previously).
- and why it mentions some "canonical glueing" as necessary for the uniqueness (in the smooth case)? Resolved: as studiosus states in the comments, the thing is that wikipedia allows ugly diffeomorphisms of spheres that can not be extended to the whole disk, which is not the case in this question,
wikipedia claims the answer is affirmative "crucially" by the disc theorem, however the article cited handles the differential case. Resolved: The disc theorem is false for the topological case, because a sum of Alexander horned sphere and the bounded component of its complement is homeo with the disk, while the unbounded component is not homeo with complement of the unit disk in $\mathbb R^3$.

I'll write a self-answer to summarise findings from the comments (mainly by George Lowther).

An assumption previously missing in the question is the one regarding orientation. For non-orientable manifolds it is not important (due to that question or a characterization of non-orientable manifolds as containing $n$-dimensional equivalent of the Möbius strip), but if both manifolds are orientable and do not admit an orientation-inversing automorphism, then depending on using chart maps consistent or inconsistent with the orientation we can obtain two different topological spaces.

The claim is true due to the annulus theorem (rather nontrivial) - which enables a proof similar as in the smooth case. The theorem highly relies on the innocent assumption: