Product of spaces is a manifold with boundary. What can be said about the spaces themselves?

Suppose I have two topological spaces $X,Y$ and I know that $X\times Y$ is homeomorphic to a manifold with boundary. Can I conclude that $X$ and $Y$ are manifolds (maybe with boundary)?

If not, suppose that $Y=[0,1]$. Is it then true? My intuition states that this is true, but I cannot see directly an elementary proof.

Bing's dogbone space is a non-manifold $X$ such that $X\times\mathbb R\cong\mathbb R^4$. It is constructed as a quotient of $\mathbb R^3$ which is the identity outside of a ball. Hence we can do the construction inside a ball, to get a modified dogbone space $W$ with $\partial W=S^2$. Then I think that $W\times\mathbb R\cong D^3\times \mathbb R,$ which has boundary $S^2\times\mathbb R$. The basic idea as I understand it is that the nested tangle of genus 2 handlebodies unknots itself in $4$ dimensions, and this doesn't appear to use anything outside of a ball. However I haven't ever gone through the proof in detail, so I might be missing something.

If you take $Y=[0,1]$ I would guess that $X$ is probably a manifold with boundary. I recall hearing in a lecture many years ago that if $X\times S^1$ is a manifold, then so is $X$, which strikes me as a very similar problem.