Why can't I replace std::map with std::unordered_map

I guess that because std::unordered_map needs to rehash, and therefore copy elements, the types need to be complete, whereas a map, only ever working with pointers to elements, will not exhibit that problem.

The solution here is to have an unordered map to a pointer:

std::unordered_map<uint32_t, std::shared_ptr<Test> >. 

Using both map and unordered_map with incomplete types involves undefined-behavior:

In particular, the effects are undefined in the following cases:

[...]

— if an incomplete type (3.9) is used as a template argument when instantiating a template component, unless specifically allowed for that component.