Webpack, html-loader, file-loader and extract-loader
This is a bug in extract-loader when used with Webpack 5:
https://github.com/peerigon/extract-loader/issues/111
I'm having the same problem and aparently there is no solution until now. If you can, try to use webpack 4.
Or you can look at html-webpack-plugin, maybe it is userful for your case.