A very, very, very big div

Changing to position: fixed seems to speed things up.

Use transform instead of top/left:

container.style.transform = 'translate(' + container.x + 'px, ' + container.y + 'px)';

A live demo at jsFiddle.

1. Answer to first quest "why". One of problems are font size. you have font size 600000px, most browser will see it as too high and render smaller, while chrome tries to render original size. Looks like chrome can not repaint such big letters with your requested styles very fast.

But combining Teemu and geert3 answers - using transform and position:fixed, makes chrome works much more faster even with big fonts.

2. Answer to 2nd question: "Is there a way ... not to try to render the whole page" - you can try to apply mouse action for elements in container, not for whole container.

Maximum font sizes: http://jsfiddle.net/74w7yL0a/

firefox 34 - 2 000 px
chrome 39 - 1 000 000 px
safari 8 - 1 000 000 px
ie 8-11 - 1 431 700 px

In addition to Teemu's answer of using translate:

container.style.transform = 'translate(' + container.x + 'px, ' + container.y + 'px)';

Which you should also use other vendor prefixes, You can simply fix this by using this on the body:

height: 100%;
width: 100%;
position: relative;
overflow: hidden;

and this on html:

height: 100%;

this will, however, disable scrolling. So what I'd do is, add a mousedown event to the body and apply those styles using a css class whenever mousedown is triggered, and removing that class on mouseup.