How do Fractal of the Mist monsters scale?

The number of enemies does not scale that dramatically. You can read about the differences on the wiki, but the main difference in difficulty is that the enemies become more powerful. This is done by slightly increasing their levels or numbers, adding new attacks at higher difficulties, or buffing certain attributes (such as the movement speed of the veteran grawl in the volcanic fractal).

The increase in number of enemies that you observed seems abnormally high; the increased numbers are usually subtle. That observed difference may be related to those dredge respawning indefinitely. To answer your question: yes they will keep increasing steadily, but not nearly as quickly as you've described.

Note that only bosses inflict agony, so additional agony and increased enemy numbers will rarely be coinciding problems.