Experimental Fractional scaling makes fonts dirty in Ubuntu 17.10
Looks like it's a work in progress. GNOME only supports integer scaling properly (source).
Quote:
Currently, we only allow to scale windows by integral factors (typically 2). This proves somewhat limiting as there are many systems that are just in between the dpi ranges that are good for scale factor 2, or unscaled.