Apparently there's a cool Pango flag that enables subpixel glyph positioning which makes things scale smoother and improves kerning! Thanks Benjamin (the GTK maintainer) for the suggestion