it's more important that your user get to say mean shit to randos than randos not semi regularly see mean shit? in my dream world, everyone accepts that Fred is on your server talking his shit as long as he refrains from bothering people. or he can bother people and most people will block your server.
the fedi is like a public square, but also not like one. it took many years to figure these rules out in IRL. fedi norms are still evolving imo.