"But they might sue us if we kick them out!" for what? Just say that they're not welcome. There's no need to say why. You're not obliged to say anything that they could argue was defamatory. Refund them if you want to avoid an argument about that.
If people could easily win lawsuits over being kicked out of an event, every club on the planet that had barred someone for being an asshole would be out of business.