A common assertion is that conferences shouldn't remove anyone who hasn't been convicted in court. This means that conferences can't remove people who scream over speakers, who drunkenly harass other attendees, who shit in the catering, and so on - this may be disruptive, may even be criminal, but even if you saw it happen they haven't been found guilty and so should be allowed to continue attending.
I don't think people want to attend conferences run by people who sincerely believe this.