It's the problem with free speech generally: it is a binary. Either you accept all of it or you go down the path to not having it.
@p makes a convincing argument along these lines. Free speech is there for the stuff that people would suppress, even if I think a lot of it is insane and stupid.