Two major points on this.
First:
If you block a server of 100 people because you are experiencing harassment from five people they won't moderate, that's really 95 failures, 95% false positives in terms of your blocking when looked at as a whole.
That is the sort of calculus that as an individual you do every day and it is _very_ worth it on an individual level. Especially if you know 0/95 people. Block away! Block together! Share the #blocklists!
But when you are talking about a server?