Ok, so here's what's on my mind: it used to be, not so long ago, that we (leftists) would say things like "If you've got a table with one Nazi and ten people sitting with them, that's a table with eleven Nazis." Similar things were said about antisemites and other bigots more generally. This was used to condemn a lot of non-leftists, especially right and center right folks, who weren't themselves Nazis, but who allowed them at their rallies and such.