The thing is, apathy puts you on one side by default. Ignoring injustice isn't what makes you insufferable (that just makes you a bad person) what makes you insufferable is having the gall to pretending that ignoring injustice makes you enlightened while also complaining about others making too much public noise.
No one buys the psuedo-centrist schtick, they just correctly percieve you as a particularly lazy right-winger.