Yes, I worry about this as well.

As the practice of Community Notes becomes more widespread, it seems likely that it will be infiltrated more and more by highly biased people.

It will simply end up being a microcosm of the very social media platform it's meant to control. People who used to have fights directly in the tweets will now have fights in the community notes system, no consensus will ever be reached on what the truth is, and thus community notes will end up being shown less and less over time, since they depend on consensus.

https://www.youtube.com/watch?v=JwzKDgBjqlU