I mean, you have to have a legal or ethical ground in order to end the relationship during litigation, and short of getting the client's consent via a signed substitution of counsel, the court must approve the termination on a noticed motion to withdraw (if you're too close to trial, you can't do either). In non-litigation representation, you can pretty much end it for any reason provided you give them sufficient time to find new counsel.

I tend to be fairly good at sizing up clients from the start, so there's very little late termination. When I have terminated further into the representation in the past, and other than for simple non-payment, the client had usually lied about or withheld some matter that was material, or otherwise failed to cooperate in the preparation of their case.

A client withholding material information or lying about a material matter is an issue not only because the late information drastically changes my opinion of the client and the matter, hence its materiality, but also because the client's dishonesty cannot be undone. I can't trust them. And if I can't trust them, I can't fulfill my obligations. I'm one of the guys that has to believe your shit, or I can't bring my A game. There's a metaphysic / teleology at work there.

In such situations, it often helps to explain to the client how their behavior irreparably damaged the relationship, and to further explain that if there is to be termination, it's best that it be voluntary, because termination via motion may send the signal that there's something wrong with the client or the case or the ability of the client to afford to continue the litigation (on such a motion, you can't state the reason for the termination, just that it's a legal one. If you lie, then the bar could yank a license if the client complains and you're found to have improperly withdrawn and to have made misrepresentations to the court. The difference is that the client waives attorney client privilege upon complaining to the bar.)