SF writing great Joe Haldeman had novels that deviated from expected plots a lot after about 2000. I recall a fragment of an interview, roughly:
I started feeling sorry for my own characters; I was frog-marching the poor bastards through this sequence of setting, rising action, conflict, climax and denouement as I put them through hell.
I had to let them break out of the forced march and be free to not be an amusement.