Of course, he both tried to be nice to me and bitched to me about the fact I left him in that same weird pattern of be nice then be mean, but I was firm. No meant no.
I was fine for a few months, by which point I had the surprise of getting another job. Someone else had seen what I could do in the buildroot irc and contacted me to see if I was interested.
It turns out that I was. I still hadn't healed from the trauma of that first job experience, but I wanted to give this person a try. He was nice, to the point he even had his team of developers switching the entire UI of the administrative backend website to an actually accessible one just for me. More than anything, it showed this person was taking me very seriously.