So on the next frame of simulation, his toe is still in the way of the door. The door isn't allowed to just pass through his toe, so it does the only other option - it bounces back. I think by default it's set to do so completely elastically, so the door bounces back with exactly the speed it came in at, slams shut, and locks again. And you're stuck.