If you're talking about momentarily passing by them and going on your way, I'm sure it's fine. I'm talking about directly interacting with them like loitering in an alley in the middle of the night for a half hour where they consider home, or being locked in a cage with them for days, or having to physically remove them from an area they're unwilling to leave.