In the end it’s your stream and you can set whatever rules you want (and need, to protect yourself from spambots etc.).
But just between you and me? Kicking people from stream for lurking (i.e. watching the stream without chatting) is not something that will endear you to me.