This is where you get problems like "wizards can secretly locate and then spy on their enemies and wait until they're asleep, then walk through the walls and kill them". Or "wizards can secretly watch over any person's shoulder to steal their passwords".
Things that an ordinary person simply cannot do anything about. If ordinary people are meant to be a credible opposition, that's a problem.