There is no contradiction between 1) early Zionists being refugees who fled persecution, violence, & antisemitism, and 2) Zionism being a settler-colonial project.
What determines whether it's colonialism isn't what pushed settlers to migrate; it's their relationship to the land & to the indigenous population.