Have a board of directors who reviews residential applications (HOA's do this). Or, require the piece of property to be sold back to the town.

Stanford University does this with the homes they own. Can only be purchased by people who Stanford approves.