There might even be another solution here that doesn’t require any UI at all: by default, let apps access only the contacts where:
1. both people have each other in their contacts,
2. both have the app installed, and
3. each already agreed to share their OWN info with that app.
Would prevent apps from spamming invites to people who don’t have the app, or getting the info of people who never consented to the app having it. Legitimate communication apps wouldn’t need to ask for contacts anymore.