Lists of unfollowed users (or muted users) is a popular request:
https://github.com/mastodon/mastodon/issues/6982
I think the original concern was about trolls using lists to share target, which happened on Twitter, where people were being added to lists without their knowledge. This was why the following requirement was brought in, to give people control over who can list them.
The solution suggested is to still require following, but you would be able to mute them on your Home timeline.