Some people are on servers where they are the only member, yet they have large followings.
It is possible to do, as long as you are communicating with people, following them, taking part in threads etc.
It can be trickier to get started if you're on a really small server, but the directories (https://fedi.directory, https://communitywiki.org/trunk, https://fediverse.info/explore/people) can help get you started.
By the way, Fosstodon is quite a large server. It's as connected as servers get.