"If the users I like are all here, and I can see other servers, what would be the point of ever switching to a different server? "
Firstly, you don't have to be on the same server as users you like. I'm on a different server, yet you are talking to me right now.
Second and most important, there are many important reasons why this network needs to be spread out onto as many servers as possible:
https://fedi.tips/why-is-the-fediverse-on-so-many-separate-servers/