You don't have to go to another server, all you have to do is log in on your own server and you will see people from other servers in your timeline.
Think of it like email:
You log in on your email provider's site, but emails you receive are from people on other providers. You don't need to be on the same provider to exchange emails.
This works because all the providers are federated together into a single global email network.
The Fediverse is this, but for social media.