Just note that "your metadata" actually extends to less things than any XMPP and Matrix homeserver can see.
Signal knows your number, when you last connected, whether your account should be discoverable by phone # , how many devices you have linked, when you created your account, and whether anyone can use sealed sender or not.
If for some reason a message doesn't arrive, they store the encrypted data until it can be sent.
Meanwhile your average matrix homeserver has a full copy of chat history and more (including any other homeserver in a room) not including encrypted contents but including:
- room participants
- timestamps for messages and other events
- unencrypted events (join, leave, invite, settings changes, avatar change)
- your avatar, name, MXID/username
- associated E-Mails
- SSO Origin (e.g what you used to login, if not just username+password)
- Media (encrypted, but kept indefinitely due to design oversight)
- Reactions (unencrypted due to design oversight)
- Any and all account settings (notifications, preferences and more)
- Full list of devices including reported names
- all rooms you're in
- all spaces you're in
- any custom settings your client adds to your account data
For XMPP, its much of the same.