You're right that SIgnal has less metadata than a Matrix / XMPP server. To me it's still a greater risk because it's centralized in one place, making it a far more attractive target to threat actors.
Also, they can send SIgnal a third party doctrine request to give them all of my data without me knowing. If they want to try that with my Matrix / XMPP server, they need to send that same request directly to me and they're getting no one else's data with it.
Decentralization matters.