I don't use Signal, so I can only repeat what I've heard. That said, my understanding is they could until a recent change and now in most cases they can't...unless by "random people" you include employees of Signal, for whom nothing has changed and they can absolutely see your number and metadata, all neatly packaged in one place with everyone else just a third party doctrine request away for the US government.
...I'll stay #selfhosting Matrix / XMPP.