The key that is stored server-side by Signal are only used to decrypt your profile, your contacts and groups, and your app settings. It is not used to decrypt your messages. And my understanding is that if you set a secure password instead of a pin, the key will be encrypted by your password before being uploaded, anyway, meaning that it’s e2ee, too.

Also, you can host your own Signal server, though I suspect doing so is more complicated than hosting a Matrix server. The code is almost fully open source (and I only say “almost” because, in the past it was not uncommon for the code on Github to be several months out of date - the license is a FOSS license). However, Signal isn’t federated, so you wouldn’t be able to talk to anyone using the Signal app or a fork on the main Signal server - unless you forked the app and made it able to manage accounts on multiple different servers.

Matrix also doesn’t encrypt metadata and it syncs conversation metadata to every involved server. As recently as 2022 Matrix had several critical vulnerabilities discovered (and patched). I wasn’t able to find any record of the audits mentioned in that article, so I have no clue how they performed, but regardless, even if just based on the metadata alone, currently Signal is more secure.

From a FOSS perspective, it makes sense to prefer Matrix over Signal (or maybe XMPP?). Signal - Moxie specifically - has been downright hostile to forks (refusing to allow them to use the Signal server with their frontend fork) and I remember him rejecting PRs and being rude toward contributors, too, though my memory’s a bit fuzzy on the specifics. That was a few years ago, so maybe it’s gotten better, but even if so, Signal isn’t federated and likely never will be, so any developer would have a lot more flexibility building things for Matrix or contributing to existing Matrix projects.