You don’t have to expose Nginx publicly. It can exist privately on your network. I have my own domain and DNS server internally. For example nginx.home.datallboy.com and jellyfin.home.datallboy.com will resolve to NPM server at 192.168.1.10. Then nginx can listen for jellyfin.home.datallboy.com, and proxy those connections to my Jellyfin VM at 192.168.1.20.
Since I own my domain (datallboy.com), I let Nginx Proxy Manager do DNS challenge which is only used to authenticate that I own the domain. This will insert a TXT record on public DNS records for verification, and it can be removed afterwards. LetsEncrypt will then issue a certificate for https://jellyfin.home.datallboy.com which I can only access locally on my network since it only resolves to private IP addresses. The only thing “exposed” is that LetsEncrypt issued a certificate to your domain, which isn’t accessible to the internet anyways.
You do not have to create your own CA server.
In the same boat. Barrier is the only software holding me back from switching to Wayland.
EDIT: Did come across https://git.sr.ht/~nickbp/nikau while searching. Looks fairly new. I’m looking for something that supports a Windows server though.