You have things set up correctly. Normally, Docker creates a virtual network connection for every container. When you use network_mode: "service:container" you are making those two containers share a single network connection instead. From a network perspective, it is effectively the same as running two pieces of software on one computer, just virtually. All your other software in that stack that is set up that way will piggyback on Gluetun’s network interface instead of creating their own, and they all see each other as being on the same localhost as if you ran multiple programs directly on your computer.

You also opened ports correctly. Since everything is using Gluetun’s network connection, you have to open ports on Gluetun. Opening ports in Docker carries no risk of leaking your connection or breaking your VPN, as long as you don’t forward those ports from your router (which you said you can’t do anyways). Ports are to let traffic in, not out. By opening it, you are telling your computer to “listen” for anybody trying to talk to it using that port; otherwise, it would just ignore anybody who tried.

Gluetun by default does something VPN software calls “split tunneling”, which is that traffic goes out to the VPN only if it’s externally bound traffic. Any traffic bound for an address in the private network spaces (192.168.x.x, 172.16.x.x, 10.x.x.x) will not use the VPN. This is done by design, specifically so you can do what you just did. This way you can access stuff that’s intended to be accessed locally, but still forward all external traffic to the VPN.

The only thing you don’t really need is the depends_on because network_mode: "service:container" already acts like depends_on, so it’s redundant, but it’s not hurting anything to explicitly call it out either.

As /u/SpacezCowboy said, for peace of mind you can test your torrent client with https://ipleak.net/. You can also test a regular connection from inside the container by running docker exec -t qbittorrent curl icanhazip.com

There’s nothing different. Just expose the ports using docker, and then point nginx at it like any other application. If it’s running on the same server, you can just point it at localhost.

For nginx proxy manager, you can define custom configs easily, but for most usage you should be able to set up each redirect with a couple of clicks in the GUI. You should set it up to generate its own certs using the built-in tool though, but if you really want you can import certs manually through the GUI as well.