Lemmy.fan and its subdomains were partially unavailable on December 6 2023.

Symptoms: Posts were loading but comments were not. Error message was “couldn’t_load_posts”.

Diagnostic steps: Restarted containers with no effect. Check of front logs shows numerous record_not_found errors.

Discovery: DNS records for real.lemmy.fan, the actual instance name, were charged to CNAME records to lemmy.fan to simplify DNS IP changes. This does not work because nginx requires the actual domain name to translate the connection to the port. This DNS mistake confused nginx enough to prevent some things from loading.

Resolution: Changed CNAME records back to A records.

Prevention: Change was documented but not thoroughly tested. Proper resolution is to update the DNS script to update lemmy.fan and all subdomains with the IP address instead of using CNAME records.

Issue has been closed.