I use FDM for this. Appropriate parts of my “.fdm.conf”:

# Where the bad stuff goes.
action "spam" mbox "%h/Mail/spam"

# Some attachments indicate spam just by existing.
$bad_ext = "*.xlsm"
match attachment any-name "*.xlsm" action "spam"

# Run through spamassassin if not too large.
match size < 500K action rewrite "/usr/local/bin/spamc" continue

# If spamassassin thinks it's spam then good enough for me.
match "^X-Spam-Status: Yes" in headers action "spam"

RSA is fine. It isn’t like you will have to worry about the length of the keys for SSH.

Just get it working locally without TLS/ssl. You probably want to set this:

c2s_require_encryption = false

If you are hoping to talk to people outside your domain you will have to figure out what to do about server to server communications.

If the attacker can attack your password hashes at full speed (they get access to the disk) then you would need a fairly hefty password or a random passphrase with a fair number of words in it. That fact can make the sort of protection you seem to be hoping for impractical in many cases.