It was never supposed to be confidential. That need arose as a direct result of using it as an ID. If the SSA was the only organization using the number, (as originally intended,) then it wouldn’t need to be kept confidential.

But when the SSA gave every single person a unique number, other organizations went “hmm this sure would be convenient for differentiating individuals with similar names and DOBs.” So other organizations started using it for identification, and suddenly you needed to keep the number secret because anyone with your number could ID themselves as you.