The difference is when your ex releases your key and you are using this system then everybody can read your email instead of just the NSA and the recipient.
So, what you are saying is that nobody else, but you, should ever have access to your private key, right? I'm pretty sure that's PKI 101, which I think was Karunamon's point; and, I'm pretty sure the solution to your proposed 'weakness' in this system is not technical.