Re-using pubkeys is the chief problem -- a problem which Bitcoin devs figured out more than a decade ago. Using either Signal's or Matrix's cryptography library would get you 80% of the way to not leaking meta-data. The remaining 20% consists of (a) using random relays; and (b) having some way for people to contact you without requiring the recipient to either to post a pubkey publicly or host an online method just to generate addresses.

I am tempted to quit my job to work on this, but I have no idea if there's enough money on the table to fund the work.