as long as you have to give them the address I don't see any practical difference between a single address and a different one every time. Swan knows you. If you wanted to obscure your identity perhaps that address they send to could be a coin mix of some sort, which then allows you to spend those withdrawals from a coin-join.