The federation should be able to publish a "proof of reserves" at some fixed time interval akin to exchanges. A merkle tree containing all the tokens might be very large, not sure if there's some private way of grouping them.. And then they would also publish signed UTXOs adding up to the same amount. Maybe have some "proof of funds" procedure like Hoseki if that calms people down. UTXOs could be moved/consolidated at the same interval to ensure that they aren't being "lent out" to someone else.

People's wallets could then verify their tokens exist in the merkle tree and that the UTXOs are correctly signed.

But yes, the guardians could go for a rug eventually. This just makes it harder to do up until that point of corruption.