No, its okay.
If you do use the hardware wallets, you have to be careful of many things.
But people using them generally dont download malicious hardware.
I would think they would be more secure than that.