No, its okay. If you do use the hardware wallets, you have to be careful of many things. But people using them generally dont download malicious hardware. I would think they would be more secure than that.