The idea behind a hardware wallet is that you're assured that there is no way for the device to be compromised. It can do one thing and one thing only: sign transactions. It cannot broadcast transactions, it cannot do anything but receive unsigned transactions, sign them and send the signed ones back. It cannot connect to any network.

This is why hardware like ledger with their online backup recovery aren't good. This is why these newer devices that connect over Bluetooth are no good. You want something that has a single channel, either wired or using QR codes and a camera, that way physical access is required and the attack surface is low.

In light of this, a phone has just too high of an attack surface. It has a mobile radio, it has bluetooth, it has WiFi, and you may not know this but even without a sik card the radio connects to towers. It probably doesn't send any data, but the part of the device that connects to these networks is a separate environment and can be backdoored by a sophisticated adversary and pull information from the memory without you knowing. That's the "government is after me" vulnerability, but beyond that, the simple fact that it can connect to networks and run other applications makes it unsuitable.

If you need to do this, make sure you do it with only a small to medium stash. Think of it less like a savings account and more like a checking account for withdrawing cash from an ATM. Your actual savings, the chunk you don't withdraw from except in emergencies or like once a year or something, should not be stored this way.