That sounds like torture. Being unhappy at home is the worst. I'm sure you've considered everything and are making the best decision you possibly can.

One thing I like to do when I'm unhappy where finances are relevant is to put a dollar amount on what I'd pay to be rid of the unhappiness. My experience pales in comparison to yours, but when we were unhappy at our last home I priced how much I'd pay to be rid of the unhappiness, and while we ended up paying 50% more in rent to move and are even more house poor, it was worth it. I'm glad it was an option for us because it was driving me absolutely bonkers.