Difficulty adjustment happens every 2016 blocks. So if they manage to solve blocks with QC they could snatch all those 2016 blocks before it kicks in. And then difficulty would be so high, that ASICS would never be able to find anything again and the QC would be the only one that finds blocks every 10 minutes.

That would be a hardcore 51% (in this case 100%) attack.