Yes, correct. Small miners would need to rely on 1 or a handful of entities that were specialists. This would be a very strong centralization effect on bitcoin and would effectively create permissioned bitcoin mining as you'd need to register with one of these entities in order to be competitive as a miner.

Then, these handful of entities could effectively dictate the rules of bitcoin in the future and censorship-resistance would be lost.

Hard to see how bitcoin would have much value if only a few entities had this much power.