You are right about this. But TOR Browser is the system that 99% of people use to connect to the TOR network. Therefore, if the browser were compromised, network access would be difficult for many people.