This Territories talk is actually a good thing, in my opinion. It's much better to experiment early, find the things that are not working, and remove them fast before they begin to fester.
I have seen this in real life community building, a couple bad moves is all it takes to get people to defect. The more important thing is to adjust on the fly, give time and attention to your community and to always be recruiting.
The overall goal with community building is to bring more fruit or bounty to everyone. My goal is to try to do that in the ~builders community. Building community is not hard it just requires a level of sacrifice most people are not accustomed to.