I think this is basically right. The optimal policy would most closely resemble what would exist in a free society that respects private property rights.

It's illegitimate for the state to prevent me from inviting someone to visit or to work for me, but an uninvited person has no abstract right to enter other people's property.