The modern passport was only invented in 1920. Before that, people could, in general, freely move between countries.

Governments have been there forever, but the ratchet of authoritarianism, directly impacting people was very much turned in the 20th century.