I think you are confusing the concept of "free speech" with the concept of censorship. It doesn't have to be a government that engages in censorship for it to be called censorship.

Yes, an admin of a Mastodon server has the right to do that. So they have not violated my right to free speech. But they have engaged in censorship, and when that occurs I begin to take steps to discontinue my support.