The latter, yea there's incredibly good deals on off-lease servers on eBay... enterprise grade stuff from Dell or HP with redundant PSU's and solid mainboards

If you buy one of those you can usually have it shipped directly to a datacenter, remote hands into it, and then install something like proxmox

That's called colocation, some datacenters only offer big room sized cages but even in those you can usually find smaller hosting/IT companies that will rent out a single U with power and bandwidth... usually $50-$150/mo depending on options like A+B power or dual 1Gb to 10GE