Since every guardian (aka server aka node) in a fedimint needs to communicate and come to consensus over the network, you largely run into a latency constraint around consensus. Especially if the network latency of those machines amongst each other is variable or high. Eric Sirion (fedimint creator), supposedly did some upper limit testing and found that consensus starts to get unreliable/weird/etc around ~15 guardians, so I'd estimate ~4-10 being a sweet spot for number of guardians. This works well with the intended scope of fedimints - small, "local", focused, second-party custody. Since each guardian might be running different hardware/operating system/etc. giving explicit upper limits can be tricky, especially given the relatively small sampling set of people running fedimints in the real world.

These limitations might be improve with a recent switch from HBBFT to Aleph BFT (https://github.com/fedimint/fedimint/pull/2621)