I have no doubt that there were people who associated with Epstein solely for social or financial access who had no interest in doing anything inappropriate with kids (or anyone else for that matter).
It seems logical that part of running an enormous blackmail ring is testing the waters with important people and trying to figure out what you can get them to do. For some, that might just be luring them into an extramarital affair, or maybe a homosexual relationship back when people cared about that, or even just make them complicit in some kind of financial crime.
The reason I'm so in favor of presuming guilt by association in this case, when I would generally oppose it, is because of the crimes he was already known to have committed when these people chose to associate with him and because of how blatant the cover up is. Until this whole thing is resolved, there needs to be serious social stigma for being on those lists.