Funny, my experience at the same park in Osaka this past summer is what changed my mind.

1. From a pure economic perspective, you'll get roughly double the number of rides by buying fastpass, so depending on how much of a priority you put into the ride experience, it's not a hard calculation to see if it's worth it.
2. For me, not buying one is penny wise and pound foolish. The point of going to these places is to have a good time. Waiting in line sucks (especially outdoors in hot weather) and detracts from your experience, so reducing that has significant utility, beyond the time savings.
3. Ultimately, this is a case of skimpflation, where they degrade the main experience because they can't easily raise the prices of the general ticket too much without taking a big PR hit. If you're on a Bitcoin standard, the tickets are getting cheaper in BTC terms, even with the additional cost of a priority pass. You don't need to be a cheapskate or have your experience debased.