Difficulty will simply readjust until mining is profitable again. It doesn't need massive Ordinal fees to survive.