I've tried and yes it's ultimately very difficult to exist in modern society without one. I even went the same route you did with an iPhone SE but I don't think iOS is customizable enough to get rid of the distractions. I ended up going with GrapheneOS and Olauncher (text based so there's no shiny icons to distract you) which feels like the exact balance I wanted.

With Graphene/Olauncher I would unlock my phone to look out of habit but there wasn't much to see.. just a short text list of apps. As the dopamine kick disappeared I found myself mindlessly unlocking/checking less and less. After a couple years on this path I def notice myself being less interested in my phone except for the few functions I truly find it useful for. I also find myself heads-up when out side, on the bus, on the sidewalk. You know what you see? Lots of people with their face in their phones missing the beauty and calm of living IRL.