Primarily the time it takes.

If something is a distraction but it only takes a couple minutes once it was only a small waste of time. Learn to live with it, don't gripe about it and move on.

If something takes significant time or repeatably wastes my mental load then it's a distraction and i'll delegate or ignore it.