I'm not a biologist but I don't think they buildup like that. Your body takes them apart and uses them for stuff afaik (in contrast to something truly toxic that acculates like mercury). The stuff your body uses seed oils for might differ, at the margins, from the stuff it'd use healthier fats for, but I think if you just stop eating them, that should be enough.