I'm not equipped to solve this problem. It's extremely complex. The difficulty lies in the fact that groups of people sliced in different ways do have real differences of all kinds and AI can uncover them in order to make better predictions. But this also means individuals within those groups may be treated differently.

For example, AI uncovers that people driving red cars suffer more accidents therefore they will raise the insurance for all of them. I drive a red car, but I'm extremely careful and my accident rate may be way lower than the average, yet I will have to suffer from the red car tax. No big issue one may say, as you can simply buy white cars instead. That's true, but what if the red car tax is a hidden tax for a group of people who can be identified by a race, age, gender or trait we should not be biased against? If we don't do the red car tax, we're punishing everyone else for the red car driver accident rates or go out of business. If we do the red car tax, we indirectly force the group of change their choices and behaviours and lose culture bit by bit. Imagine when everything is permeated by AI. There could be massive cultural shifts where groups are marginilized by AI indirectly, even if we try to manually control for direct discrimination by not allowing a religion, political affiliation or other obvious groups to be judged differently.

Not to mention that such manual overrides can backfire in spectacular ways. This happens without AI already. For example, we force hiring from a certain group despite their real world statistical lack of competence. A truly expert hire from this group will suffer, because everyone in the company will be aware of their group's preferential treatment and thus will not trust their real skills. This individual despite their real skill will be marginalized because of this artificial situation, which is unfair because they didn't need the forced hire policy in the first place.