I make AI for a living and it would be very easy to bias the data.

Let's for instance say that I use "address" as a raw feature to give to the model. This will definitely be an important feature because education and competence are associated with where you live.

However this correlation is an artifact of other things. The AI can not tell the difference between correlation and causation. So in the example, address correlates with competence but does not cause competence. Where maybe something like the ability to solve a math problem is actual evidence of competence.

Basically, the raw data can be biased. If you just take all your company's hiring data and feed it into a model, the model will learn to replicate any discriminatory practices that historically existed at your company. (And there are plenty of studies that suggest such bias exists even among well-meaning hiring managers who attempt to be race/gender neutral.) Suppose you have a raw dataset where 20% of white applicants are hired and only 10% of applicants of color are hired. Even if you exclude the applicants' race from the features used by the model, you will likely end up with a system that is half as likely to hire applicants' of color compared to white applicants. AI is extremely good at extracting patterns from disparate data points, so it will find other, subtler indicators of race and learn to penalize them. Maybe it decides that degrees from historically black universities are less valuable than degrees from predominantly white liberal arts schools. Maybe it decides that guys named DeSean are less qualified than guys named Sean. You get the picture. Correcting these biases in the raw data isn't quite the same as filling quotas. The idea is that two equally qualified applicants have the same likelihood of getting hired. You could have a perfectly unbiased model and still fail to meet a quota because no people of color apply in the first place.

A good real life example comes from Amazon a few years ago. They implemented AI that "learned" to prefer male candidates. From this article:

>In effect, Amazon’s system taught itself that male candidates were preferable. It penalized resumes that included the word “women’s,” as in “women’s chess club captain.” And it downgraded graduates of two all-women’s colleges, according to people familiar with the matter.

What it came down to was that the tech industry tends to be male dominated, so based on the resumes fed to the AI, it identified a pattern that successful candidates don't have words like "women's" in their resume or go to all-women schools.

needs to be biased in an approved way.

Let's say your looking to fill a position for an engineer. You train your AI by looking at the resumes if your current engineering employees. It picks up that they are all old white dudes by looking at their names and experiences.

Guess what's going to get past your AI.

>How can AI be biased if it's only looking at raw data. Wouldn't it be inherently unbiased? I don't know just asking

Data can be bad looking at groups, not reflecting individuals.

If you have one person who belongs to group z, and this person is a criminal, steals, and commits assault, you wouldn't want to hire him. But the AI just choses not to hire him because he belongs to group z, and group z on average commits 10x the crime of any other group. It does the same to another guy of group z, who has a spotless record, or who has a brother that died to crime, so he is at risk of committing crime due to the association of others that revenge kill.

Basically AI can only see aggregate behavior, because judging individuals would require a level of insight that would require a dystopian amount of real time access to that persons data.

Technically an AI could look at groups and be like " On average these guys have good traits" but that's literally the definition of bigotry.

I recommend watching Coded Bias on Netflix. It sums it up pretty well.

AI is still an interpreter of data; there is no perfectly “true” interpretation of raw data. There is always a process of interpreting data to have some meaning. Interpretation is prone to bias.

If the machine learning model makes interpretations based on prior interpretations made (eg “historically only white or male candidates have been successfully hired in this role”) then this can perpetuate existing bias. Until recently the engineers building these models have not been thinking to build in safeguards against bias. Laws like these ensure that these kinds of biases are safeguarded against.

Think of this like building codes in architecture/structural engineering.

Adding on to what others are saying, the raw data is a measurement of our world, and the way we have constructed and formed our world is inherently biased. People are congregated into clusters physically, economically, and socially for all manner of reasons, many of which are unfit criteria for selection. Even after unjust actions are halted, they leave echoes in how the lives of those people and their children are affected: where they grew up, where and how much property they may own, where they went to school, and so on. Those unfit criteria are leaked through anything that gives a proxy measure of those clusters, sometimes in surprising and unintuitive ways that cannot necessarily be scrubbed out or hidden.

So from what I understand, the models can be biased if they’re created by humans with particular bias - it’s hard to measure exactly how this happens which is why when this law comes in, companies using automated systems will have to have them audited by independent organizations. The goal is of course for the models to be as unbiased as possible, but what happens today (in some cases, not all) is that the AI model will have inherent biases against certain profiles.

My take on it is this: Say on the resume the applicant misspelt ask as aks. The AI might rule out the resume due to the spelling error suggesting a low quality / careless applicant. The problem is if the AI starts correlating the misspelling of aks to certain cultural groups that do this normally and consequently associates that group with the 'careless' behaviour by default.