This brings to mind this discussion from the other day where the author was considering the idea of AI supported judging. https://tildes.net/~misc/1epi/theyre_always_guilty_or_are_they The...
This brings to mind this discussion from the other day where the author was considering the idea of AI supported judging.
One of the promises of AI-assisted judging is to help judges consistently apply particular methodologies without engaging in motivated reasoning. The AI could break a case apart as I have above, and then say, “in prior case X you applied the principle that a court should disregard a policy concern because it diverges from the policy concern that led to the rule’s enactment, here’s how the same principle should apply here.” Hopefully, the AI would be braver about pointing out methodological inconsistency across cases than the judge’s law clerks. Today’s judges would likely ignore such advice, but tomorrow’s judges might come to rely on it.
This article seems like a prime example of the risks of AI-supported judging.
The problem with AI is that it is a black box, it doesn't really think and you aren't able to see the patterns it recognizes. While I think you could put some conditions that it should follow, you...
The problem with AI is that it is a black box, it doesn't really think and you aren't able to see the patterns it recognizes. While I think you could put some conditions that it should follow, you need to do that case-by-case and even then it's not certain it won't (accidentally) circumvent it some other way because the "logic" it follows is unknown to us (I think, not an expert).
I'd agree that model interpretability and explainability are key elements of the problem. However, I'm unsure that AI's current lack of ability to think is. People "think" but have the same...
I'd agree that model interpretability and explainability are key elements of the problem. However, I'm unsure that AI's current lack of ability to think is. People "think" but have the same problem, and a sufficiently advanced machine may eventually get to the point where we all agree that it is "thinking", but again, it will still be susceptible to the same underlying problem.
Human brains are also for the most part black boxes. Even though we claim to be able to bring forth explanations for decisions, it seems like our purported explanations are more akin to parallel construction than an actual description of the processes we used to come to those decisions.
Yeah, I agree, whether it thinks or not isn't the problem. That was just to say that AI can't think its way out of a problem that hasn't been explicitly explained to it (unless of course that...
Yeah, I agree, whether it thinks or not isn't the problem. That was just to say that AI can't think its way out of a problem that hasn't been explicitly explained to it (unless of course that problem doesn't occur in the first place) because it can't be aware of it.
Edit. And thanks for taking the time to link stuff, it was interesting.
And the real problem isn't this singular occurrence of bias (I think it's a solvable problem), it's the fact that AI is inherently biased because the data is biased. This study just highlights...
And the real problem isn't this singular occurrence of bias (I think it's a solvable problem), it's the fact that AI is inherently biased because the data is biased. This study just highlights that problem with AI.
Absolutely not. There are countless examples out there of explicitly removing race and ethnicity from a dataset, only to find the AI is still able to figure it out. There's a lot of other cues,...
Absolutely not. There are countless examples out there of explicitly removing race and ethnicity from a dataset, only to find the AI is still able to figure it out. There's a lot of other cues, social and otherwise, to clue someone into the race or ethnicity of a person. Their name. How they speak or what words they use. How they frame things. Their socioeconomic status. All kinds of variables can determine someone's race, ethnicity, gender and other demographics. Here's a non-exhaustive list of a few situations in which demographic data was explicitly removed and yet the AI ended up with a bias:
AI on medical imaging is able to determine race with only an image
Apple credit card ended up being sexist, despite not knowing gender
Amazon's recruit engine selected for men despite not having data on sex or gender
ML tools to determine who to lend money to discriminated against women at higher rates due to the exclusion of gender data
An article on how the removal of demographic data can make discrimination worse
The long and short of it is that the issue is the data it's trained on. The data that these models are trained on are systems which are already racist, sexist, and bigoted in other ways. The best we can do today is to take extra care to do our best to offset the bias by tweaking the data we put in (typically via upsampling or downsampling), adding layers to compensate for bias (such as another model trained to detect discrimination, or by calculating gender from proxies), and other methods meant to identify and counteract the forces we know exist in society. There are plenty of AI ethicists working on this particular problem with all kinds of novel and interesting approaches, but there is unfortunately neither a simple fix nor a fix without both pros and cons. Realistically, however, the human that the AI is meant to replace is also imperfect, so our goal should simply be better and we should pay close attention to whether we're making things better or worse.
Isn't that the entire point of this article, in that it was not provided the ethnicity but it figured it out anyway? And y'know, then proceeded to murder them.
Isn't that the entire point of this article, in that it was not provided the ethnicity but it figured it out anyway? And y'know, then proceeded to murder them.
Then the AI will be biased against people with certain patterns of names, or with certain patterns of street address, cases including certain types of admitted evidence, victims with similar...
Then the AI will be biased against people with certain patterns of names, or with certain patterns of street address, cases including certain types of admitted evidence, victims with similar patterns, transcripts containing certain vernacular, etc. Whatever factors that stand out as patterns of sentencing (which may very well not be transparent) will cluster around the same outcomes.
Curve-fitting a program to a system that we already know to be biased, using historical results, will only ever return similar bias.
It feels to me like this just shifts the problem around. Can we really account for all the context clues that an AI might be using to infer non-essential characteristics that it then proceeds to...
It feels to me like this just shifts the problem around. Can we really account for all the context clues that an AI might be using to infer non-essential characteristics that it then proceeds to misuse in sentencing? And what kinds of new biases and sources of error might we be inserting by layering stacks of PII-stripping AI preprocessors into this system? Might there even be cases where the anonymization process removes elements that are actually germane to matter? I'm thinking of cases such as hate crimes where a victim or perpetrator's ethnicity is actually key to a correct judgement.
There are so many identifiers that are strongly correlated with race, that it would be impossible to anonymize a case such that a sufficiently trained model couldn't make a reasonable guess and be...
There are so many identifiers that are strongly correlated with race, that it would be impossible to anonymize a case such that a sufficiently trained model couldn't make a reasonable guess and be better than random chance. Even simply the crime they're accused of alone gives a strong indicator. Black people tend to be accused of certain crimes at a much higher rate than white people, and vice versa.
This isn't about racism (well, not only). This is about prejudice and bias. And the ai could end up biased against/for anything. Skin color is one example, but it could just as easily be car brand...
This isn't about racism (well, not only). This is about prejudice and bias. And the ai could end up biased against/for anything. Skin color is one example, but it could just as easily be car brand or amount of makeup.
No, given race could be circumstantially derived from an amalgamation of other factors, giving you the same bias - the AI will hold the biases of the material it trained on, which will also...
No, given race could be circumstantially derived from an amalgamation of other factors, giving you the same bias - the AI will hold the biases of the material it trained on, which will also contain the uncountable host of linking factors that would associate to race and/or other prejudicial leanings.
Arrested by officer X (who happens to arrest 80% African Americans)? Path to bias.
Defendant home address listed in county Y or district Z (which happens to have a higher percentage minority population)? Path to bias.
Hell, defendant accused of crime R, the total arrests for which have a statistical leaning of cases in that jurisdiction towards African Americans (which in turn have a corresponding correlation to higher prosecution and convictions because systemic racism)? Path to bias.
AI model training requires a HUGE pool of data to train on to be effective. If that data is real-world sampling, it will be VERY HARD (if not impossible) to have that training data free of the prejudices and biases that exist in humanity.
I think it would be more charitable to assume he means some hypothetical future AI that’s designed for the job and extensively tested for things like bias? Otherwise, it would be like asking a...
I think it would be more charitable to assume he means some hypothetical future AI that’s designed for the job and extensively tested for things like bias?
Otherwise, it would be like asking a general-purpose chatbot to analyze radiology images or something silly like that. That’s not really how it works in regulated safety-related fields. Instead, what tends to happen is that some company tries to build an AI and discovers that it’s a lot harder than they thought, so it never gets deployed. These projects often fail.
Of course, there are idiots like those lawyers who tried to use an AI chatbot to write briefs, but they tend to make headlines and become a cautionary tale.
I think the proper response is to refuse to answer the question. “You’re asking an AI for sentencing guidelines? Seriously?” Until they get that right, people are going to keep trolling, asking...
I think the proper response is to refuse to answer the question. “You’re asking an AI for sentencing guidelines? Seriously?” Until they get that right, people are going to keep trolling, asking gullible AI’s inappropriate hypothetical questions for the lulz.
I wholeheartedly agree, but most non-tech people I talk to don't seem to feel this way. I thinknthe problem is they don't understand the risks and they experience something that mostly works and...
I wholeheartedly agree, but most non-tech people I talk to don't seem to feel this way. I thinknthe problem is they don't understand the risks and they experience something that mostly works and is pretty useful. They set the bit that it's "probably OK" and stop critically engaging with the output, because that's just how people are wired. It's like walking on a bridge that's missing every thousandth plank. Someone can tell you you have to watch every step, but eventually your going to stop looking because the failures are too rare for you to accurately idea the risk in your brain, but not rare enough that they don't cause serious problems.
Edit: after writing this, I ended up reading this article from the blog of a public defender after following it from another article posted by @skybrian. I think this highlights how capricious...
I think this highlights how capricious American justice is, which is something I wish were fixable, but it also makes clear my point about how vague our notion of justice actually is.
Caveat: when I talk about the legal/justice system, I generally mean in the US.
What if a future model is shown to make more just decisions than human judges
The biggest problem I see with this hypothetical is that we have no objective measure for justice. There's no training function to optimize for that, because even as humans we just kind of know it when we see it. I don't even know how one would say that X court system is more just than Y court system, so how would one begin to make a claim like thst about an LLM?
One one of the things that makes the legal system better (not perfect) is the adversarial nature of it. There is supposed to be someone advocating for both sides. I think this could be an interesting application for lawGPT - make it adversarial with the human court.
Suppose you have a way for lawGPT to take in everything the court takes in for a trial, and it produces a blind second judgement that the human court has no access to (and vice versa) until a trial is finished. If the judgements are in agreement, the judgements stands(appeals are still allowed), but if they differ, there is an automatic review. It is not automatically assumed that the human is wrong, or the GPT is wrong, but the outcomes are compared.
Couple of challenges to this: human juries are typically not asked to justify their decision, and if they were, how would a lay person not necessarily schooled in logic, rhetoric, or debate, defend their decision? (This, of course, assumes that the GPT can be interrogated for its motives). So what the review would be might be murky.
Another problem is this makes justice more expensive, and typically people are turning to LLMs to make things more efficient. So the desire will be to replace the human trial, not go alongside it. Our legal system already perpetrates a lot of injustice in the name of saving money, so we have already shown as a society that we aren't willing to bear those costs to see justice done.~~~~
That sounds like putting a bandaid over this one specific case. It doesn’t address the harms that come from the technology being deployed in the near infinite other use cases where the harms...
That sounds like putting a bandaid over this one specific case. It doesn’t address the harms that come from the technology being deployed in the near infinite other use cases where the harms aren’t as clearly identifiable.
It’s easy to see the exact harms of, for instance, a specific person having their sentence influenced by an AI. It’s less clear what the harm is in billions of people interacting with AI generated social media content that is suffused with intentional and unintentional bias amplifying language.
Maybe we should do it the other way: there are no general-purpose tasks, and AI has to be verified to be working well for each specific task before it can answer questions about that?...
Maybe we should do it the other way: there are no general-purpose tasks, and AI has to be verified to be working well for each specific task before it can answer questions about that?
Unfortunately that’s a pretty high bar for more casual usage, and the possible tasks you could use it for is fractal in complexity.
For software development, there are a few highly-regulated safety-related fields, and then there is everything else.
And still people will run and complain about AI working to counter those biases because god forbid a person of color is generated in an Image when it’s not supposed to be. Also interesting to...
And still people will run and complain about AI working to counter those biases because god forbid a person of color is generated in an Image when it’s not supposed to be.
Also interesting to consider the data that is being put it. Human bias is modeled in the data as well.
I agree that bias needs to be countered, but it matters to me how that's done. As with education and hiring pipelines, earlier in the process is better than later. The "diverse Nazis" phenomenon...
I agree that bias needs to be countered, but it matters to me how that's done. As with education and hiring pipelines, earlier in the process is better than later. The "diverse Nazis" phenomenon smacks of late stage bias correction. It's better than nothing, but I'm hoping that with more investment we can approach end-to-end fairness.
Hofmann said that, because overt racism is decreasing in LLMs, there could be a risk that those interpreting the study are taking it as "a sign that racism has been solved," instead of showing that the way LLMs show racial bias is changing.
The regular way of teaching LLMs new patterns of retrieving information, by giving human feedback, doesn’t help counter covert racial bias, the study showed.
Instead, it found that it could teach language models to "superficially conceal the racism they maintain on a deeper level".
I think Hofmann would agree that late stage bias correction is dangerous for that superficial concealment.
Since AI is not used for sentencing anyone while image generators are already used relatively commonly to generate illustrations for online articles, concept art for marketing or videogames and...
Since AI is not used for sentencing anyone while image generators are already used relatively commonly to generate illustrations for online articles, concept art for marketing or videogames and other uses, I don't think it's strange that people care more about generating black nazis than about hypotheticals that are usually generated for clickbait anyway.
I would also add that we already had an example of an image generation AI that usually creates diversity by default, Dall-E 3, and people generally didn't complain about it much because it doesn't go completely nonsensically over the top. Though the fact that it refuses to generate anything even mildly controversial may help as well - people may just complain about that instead.
The AI that people are talking about now is the ChatGPT-like AI which is quite well known not to be good at logic. Just ask it rather complicated Math questions or anything in your specific field...
The AI that people are talking about now is the ChatGPT-like AI which is quite well known not to be good at logic. Just ask it rather complicated Math questions or anything in your specific field of interest.
And even if it did get it right, try suggesting that it was wrong. They would quickly backtrack!
And even if they fixed all of the above problems, would you be comfortable with an AI sentencing you or one of your loved ones?
While researching this topic, I came across this interesting tool: COMPAS. It’s an actual live tool used in the real world right now for sentencing. https://en.wikipedia.org/wiki/COMPAS_(software)...
While researching this topic, I came across this interesting tool: COMPAS. It’s an actual live tool used in the real world right now for sentencing.
Another general criticism of machine-learning based algorithms is since they are data-dependent if the data are biased, the software will likely yield biased results.
Specifically, COMPAS risk assessments have been argued to violate 14th Amendment Equal Protection rights on the basis of race, since the algorithms are argued to be racially discriminatory, to result in disparate treatment, and to not be narrowly tailored.
I wonder if the bias is nearly as much classism as it is racism. Most people who grew up with resources are more likely to be well educated and have been exposed to a wider variety of language...
I wonder if the bias is nearly as much classism as it is racism. Most people who grew up with resources are more likely to be well educated and have been exposed to a wider variety of language styles.
People tend to lean towards more formal language in a setting like a courtroom, but they can only do that credibly if they're comfortable with formal language.
People with more resources are more likely to have good representation in court. They also tend to present as better put together and more trustworthy.
We know from statistics that people on the lower end of the socioeconomic scale are more often arrested and more often convicted. We also know that, as a result of systemic racism, black people are more likely to be be poor than white people.
All of which means, perhaps, that from the perspective of an LLM, elements of "poor" language are closer, in terms of weight, to words like guilty and conviction.
One thing this study exposes is how completely absurd it is to use an LLM for anything important (if that wasn't already obvious enough). But it's also just generally interesting how LLMs can show how deeply classism and racism are baked into both language and our systems.
This brings to mind this discussion from the other day where the author was considering the idea of AI supported judging.
https://tildes.net/~misc/1epi/theyre_always_guilty_or_are_they
The relevant quote was
This article seems like a prime example of the risks of AI-supported judging.
Couldn’t this be solved by not supplying the AI with the ethnicity or race of the accused?
The problem with AI is that it is a black box, it doesn't really think and you aren't able to see the patterns it recognizes. While I think you could put some conditions that it should follow, you need to do that case-by-case and even then it's not certain it won't (accidentally) circumvent it some other way because the "logic" it follows is unknown to us (I think, not an expert).
I'd agree that model interpretability and explainability are key elements of the problem. However, I'm unsure that AI's current lack of ability to think is. People "think" but have the same problem, and a sufficiently advanced machine may eventually get to the point where we all agree that it is "thinking", but again, it will still be susceptible to the same underlying problem.
Human brains are also for the most part black boxes. Even though we claim to be able to bring forth explanations for decisions, it seems like our purported explanations are more akin to parallel construction than an actual description of the processes we used to come to those decisions.
Yeah, I agree, whether it thinks or not isn't the problem. That was just to say that AI can't think its way out of a problem that hasn't been explicitly explained to it (unless of course that problem doesn't occur in the first place) because it can't be aware of it.
Edit. And thanks for taking the time to link stuff, it was interesting.
There’s no guarantee that the model is even “consciously” aware of this fact though. That’s not how the current generation of models works.
And the real problem isn't this singular occurrence of bias (I think it's a solvable problem), it's the fact that AI is inherently biased because the data is biased. This study just highlights that problem with AI.
Absolutely not. There are countless examples out there of explicitly removing race and ethnicity from a dataset, only to find the AI is still able to figure it out. There's a lot of other cues, social and otherwise, to clue someone into the race or ethnicity of a person. Their name. How they speak or what words they use. How they frame things. Their socioeconomic status. All kinds of variables can determine someone's race, ethnicity, gender and other demographics. Here's a non-exhaustive list of a few situations in which demographic data was explicitly removed and yet the AI ended up with a bias:
The long and short of it is that the issue is the data it's trained on. The data that these models are trained on are systems which are already racist, sexist, and bigoted in other ways. The best we can do today is to take extra care to do our best to offset the bias by tweaking the data we put in (typically via upsampling or downsampling), adding layers to compensate for bias (such as another model trained to detect discrimination, or by calculating gender from proxies), and other methods meant to identify and counteract the forces we know exist in society. There are plenty of AI ethicists working on this particular problem with all kinds of novel and interesting approaches, but there is unfortunately neither a simple fix nor a fix without both pros and cons. Realistically, however, the human that the AI is meant to replace is also imperfect, so our goal should simply be better and we should pay close attention to whether we're making things better or worse.
Isn't that the entire point of this article, in that it was not provided the ethnicity but it figured it out anyway? And y'know, then proceeded to murder them.
Then the AI will be biased against people with certain patterns of names, or with certain patterns of street address, cases including certain types of admitted evidence, victims with similar patterns, transcripts containing certain vernacular, etc. Whatever factors that stand out as patterns of sentencing (which may very well not be transparent) will cluster around the same outcomes.
Curve-fitting a program to a system that we already know to be biased, using historical results, will only ever return similar bias.
Including not supplying their speech patterns or accent?
I mean, yeah. If we really care to do this, why not run the defendant's statement through another AI that will formalize the language they use?
It feels to me like this just shifts the problem around. Can we really account for all the context clues that an AI might be using to infer non-essential characteristics that it then proceeds to misuse in sentencing? And what kinds of new biases and sources of error might we be inserting by layering stacks of PII-stripping AI preprocessors into this system? Might there even be cases where the anonymization process removes elements that are actually germane to matter? I'm thinking of cases such as hate crimes where a victim or perpetrator's ethnicity is actually key to a correct judgement.
There are so many identifiers that are strongly correlated with race, that it would be impossible to anonymize a case such that a sufficiently trained model couldn't make a reasonable guess and be better than random chance. Even simply the crime they're accused of alone gives a strong indicator. Black people tend to be accused of certain crimes at a much higher rate than white people, and vice versa.
This isn't about racism (well, not only). This is about prejudice and bias. And the ai could end up biased against/for anything. Skin color is one example, but it could just as easily be car brand or amount of makeup.
No, given race could be circumstantially derived from an amalgamation of other factors, giving you the same bias - the AI will hold the biases of the material it trained on, which will also contain the uncountable host of linking factors that would associate to race and/or other prejudicial leanings.
Arrested by officer X (who happens to arrest 80% African Americans)? Path to bias.
Defendant home address listed in county Y or district Z (which happens to have a higher percentage minority population)? Path to bias.
Hell, defendant accused of crime R, the total arrests for which have a statistical leaning of cases in that jurisdiction towards African Americans (which in turn have a corresponding correlation to higher prosecution and convictions because systemic racism)? Path to bias.
AI model training requires a HUGE pool of data to train on to be effective. If that data is real-world sampling, it will be VERY HARD (if not impossible) to have that training data free of the prejudices and biases that exist in humanity.
I think it would be more charitable to assume he means some hypothetical future AI that’s designed for the job and extensively tested for things like bias?
Otherwise, it would be like asking a general-purpose chatbot to analyze radiology images or something silly like that. That’s not really how it works in regulated safety-related fields. Instead, what tends to happen is that some company tries to build an AI and discovers that it’s a lot harder than they thought, so it never gets deployed. These projects often fail.
Of course, there are idiots like those lawyers who tried to use an AI chatbot to write briefs, but they tend to make headlines and become a cautionary tale.
I think the proper response is to refuse to answer the question. “You’re asking an AI for sentencing guidelines? Seriously?” Until they get that right, people are going to keep trolling, asking gullible AI’s inappropriate hypothetical questions for the lulz.
I wholeheartedly agree, but most non-tech people I talk to don't seem to feel this way. I thinknthe problem is they don't understand the risks and they experience something that mostly works and is pretty useful. They set the bit that it's "probably OK" and stop critically engaging with the output, because that's just how people are wired. It's like walking on a bridge that's missing every thousandth plank. Someone can tell you you have to watch every step, but eventually your going to stop looking because the failures are too rare for you to accurately idea the risk in your brain, but not rare enough that they don't cause serious problems.
Edit: after writing this, I ended up reading this article from the blog of a public defender after following it from another article posted by @skybrian.
I think this highlights how capricious American justice is, which is something I wish were fixable, but it also makes clear my point about how vague our notion of justice actually is.
Caveat: when I talk about the legal/justice system, I generally mean in the US.
The biggest problem I see with this hypothetical is that we have no objective measure for justice. There's no training function to optimize for that, because even as humans we just kind of know it when we see it. I don't even know how one would say that X court system is more just than Y court system, so how would one begin to make a claim like thst about an LLM?
One one of the things that makes the legal system better (not perfect) is the adversarial nature of it. There is supposed to be someone advocating for both sides. I think this could be an interesting application for lawGPT - make it adversarial with the human court.
Suppose you have a way for lawGPT to take in everything the court takes in for a trial, and it produces a blind second judgement that the human court has no access to (and vice versa) until a trial is finished. If the judgements are in agreement, the judgements stands(appeals are still allowed), but if they differ, there is an automatic review. It is not automatically assumed that the human is wrong, or the GPT is wrong, but the outcomes are compared.
Couple of challenges to this: human juries are typically not asked to justify their decision, and if they were, how would a lay person not necessarily schooled in logic, rhetoric, or debate, defend their decision? (This, of course, assumes that the GPT can be interrogated for its motives). So what the review would be might be murky.
Another problem is this makes justice more expensive, and typically people are turning to LLMs to make things more efficient. So the desire will be to replace the human trial, not go alongside it. Our legal system already perpetrates a lot of injustice in the name of saving money, so we have already shown as a society that we aren't willing to bear those costs to see justice done.~~~~
That sounds like putting a bandaid over this one specific case. It doesn’t address the harms that come from the technology being deployed in the near infinite other use cases where the harms aren’t as clearly identifiable.
It’s easy to see the exact harms of, for instance, a specific person having their sentence influenced by an AI. It’s less clear what the harm is in billions of people interacting with AI generated social media content that is suffused with intentional and unintentional bias amplifying language.
Maybe we should do it the other way: there are no general-purpose tasks, and AI has to be verified to be working well for each specific task before it can answer questions about that?
Unfortunately that’s a pretty high bar for more casual usage, and the possible tasks you could use it for is fractal in complexity.
For software development, there are a few highly-regulated safety-related fields, and then there is everything else.
And still people will run and complain about AI working to counter those biases because god forbid a person of color is generated in an Image when it’s not supposed to be.
Also interesting to consider the data that is being put it. Human bias is modeled in the data as well.
I agree that bias needs to be countered, but it matters to me how that's done. As with education and hiring pipelines, earlier in the process is better than later. The "diverse Nazis" phenomenon smacks of late stage bias correction. It's better than nothing, but I'm hoping that with more investment we can approach end-to-end fairness.
I think Hofmann would agree that late stage bias correction is dangerous for that superficial concealment.
Since AI is not used for sentencing anyone while image generators are already used relatively commonly to generate illustrations for online articles, concept art for marketing or videogames and other uses, I don't think it's strange that people care more about generating black nazis than about hypotheticals that are usually generated for clickbait anyway.
I would also add that we already had an example of an image generation AI that usually creates diversity by default, Dall-E 3, and people generally didn't complain about it much because it doesn't go completely nonsensically over the top. Though the fact that it refuses to generate anything even mildly controversial may help as well - people may just complain about that instead.
The AI that people are talking about now is the ChatGPT-like AI which is quite well known not to be good at logic. Just ask it rather complicated Math questions or anything in your specific field of interest.
And even if it did get it right, try suggesting that it was wrong. They would quickly backtrack!
And even if they fixed all of the above problems, would you be comfortable with an AI sentencing you or one of your loved ones?
While researching this topic, I came across this interesting tool: COMPAS. It’s an actual live tool used in the real world right now for sentencing.
https://en.wikipedia.org/wiki/COMPAS_(software)
And there’s this ProPublica article which sums it up:
I wonder if the bias is nearly as much classism as it is racism. Most people who grew up with resources are more likely to be well educated and have been exposed to a wider variety of language styles.
People tend to lean towards more formal language in a setting like a courtroom, but they can only do that credibly if they're comfortable with formal language.
People with more resources are more likely to have good representation in court. They also tend to present as better put together and more trustworthy.
We know from statistics that people on the lower end of the socioeconomic scale are more often arrested and more often convicted. We also know that, as a result of systemic racism, black people are more likely to be be poor than white people.
All of which means, perhaps, that from the perspective of an LLM, elements of "poor" language are closer, in terms of weight, to words like guilty and conviction.
One thing this study exposes is how completely absurd it is to use an LLM for anything important (if that wasn't already obvious enough). But it's also just generally interesting how LLMs can show how deeply classism and racism are baked into both language and our systems.
Link to the paper.