The conclusion is interesting: the author suggests using AIs to help judges interpret law consistently. To my mind, it seems like an AI is at least as likely to contain biases than a human judge,...
The conclusion is interesting: the author suggests using AIs to help judges interpret law consistently. To my mind, it seems like an AI is at least as likely to contain biases than a human judge, but it may be much harder to surface and examine those biases in a human comprehensible way.
Certainly the biggest risk is humans beginning to automatically defer to the machines, because they will in all likelihood be correct most of the time — but those unexamined edge cases where the machine is wrong may result in some very bad case law making it into the books.
It concerns me how many people think of AIs as unbiased and logical, when the whole purpose of training them is to imitate the behavior in the training data: to imitate the illogical humans...
It concerns me how many people think of AIs as unbiased and logical, when the whole purpose of training them is to imitate the behavior in the training data: to imitate the illogical humans they're supposed to replace.
To be fair, while that applies to LLMs and certain other models, there are alternative ideas that we may look towards in the future that are not so imitation based. It wasn’t long ago that systems...
To be fair, while that applies to LLMs and certain other models, there are alternative ideas that we may look towards in the future that are not so imitation based. It wasn’t long ago that systems like Prolog were in vogue, and I think we may see the deep integration of hard mathematical systems into LLMs relatively soon. We can see some small steps towards that in the plugin architecture in GPT for instance.
That’s not to say that these hypothetical systems will be infallible either though, or not subject to deliberate alignment bending by malicious actors.
There's an entire subfield dedicated to fighting these biases and while sometimes it works out sort of ok, a judicial AI would have to be reliably significantly better than humans. Which...
There's an entire subfield dedicated to fighting these biases and while sometimes it works out sort of ok, a judicial AI would have to be reliably significantly better than humans. Which disturbingly enough is a rather low bar. And yet even the best efforts are far from that.
Having failed to learn my lesson, I will once again address an obscure case about evidence law: Diaz v. United States, which will be argued in the Supreme Court on March 19. Diaz is an intriguing case because, on the surface, it looks extremely simple. The facts are easily comprehensible to a lay person. The legal provision at issue is two sentences long. And best of all, there is no relevant case law to weigh us down. You’d think the case would be resolved easily—either there’s a straightforward right answer, or there’s a single dimension of disagreement in which the Court can pick a position without too much fuss.
Unfortunately not. The case abounds with disagreements. Textual disagreements, philosophical disagreements, empirical disagreements. It’s like a fractal—if you zoom in on any one sub-issue, the case does not get any simpler, but instead you get a bunch of disagreements about that sub-issue. Why that is—and what that says about legal reasoning—is the subject of today’s post.
Jumped right to the end to read the conclusion to see if this long article would be worth the time. In the summary, the author puts forth 4 points. Judging is complicated. Judging is political....
Exemplary
Jumped right to the end to read the conclusion to see if this long article would be worth the time. In the summary, the author puts forth 4 points.
Judging is complicated.
Judging is political.
Judging should be repeatable or consistent for a judge.
AI can help judges.
There's really not a lot to deal with here. Yes of course, judging is complicated, there's a reason that we treat judges with respect - a lot of schooling and work goes into becoming a judge, and being a judge on a higher court generally comes with a ton of experience. Judging has always been political, and there isn't a country out there where it isn't, because what is political is constantly expanding. I personally don't think there's any issue with judges being political, but I do agree that politics in judging should be minimized on certain axes. Being humanistic, for example, is political. Choosing to value all human rights, is also political. I think it's okay for folks to center these values and in fact I would prefer a system where they do.
On item 3 I'm not sure many people would disagree. We'd all like to see experts be consistent, as it would reflect a grounded system of reasoning. We might expect certain issues to drift over time as their politics change, such as becoming more humanistic as a sum of their experiences with humans, but there's really nothing we can do about this except teach people to critically evaluate themselves. The final point, however, is laughable. We already have multiple court cases about AI hallucinating law. Using an AI to assist judges is rife with potential problems. Suggesting that we use AI to enhance judges ignores all the problems we have with AI. Of particular note, the idea that a machine trained on human data would be less biased than a human is a game of statistical chance - it will certainly be less biased than some humans, but because it is trained on the sum of human bias, it will also be more biased than some humans. Should we not strive to ensure that judges get training to reduce their bias, firmly placing the majority of them on the less biased side? Also, who's to say the very judges the author is criticizing is not already using AI to refine their arguments?
I think of particular note here is the very real risk of biased judges choosing biased AI to reinforce their beliefs. I'm reminded of an image I stumbled upon a few days ago, where someone reasonably easily got the "uncensored and unbiased AI" AI of gab to reveal its prompting instructions. Any discussion which talks about opening the door for judges to use AI to assist in their decisions must include a discussion about how to stop against bad actors like this because there will be people who sell products which exist to increase bias. The judicial system has a history of political wars happening for hundreds of years, with the supreme court being a popular battlefield to gain and remove rights. What's to stop a company from engineering an AI to provide a biased response, through the use of instructions such as the above or being even more clever and deciding which pieces of law or interpretation of law the AI gets trained on?
Yes, I’m quite dubious of the AI recommendation. I thought the article was interesting for showing how deep the rabbit hole goes, even for a simple question. It seems to me that the difficulty...
Yes, I’m quite dubious of the AI recommendation. I thought the article was interesting for showing how deep the rabbit hole goes, even for a simple question.
It seems to me that the difficulty comes from wanting consistency, because there are so many things that a ruling could be consistent with in so many ways. Considered as a standalone policy question, it seems simpler: how much evidence should we require for punishing someone as a drug mule? Is finding a large amount of fentanyl in the car enough evidence, or is more needed to avoid punishing unwitting drug mules? Even if that possibility is rare, it should be considered.
A legislature considering the policy question should probably take into account how drug smuggling currently works, with better evidence than one expert’s opinion. And then, also take into account how criminals respond to incentives, and that’s going to be a guess.
It seems like the court system isn’t a great place to have this policy debate, because everything becomes a consistency argument?
I think people view objectivity as necessary for consistency, but I'm not so sure that's true. Someone who is afraid of snakes will pretty consistently remain afraid of snakes and act in a manner...
I think people view objectivity as necessary for consistency, but I'm not so sure that's true. Someone who is afraid of snakes will pretty consistently remain afraid of snakes and act in a manner that someone afraid of snakes would act. In the realm of objective science, objectivity is a necessary part of remaining consistent, but in the realm of policy this isn't necessarily true. I think we should strive to have policy and law be very specific about what we consider should be punished versus that which should not (of note, the word punish implies a punitive justice system but I am using it more broadly here to indicate something we have decided is incorrect behavior and the response to said behavior which hopefully involves restoration and education) . I also think we should design a system in which policy makers must include experts in the field and listen to their advice when setting policy. We also should strive to train policy makers, experts in the field, and judicial staff in critical thinking so that they can properly examine issues in ways which minimize bias.
The reality, however, is that we have not designed a system like this and we need to consider how we can work best with the system we have, and how we can strive to improve the system. Bandages like AI to solve an immediate problem which are just as likely to be hijacked as the current system (perhaps more likely, given their complete lack of regulation and how little humanity currently understands it) are not reasonable solutions. The article could have just as easily focused on all the ways in which the system is currently broken - the places in which it is not regulated and the places in which money interferes with the desires of the people are perhaps the two most salient existing attack vectors. Ultimately how a judge decides a case matters very little if the laws themselves already reflect a broken system - it's just one more space in which reasoning and human nature can continue to make the system inconsistent.
In short, I agree that the court system isn't a great place to have this debate, but ultimately none of the existing systems are great places for this inconsistency to be created and I don't think there's anything special about the court system which warrants this particular attention or an attempt to fix things. It seems more like a red herring to me, to highlight that judges can be inconsistent when they happen to be much more isolated from the corrupting influence of capital than the legislators themselves. Both good and bad has come from the ability for judges to interpret law and we cannot be narrow sighted by focusing only on the bad and fail to adequately acknowledge the good. For what it's worth, I'm not sure that more bad or more good has come from the ability for judges to interpret law, but our system is designed rather uniquely in that we give the text of law much more precedence than the spirit of law. Systems of law which do the opposite tend to have more consistent results (although I'm sure many will argue they are less consistent) because they tend to design their judicial system with safeguards around the additional leeway they give to judges which do a lot of interpretation in order to ensure that the spirit is appropriately upheld
Yes, there is no good answer here. But one side-effect of judges making policy decisions under the current system is that they don't necessarily admit to doing so - they can justify a decision to...
Yes, there is no good answer here. But one side-effect of judges making policy decisions under the current system is that they don't necessarily admit to doing so - they can justify a decision to be based on precedent with a pretty solid argument, but actually have a lot of freedom to be "results-oriented."
I'm not sure that being results-oriented is bad when a legislature is dysfunctional, though? Or in general. There's something to be said for making sure the results of a judgement aren't unjust. But, it invites skepticism about whether a judge's stated reasons are their real reasons.
I read another blog post by the same author where he wrote about being somewhat too results-oriented. It's long and I didn't follow the first part about what are apparently some rather dubious rules about Louisiana redistricting. Skipping down to The lions (no longer) sleep tonight:
I’ve been following federal appellate courts since I graduated from law school in 2007. To my mind, the biggest change to the judiciary since that time is the disappearance of a certain style of progressive judging that a select number of judges used to employ. Most of these judges were appointees of President Carter; notable examples were Judges Stephen Reinhardt and Harry Pregerson on the Ninth Circuit and Judge Boyce Martin on the Sixth Circuit. Obituaries of Judge Reinhardt and Judge Pregerson refer to them as “liberal lions,” so I’ll call these judges “lions” for lack of a better term.
The lions are gone. Although many Clinton and Obama appointees are quite progressive, none of them are lions in the same way as the Carter appointees. (Perhaps some Biden appointees will be lions; we’ll have to wait and see.)
What made the lions unique? Their jurisprudential style had certain hallmarks:
They ruled on grounds never argued by the parties. [...]
They believed that a judge’s personal sense of justice could constitute clearly established law. [...]
They disregarded standards of review. [...]
They took extraordinary measures to ensure that the “right” judges decided important cases. [...]
They viewed the Supreme Court as an antagonist [...]
...
I think the “horseshoe theory” applies to judging too: the jurisprudential styles of very progressive and very conservative judges tend to converge. Obviously, the conservative lions’ and liberal lions’ visions of social justice diverge, but their views of how judges should go about achieving those visions are aligned.
The Fifth Circuit’s mandamus order is a case in point. Although the judges in the panel majority are considered to be quite conservative, the panel’s mandamus order ticks every “lion” box:
Ruling sua sponte. [...]
Believing that a judge’s personal sense of justice could constitute clearly established law. [...]
Disregarding standards of review. [...]
Granting a stay in a case that was already assigned to a different merits panel. [...]
Viewing the Supreme Court as an antagonist. [...]
So it does seem like being "results-oriented" is a tactic that can be used for opposing purposes, and can go too far.
Yeah and in extreme cases they can claim it's the proper interpretation of the constitution, focusing on a few specific words and disregarding others. There's no way to stop this from happening -...
they can justify a decision to be based on precedent with a pretty solid argument
Yeah and in extreme cases they can claim it's the proper interpretation of the constitution, focusing on a few specific words and disregarding others. There's no way to stop this from happening - people will always find inventive and creative ways to exploit the system. The key, I believe, is strong safeguards. Regulation to prevent the influence of capital, a strong educational system to instill critical thinking, designing the system to be responsible to the population or outcomes of their judgements, and designing the system itself to be resilient such as with panels of judges as opposed to individuals.
The conclusion is interesting: the author suggests using AIs to help judges interpret law consistently. To my mind, it seems like an AI is at least as likely to contain biases than a human judge, but it may be much harder to surface and examine those biases in a human comprehensible way.
Certainly the biggest risk is humans beginning to automatically defer to the machines, because they will in all likelihood be correct most of the time — but those unexamined edge cases where the machine is wrong may result in some very bad case law making it into the books.
It concerns me how many people think of AIs as unbiased and logical, when the whole purpose of training them is to imitate the behavior in the training data: to imitate the illogical humans they're supposed to replace.
To be fair, while that applies to LLMs and certain other models, there are alternative ideas that we may look towards in the future that are not so imitation based. It wasn’t long ago that systems like Prolog were in vogue, and I think we may see the deep integration of hard mathematical systems into LLMs relatively soon. We can see some small steps towards that in the plugin architecture in GPT for instance.
That’s not to say that these hypothetical systems will be infallible either though, or not subject to deliberate alignment bending by malicious actors.
There's an entire subfield dedicated to fighting these biases and while sometimes it works out sort of ok, a judicial AI would have to be reliably significantly better than humans. Which disturbingly enough is a rather low bar. And yet even the best efforts are far from that.
From the blog post:
Jumped right to the end to read the conclusion to see if this long article would be worth the time. In the summary, the author puts forth 4 points.
There's really not a lot to deal with here. Yes of course, judging is complicated, there's a reason that we treat judges with respect - a lot of schooling and work goes into becoming a judge, and being a judge on a higher court generally comes with a ton of experience. Judging has always been political, and there isn't a country out there where it isn't, because what is political is constantly expanding. I personally don't think there's any issue with judges being political, but I do agree that politics in judging should be minimized on certain axes. Being humanistic, for example, is political. Choosing to value all human rights, is also political. I think it's okay for folks to center these values and in fact I would prefer a system where they do.
On item 3 I'm not sure many people would disagree. We'd all like to see experts be consistent, as it would reflect a grounded system of reasoning. We might expect certain issues to drift over time as their politics change, such as becoming more humanistic as a sum of their experiences with humans, but there's really nothing we can do about this except teach people to critically evaluate themselves. The final point, however, is laughable. We already have multiple court cases about AI hallucinating law. Using an AI to assist judges is rife with potential problems. Suggesting that we use AI to enhance judges ignores all the problems we have with AI. Of particular note, the idea that a machine trained on human data would be less biased than a human is a game of statistical chance - it will certainly be less biased than some humans, but because it is trained on the sum of human bias, it will also be more biased than some humans. Should we not strive to ensure that judges get training to reduce their bias, firmly placing the majority of them on the less biased side? Also, who's to say the very judges the author is criticizing is not already using AI to refine their arguments?
I think of particular note here is the very real risk of biased judges choosing biased AI to reinforce their beliefs. I'm reminded of an image I stumbled upon a few days ago, where someone reasonably easily got the "uncensored and unbiased AI" AI of gab to reveal its prompting instructions. Any discussion which talks about opening the door for judges to use AI to assist in their decisions must include a discussion about how to stop against bad actors like this because there will be people who sell products which exist to increase bias. The judicial system has a history of political wars happening for hundreds of years, with the supreme court being a popular battlefield to gain and remove rights. What's to stop a company from engineering an AI to provide a biased response, through the use of instructions such as the above or being even more clever and deciding which pieces of law or interpretation of law the AI gets trained on?
Yes, I’m quite dubious of the AI recommendation. I thought the article was interesting for showing how deep the rabbit hole goes, even for a simple question.
It seems to me that the difficulty comes from wanting consistency, because there are so many things that a ruling could be consistent with in so many ways. Considered as a standalone policy question, it seems simpler: how much evidence should we require for punishing someone as a drug mule? Is finding a large amount of fentanyl in the car enough evidence, or is more needed to avoid punishing unwitting drug mules? Even if that possibility is rare, it should be considered.
A legislature considering the policy question should probably take into account how drug smuggling currently works, with better evidence than one expert’s opinion. And then, also take into account how criminals respond to incentives, and that’s going to be a guess.
It seems like the court system isn’t a great place to have this policy debate, because everything becomes a consistency argument?
I think people view objectivity as necessary for consistency, but I'm not so sure that's true. Someone who is afraid of snakes will pretty consistently remain afraid of snakes and act in a manner that someone afraid of snakes would act. In the realm of objective science, objectivity is a necessary part of remaining consistent, but in the realm of policy this isn't necessarily true. I think we should strive to have policy and law be very specific about what we consider should be punished versus that which should not (of note, the word punish implies a punitive justice system but I am using it more broadly here to indicate something we have decided is incorrect behavior and the response to said behavior which hopefully involves restoration and education) . I also think we should design a system in which policy makers must include experts in the field and listen to their advice when setting policy. We also should strive to train policy makers, experts in the field, and judicial staff in critical thinking so that they can properly examine issues in ways which minimize bias.
The reality, however, is that we have not designed a system like this and we need to consider how we can work best with the system we have, and how we can strive to improve the system. Bandages like AI to solve an immediate problem which are just as likely to be hijacked as the current system (perhaps more likely, given their complete lack of regulation and how little humanity currently understands it) are not reasonable solutions. The article could have just as easily focused on all the ways in which the system is currently broken - the places in which it is not regulated and the places in which money interferes with the desires of the people are perhaps the two most salient existing attack vectors. Ultimately how a judge decides a case matters very little if the laws themselves already reflect a broken system - it's just one more space in which reasoning and human nature can continue to make the system inconsistent.
In short, I agree that the court system isn't a great place to have this debate, but ultimately none of the existing systems are great places for this inconsistency to be created and I don't think there's anything special about the court system which warrants this particular attention or an attempt to fix things. It seems more like a red herring to me, to highlight that judges can be inconsistent when they happen to be much more isolated from the corrupting influence of capital than the legislators themselves. Both good and bad has come from the ability for judges to interpret law and we cannot be narrow sighted by focusing only on the bad and fail to adequately acknowledge the good. For what it's worth, I'm not sure that more bad or more good has come from the ability for judges to interpret law, but our system is designed rather uniquely in that we give the text of law much more precedence than the spirit of law. Systems of law which do the opposite tend to have more consistent results (although I'm sure many will argue they are less consistent) because they tend to design their judicial system with safeguards around the additional leeway they give to judges which do a lot of interpretation in order to ensure that the spirit is appropriately upheld
Yes, there is no good answer here. But one side-effect of judges making policy decisions under the current system is that they don't necessarily admit to doing so - they can justify a decision to be based on precedent with a pretty solid argument, but actually have a lot of freedom to be "results-oriented."
I'm not sure that being results-oriented is bad when a legislature is dysfunctional, though? Or in general. There's something to be said for making sure the results of a judgement aren't unjust. But, it invites skepticism about whether a judge's stated reasons are their real reasons.
I read another blog post by the same author where he wrote about being somewhat too results-oriented. It's long and I didn't follow the first part about what are apparently some rather dubious rules about Louisiana redistricting. Skipping down to The lions (no longer) sleep tonight:
...
So it does seem like being "results-oriented" is a tactic that can be used for opposing purposes, and can go too far.
Yeah and in extreme cases they can claim it's the proper interpretation of the constitution, focusing on a few specific words and disregarding others. There's no way to stop this from happening - people will always find inventive and creative ways to exploit the system. The key, I believe, is strong safeguards. Regulation to prevent the influence of capital, a strong educational system to instill critical thinking, designing the system to be responsible to the population or outcomes of their judgements, and designing the system itself to be resilient such as with panels of judges as opposed to individuals.