17 votes

California abolishes cash bail, replacing with algorithmic based risk assessment

14 comments

  1. [12]
    arghdos
    Link
    What could possibly go wrong?

    People charged with certain misdemeanors would be released within 12 hours of booking without seeing a judge, while the rest would undergo what’s called a “risk assessment,” using an algorithm that would consider a number of factors about the person’s history.

    What could possibly go wrong?

    13 votes
    1. [11]
      Jedi_Tounges
      Link Parent
      Is this algorithm going to be biased like a lot of others. A few examples: A soap dispenser wouldn't dispense soap because it could not recognize non-Caucasians as humans A "racist" google beauty...

      Is this algorithm going to be biased like a lot of others. A few examples:

      A soap dispenser wouldn't dispense soap because it could not recognize non-Caucasians as humans
      A "racist" google beauty prediction AI
      and so on?

      How to mitigate this bias?

      10 votes
      1. teaearlgraycold
        Link Parent
        Don't use opaque algorithms. If you're just running data through a neural net you can't understand what's going on. A human should be able to understand and rationalize any decision made by the...

        How to mitigate this bias?

        Don't use opaque algorithms. If you're just running data through a neural net you can't understand what's going on. A human should be able to understand and rationalize any decision made by the algorithm. You still need testing and strict review of procedure, but there should be less surprises. Better yet, when something goes wrong you'll probably understand why.

        8 votes
      2. [2]
        Litmus2336
        Link Parent
        Hire more diverse programmers, actually test your code with all types of people. I can go into a long winded discussion about machine learning algorithms, and biased testing, but the general gist...

        Hire more diverse programmers, actually test your code with all types of people.

        I can go into a long winded discussion about machine learning algorithms, and biased testing, but the general gist is when a bunch of white & asian people (who are the majority of programmers) develop the machine learning datasets and testing sets for a program (likely reflecting their internal or visible biases) then you get algorithms that reflect the biases of their creators.

        4 votes
        1. Tenar
          Link Parent
          testing with such large groups is notoriously difficult though. I mean, there's an example of certain types of medicine that went through all FDA phases of trial (and in the end that means...

          testing with such large groups is notoriously difficult though. I mean, there's an example of certain types of medicine that went through all FDA phases of trial (and in the end that means thousands upon thousands of people) but then was released and found to have very serious adverse effects in certain chinese people. I think it was painkillers, causing serious heart damage in a small subsection (one in thousands) of a chinese subgroup.

          To be able to test that you've gotta use millions of people. It's not serious for almost anyone, but then it's in production and you find it kills certain people.

          Obviously AI-based recognition isn't gonna be that harmful. Until applied to real world stuff like this, where even the makers have no idea what biases or thought patterns are implicit in the software. And it isn't just mislabeling someone, it could be structurally discriminating against whoever, without anyone knowing, or even being able to know. Not even if you test with larger groups, because the effect might only be done on a small enough subsection that even thorough testing wouldn't show anything happening.

          3 votes
      3. [6]
        Neverland
        Link Parent
        Better training data is the only way, right? I imagine that in this case they could use historical data to test and train their algorithms, while hopefully leaving off markers like race.

        Better training data is the only way, right? I imagine that in this case they could use historical data to test and train their algorithms, while hopefully leaving off markers like race.

        2 votes
        1. [5]
          arghdos
          Link Parent
          I think the real risk is that racial (or importantly, economic) bias is inherent in the other inputs that these risk assessments use. For instance, the propublica study linked above explicitly...

          I think the real risk is that racial (or importantly, economic) bias is inherent in the other inputs that these risk assessments use. For instance, the propublica study linked above explicitly states that race was not a consideration in the Northpointe risk assessment algorithm:

          Northpointe’s core product is a set of scores derived from 137 questions that are either answered by defendants or pulled from criminal records. Race is not one of the questions

          However, there are a number of questions that could lead to biases:

          Was one of your parents ever sent to jail or prison?

          How many of your friends/acquaintances are taking drugs illegally?

          How often did you get in fights while at school?

          A hungry person has a right to steal

          (see the whole list of questions here)

          Are all questions I could see being biased against certain minorities or economic groups. To take a simple example, it wouldn't be unreasonable to me if people who actually deal with hunger on a recurring basis were more likely to answer that hungry people have the right to steal food.

          5 votes
          1. [4]
            Neverland
            (edited )
            Link Parent
            Well at some point, the algo is going to have to make decisions based on something, and if you follow the subsets up, eventually they are likely to lead to a larger sets which are likely to lead...

            Well at some point, the algo is going to have to make decisions based on something, and if you follow the subsets up, eventually they are likely to lead to a larger sets which are likely to lead to racial and socioeconomic divisions. I mean that's just how this all works, right? I am not defending this, just stating what seems to be reality.

            One thing though, a survey seems like a great way to get compromised data into the system.

            A hungry person has a right to steal

            Who is going to answer that question, posed by law enforcement, in an honest way if they believe that "yes, a hungry person has a right to steal?" I don't understand why the "user" has any personal input. People lie.

            What do you think? Should algorithms be used in situations like this, or any other for that matter?

            edit: And what parameters and signals are humans using in this case currently? Is a somewhat racist algo better than humans if it shown to be at least a bit less racist and better at predicting bail jumpers?

            Edit2: spelling

            4 votes
            1. [2]
              arghdos
              Link Parent
              I tried to be rather cautious in my examples here -- clearly this is the crux of the matter, how do you account for the multitude of ways that racial & socioeconomic status influence the input...

              Well at some point, the algo is going to have to make decisions based on something, and if you follow the subsets up, eventually they are likely to lead to a larger sets which are likely to lead to racial and socioeconomic divisions. I meant that's just how this all works, right? I am not defending this, just stating what seems to be reality.

              I tried to be rather cautious in my examples here -- clearly this is the crux of the matter, how do you account for the multitude of ways that racial & socioeconomic status influence the input data into a system like this such that:

              a) it remains useful, and
              b) it is not perpetuating disadvantages on already marginalized groups

              You should take a look at the EFF article @Deimos posted, it goes way deeper into what sorts of data should be used, and the ethics of building such a system.

              Who is going to answer that question, posed by law enforcement, in an honest way if they believe that "yes, a hungry person has a right to steal?" I don't understand why the "user" has any personal input. People lie.

              Yeah, I have to say I don't really understand why they'd ask people questions like this. Some of the information (e.g., the question about parents going to jail) is probably pulled automatically but I don't see the point of posing basic philosophy questions to possible felons other than to get like, the 0.00001% not smart enough to lie about their belief's.

              What do you think? Should algorithms be used in situations like this, or any other for that matter?

              It's possible (though, in my opinion unlikely) that there is a way that a for-profit corporation to create this type of algorithm that satisfies (say) the EFF's ethical criteria, but:

              The company [Northpointe] does not publicly disclose the calculations used to arrive at defendants’ risk scores, so it is not possible for either defendants or the public to see what might be driving the disparity. (On Sunday, Northpointe gave ProPublica the basics of its future-crime formula — which includes factors such as education levels, and whether a defendant has a job. It did not share the specific calculations, which it said are proprietary.)

              does not instill much confidence that the current players are attempting to do so.

              4 votes
              1. Neverland
                Link Parent
                I agree with that completely. This is the biggest problem I see with privately owned algorithms that make decisions in the public forum. There is zero accountability, the logic is “proprietary”...

                The company [Northpointe] does not publicly disclose the calculations used to arrive at defendants’ risk scores, so it is not possible for either defendants or the public to see what might be driving the disparity.

                does not instill much confidence that the current players are attempting to do so.

                I agree with that completely. This is the biggest problem I see with privately owned algorithms that make decisions in the public forum. There is zero accountability, the logic is “proprietary” and cannot be shared. IMO, this is why any algorithms adopted by the government should be open source.

                We need to come up with a law regarding this... looking at you EU, you’re our only hope.

                1 vote
            2. toobsock
              Link Parent
              If the algorithm has been trained on a huge dataset of inmates, and is incredibly accurate in predicting recidivism rates using a test set of historical cases, then absolutely. This seems easy...

              If the algorithm has been trained on a huge dataset of inmates, and is incredibly accurate in predicting recidivism rates using a test set of historical cases, then absolutely. This seems easy enough to answer. Just compare the results to the accuracy of the humans.

              1 vote
  2. [2]
    Comment deleted by author
    Link
    1. arghdos
      Link Parent
      Ahh, thanks. Didn't catch that one yesterday.

      Ahh, thanks. Didn't catch that one yesterday.

  3. teaearlgraycold
    Link
    From a utilitarian perspective you could consider this change as a way to simply reduce the harm done by interim incarceration. Instead of the process being "You're poor? You're in jail." (and I...

    From a utilitarian perspective you could consider this change as a way to simply reduce the harm done by interim incarceration. Instead of the process being "You're poor? You're in jail." (and I assume poorer people are much more likely to be criminally charged) it should be based on something else.

    People will get fucked over by the system either way. But we can probably do better than just jailing all of the poor people.

    3 votes