43 votes

She didn’t get an apartment because of an AI-generated score – and sued to help others avoid the same fate

67 comments

  1. [56]
    Wes
    Link
    I don't want to sound unsympathetic, but falling behind on credit card bills is a strong marker for a risky tennant. I can understand a landlord feeling uneasy about that. Having a strong...

    "But the AI doesn’t know my behavior – it knew I fell behind on paying my credit card but it didn’t know I always pay my rent."

    I don't want to sound unsympathetic, but falling behind on credit card bills is a strong marker for a risky tennant. I can understand a landlord feeling uneasy about that. Having a strong reference may help, but these problems often compound, especially when CC debt is concerned.

    It mentions her son having a high credit score, but that doesn't really matter unless his name is on the lease, too. Cosigning would likely have increased their chances of being approved. People with poor credit scores often need a cosigner to convince a lender/landlord that they're lower risk.

    It's not an easy circumstance, and I don't especially like the concept of credit scores or other risk-based algorithms being used at all. I don't really judge landlords for wanting to use them either though, since accepting a tenant that can't pay is an extremely difficult situation to get out of. This was especially true during the pandemic, when these events took place.

    As a side note, this story really doesn't seem to have anything to do with AI at all. SafeRent used an in-house algorithm which rejected her based on her low credit score. Their product page doesn't mention AI anywhere, and there's no indication they use machine learning or other AI technologies in their score. If publications continue to arbitrarily insert the term AI for clicks, it's just going to mislead people about what artificial intelligence even is and what it can do.

    45 votes
    1. [33]
      unkz
      Link Parent
      Having done credit risk scoring for many years, I struggle to think of a viable alternative. In every instance where I have seen a company operating based on humans simply eyeballing applications,...
      • Exemplary

      It's not an easy circumstance, and I don't especially like the concept of credit scores or other risk-based algorithms being used at all.

      Having done credit risk scoring for many years, I struggle to think of a viable alternative. In every instance where I have seen a company operating based on humans simply eyeballing applications, introducing automated scoring has in every single case improved their default rates.

      26 votes
      1. [32]
        Minori
        Link Parent
        People forget the system before credit scores was that the lender would simply reject people they didn't know or filter exclusively by income. Credit scores are more equitable than what came...

        People forget the system before credit scores was that the lender would simply reject people they didn't know or filter exclusively by income. Credit scores are more equitable than what came before.

        Credit scores are calculated from a formula based on credit history. There's no race, class, or zip code parameter in the formula.

        20 votes
        1. [31]
          dangeresque
          Link Parent
          That in itself introduces bias against those who don't have ready access to credit, banking, etc., to establish a credit score in the first place. I'm not here to claim that the system is bad, nor...

          There's no race, class, or zip code parameter in the formula.

          That in itself introduces bias against those who don't have ready access to credit, banking, etc., to establish a credit score in the first place. I'm not here to claim that the system is bad, nor do I have any alternative suggestions, but I just want to say that it's misleading to imply or allow people to believe that the absence of race, class, or zip code parameters alleviates any concerns of bias around those qualities. Somebody could be the cleanest, quietest, most reliable renter on the planet, but if they have no bank account and they pay rent in cash to some slumlord in the ghetto because that's where and how they grew up, they won't have enough of a credit history to get a nicer place even if they can afford it.

          As an aside, that's exactly the sort of societal effect that critical race theory sets out to study.

          16 votes
          1. [26]
            unkz
            Link Parent
            How many unbanked people do you deal with on a regular basis? Because clean, quiet, and reliable are not typically appropriate adjectives. I'm not sure "bias" is the right word to use here in...

            Somebody could be the cleanest, quietest, most reliable renter on the planet, but if they have no bank account

            How many unbanked people do you deal with on a regular basis? Because clean, quiet, and reliable are not typically appropriate adjectives. I'm not sure "bias" is the right word to use here in terms of credit risk. It's more like, a highly predictive variable.

            Something people do not in general understand about the credit industry is that lenders really do not want to introduce bias into our models. If there is something in our model that is preventing us from offering credit to someone who is going to pay us back, that's a really big problem. We spend a very large amount of time and money trying to find ways to justify giving out more money to people. I've worked with some of the major bureaus to develop alternative credit models specifically to find ways to eliminate bias from models to more effectively lend to the unbanked, unemployed, retired, and disabled borrowing sectors. Anything we can find to help separate a signal into its components so we can identify how much of it is just telling us a non-risk factor like gender or race and how much is actual credit-worthiness is like striking gold.

            6 votes
            1. [9]
              DefinitelyNotAFae
              Link Parent
              We're going with "dirty and loud" now for unbanked folks? I can understand "unreliable", that is more a function of their current experiences than anything. This bias is why people can't ever get...

              How many unbanked people do you deal with on a regular basis? Because clean, quiet, and reliable are not typically appropriate adjectives. I'm not sure "bias" is the right word to use here in terms of credit risk. It's more like, a highly predictive variable.

              We're going with "dirty and loud" now for unbanked folks? I can understand "unreliable", that is more a function of their current experiences than anything.

              This bias is why people can't ever get on their feet after incarceration or eviction, etc. Because they're not clean, quiet, or reliable.

              And I do understand lenders as a whole* don't "want" bias, but they still have it.

              *Some would be quite fine charging certain groups more, especially if they don't have other options. Even lenders can be irrational in their bigotry.

              4 votes
              1. [8]
                unkz
                Link Parent
                I mean yeah. There is this tendency to romanticize poverty, but I’ve spent time in the trenches in cheque cashing and temporary labour. I was born in the poorest postal code in Canada (V6A). I’m...

                We're going with "dirty and loud" now for unbanked folks? I can understand "unreliable", that is more a function of their current experiences than anything.

                I mean yeah. There is this tendency to romanticize poverty, but I’ve spent time in the trenches in cheque cashing and temporary labour. I was born in the poorest postal code in Canada (V6A). I’m not basing this on some abstract idea of who these people are, I am these people and we are a mess.

                6 votes
                1. [7]
                  DefinitelyNotAFae
                  Link Parent
                  I'm not romanticizing anything, but I think unbanked/poor = loud and dirty (and yeah even inherently unreliable) is shitty with racist undertones.

                  I'm not romanticizing anything, but I think unbanked/poor = loud and dirty (and yeah even inherently unreliable) is shitty with racist undertones.

                  2 votes
                  1. [4]
                    unkz
                    Link Parent
                    It’s almost tautological that poor people are bad with money. There’s no reason to bring claims of veiled racism into this.

                    It’s almost tautological that poor people are bad with money. There’s no reason to bring claims of veiled racism into this.

                    6 votes
                    1. [3]
                      DefinitelyNotAFae
                      Link Parent
                      I'm not making veiled claims of racism, I'm saying that calling poor people dirty and loud has racist undertones. Outside of lacking the facilities to clean oneself, neither are related to one's...

                      It’s almost tautological that poor people are bad with money. There’s no reason to bring claims of veiled racism into this.

                      I'm not making veiled claims of racism, I'm saying that calling poor people dirty and loud has racist undertones. Outside of lacking the facilities to clean oneself, neither are related to one's skill with money.

                      2 votes
                      1. [2]
                        unkz
                        Link Parent
                        This isn't financial wizardry, this is managing to have a bank account. We're looking at the bottom 4.2% of the population in terms of "skill with money" if you can call being actively self...

                        This isn't financial wizardry, this is managing to have a bank account. We're looking at the bottom 4.2% of the population in terms of "skill with money" if you can call being actively self destructive a "skill". At that end of the scale, there are what I would basically describe as comorbidities -- these people have a lot of other problems in their life.

                        I realize this isn't a pretty picture, but I think spending a few days down at a cheque cashing storefront actually dealing with these people would be highly educational.

                        2 votes
                        1. DefinitelyNotAFae
                          Link Parent
                          I have plenty of relevant work experience. And you're right, there are a lot of other problems that go along with it, which is why I can grant "reliable" as a side effect of all of those other...

                          I have plenty of relevant work experience. And you're right, there are a lot of other problems that go along with it, which is why I can grant "reliable" as a side effect of all of those other things.

                          I literally just worked with someone whose job loss and relatively brief gap in employment led to her missing rent, having an eviction process start, and despite her being employed now, it took legal aid and assistance to get back on top of rent, catching up on back payments. And it just happened that I knew where to send her. Her tires were bald, her car has been hit and she can't fix it.

                          I don't know her entire life, and she was at least briefly unbanked - I think she was able to re-open an account thankfully but the paycheck delay was yet another issue. She's got a kid. Housing locally costs far too much for her to build up a rainy day fund. She's doing the best she can, and it wasn't enough without that help. There but for the grace of God having a family who could and would help me out, go I.

                          And we haven't even gotten into ChexSystems and the whole separate ways you can be blacklisted from regular banking even if you have good credit otherwise.

                          I've worked with people on parole, I've volunteered at food banks, I've worked with people who are seriously mentally ill and struggling to make it on disability. I've worked for a bank and in retail and restaurant service. I have lived in apartment complexes and cities where I've been approached by panhandlers looking for rides or money regularly. They weren't all great places to live quality wise. Is that enough "education?"

                          Your lived experiences are yours, they're valid. But calling 5.6 million American households loud, dirty, and unreliable is stereotyping to an extreme. In Canada that's 2-3% rather than 4.2%, but an estimated 15 million are underbanked.

                          The stereotypes of poor people are directly tied to race and class and used to exclude poor people from "polite" society. Whether that is your intent or not, it plays into the exact same stereotypes. And then, people assume that despite their voucher or income or whatever it is, they shouldn't rent to the poor, because they're dirty and loud and unreliable. After all, they're poor and clearly they're to blame for their situation due to their laziness. It's shitty and dirty/loud/unreliable along with lazy and violent are absolutely used against Black people, immigrants and other minority groups as much as they are against poor people. And insisting that no, this group, this stereotype is correct? No thank you.

                          2 votes
                  2. [2]
                    Minori
                    Link Parent
                    If that's where u/unkz grew up, and that's how they describe it. It seems a bit much to police the language they use to describe where they're from.

                    If that's where u/unkz grew up, and that's how they describe it. It seems a bit much to police the language they use to describe where they're from.

                    5 votes
                    1. DefinitelyNotAFae
                      (edited )
                      Link Parent
                      They can describe themselves in whatever way they'd like. What they chose to describe here was "the unbanked" not just their neighborhood growing up. I cannot think of a minority group it would be...

                      They can describe themselves in whatever way they'd like. What they chose to describe here was "the unbanked" not just their neighborhood growing up.

                      I cannot think of a minority group it would be acceptable to call the opposite of "clean, quiet and reliable" regardless of one's membership on it.

                      3 votes
            2. [12]
              DrEvergreen
              Link Parent
              Edited to add: Credit scores only tell banks how good you are at maintaining a level of debt. Because that means they will earn fees off of you. Paying off loans, and not having credit cards...

              Edited to add:

              Credit scores only tell banks how good you are at maintaining a level of debt. Because that means they will earn fees off of you. Paying off loans, and not having credit cards (which are also borrowed money until paid back) will see your score drop. Credit scores are only good for what is literally the name: how good you are at maintaining a credit. I.E. keeping a level of debt.

              If you avoid debt to begin with, you have a shitty score. Despite potentially being much better at financial planning than someone that relies on keeping a debt to "build a credit score".


              My ex. Who was not set up by his mum (this is the UK) early in life, turned 18 and was still living in the world of cash. Mum not renewing his childhood passport. And then he became an adult and was out of college and...

              One of his exes put him on a bank account of hers so he could have a card to use. Teenagers as they were they didn't think much of this. When they split, they talked with the bank and got that account split off to him alone. He lived with girlfriends a couple of times, but other than that lived with his family at their rather large place. Not wealthy, just something bought back before prices exploded.

              Then at 30-something was suddenly out on his own when the family couldn't afford the place anymore. At least a proper bank acct, other than that, nothing to his name. No passport for the last 20 years, no tenancies to his name, nothing.

              Despite having worked his entire adult life, but mostly for family or on the farm of an ex, or in an ex's shop etc. Phone number from before they started registering those to the degree they do today.

              He didn't exist as far as credit score goes, really. Was quite the process when he wanted a passport to go abroad with me.

              I'd much prefer the way we do it in my country. You get marks against you for failed payments that go to collections, and then those marks stay for 3 years beyond being fully paid off if they went to collection via a bailiff. But sometimes those are struck off as well when paid. Depends on type of claim.

              You are marked as a non-payer if you have bills that have not been paid, only.

              A phone bill from your younger years wouldn't scare a bank that sees you haven't miskept anything major.

              2 votes
              1. [4]
                unkz
                Link Parent
                I think this kind of misses the mark on how credit scores intersect with creditworthiness. Creditworthiness is intrinsically bound up with financial literacy, long term planning, decision making,...

                I think this kind of misses the mark on how credit scores intersect with creditworthiness.

                Creditworthiness is intrinsically bound up with financial literacy, long term planning, decision making, and the ability to accurately assess risk.

                You and I see totally different things when we look at your ex. I think you see someone who is employed but naive who is not an immoral person and thus should be entitled to credit.

                I see someone who is financially illiterate with a history of making very bad financial decisions.

                If you avoid debt to begin with, you have a shitty score. Despite potentially being much better at financial planning than someone that relies on keeping a debt to "build a credit score".

                There’s no “potential” about this — anyone who fails to build a credit score is by definition bad at financial planning. Building a credit score is part and parcel of financial planning, and not knowing that is financial illiteracy.

                8 votes
                1. [3]
                  DrEvergreen
                  Link Parent
                  It's a fundamental difference in world view. Paying rent is not credit. I've always chosen to pay ahead of the month, meaning no later than the last day of the month before the payment is...

                  It's a fundamental difference in world view. Paying rent is not credit. I've always chosen to pay ahead of the month, meaning no later than the last day of the month before the payment is covering. So Jan 31 for the month of Feb.

                  No incurred credit/debt/temporary loan incurred at all.

                  1. [2]
                    unkz
                    Link Parent
                    At least in my area, extending an offer of rent is in fact a form of credit. Once a person has a rental agreement, they are unevictable for at least 6 months and more likely a year -- there's a...

                    At least in my area, extending an offer of rent is in fact a form of credit. Once a person has a rental agreement, they are unevictable for at least 6 months and more likely a year -- there's a lot of risk in renting to an unknown quantity.

                    This kind of story is not uncommon: Windsor landlord says tenants who haven't paid 16 months of rent have now paused eviction order

                    3 votes
                    1. DrEvergreen
                      Link Parent
                      Here in Norway the maximum such issues can be stretched without proof of an actual issue with the place (structural, lacks according to what the contract states etc) is 3 months. And even then,...

                      Here in Norway the maximum such issues can be stretched without proof of an actual issue with the place (structural, lacks according to what the contract states etc) is 3 months.

                      And even then, rent has to be kept aside and preferably in a specific type of acct one can open with a bank, until the matter is settled and you either get a reduction in rent (still have to pay the rest at that point), or not (have to pay all of it).

                      This is different from the deposit account btw. Which is also legally required, a specific type of acct. in the renter's name, but where neither party can take the money without proof of rent agreement having come to an end, or a court order due to an issue like described above. It is illegal to not put deposits in a specific account like that. Lots of people dont, but it is illegal and the landlord will be then be liable to pay an 8.5% interest rate on the original deposit whenever the tenancy ends, for whatever reason.

                      1 vote
              2. [4]
                stu2b50
                Link Parent
                Given these three drivers, who would you trust to drive you more? Someone who has driven for over 100,000 miles without a traffic accident Someone who has driven for over 100,000 miles with 300...

                Given these three drivers, who would you trust to drive you more?

                1. Someone who has driven for over 100,000 miles without a traffic accident

                2. Someone who has driven for over 100,000 miles with 300 traffic accidents

                3. Someone who has picked up their driver's license but otherwise never driven more than 100 miles.

                Personally, I'd go with 1 > 3> 2. And it's the same with credit.

                It's not some invention of the credit score, it's just factual that you'd rather have someone with a history of paying back their loans over someone that doesn't. Of course, not having a credit score is still better than having a bad one, with a history of delinquency.

                If credit scores didn't exist, banks would still prefer people who can prove that they took loans and paid them back. It's not about keeping a balance - that won't help your credit at all.

                6 votes
                1. [3]
                  DrEvergreen
                  Link Parent
                  In my eyes, 1 and 3 show the same: A lack of issues that the insurance company will have to pay for. 2 shows that you incur extra expenses all the time. To me, the entire notion of a credit score...

                  Someone who has driven for over 100,000 miles without a traffic accident

                  Someone who has driven for over 100,000 miles with 300 traffic accidents

                  Someone who has picked up their driver's license but otherwise never driven more than 100 miles.

                  In my eyes, 1 and 3 show the same: A lack of issues that the insurance company will have to pay for. 2 shows that you incur extra expenses all the time.

                  To me, the entire notion of a credit score is a variant of your second suggestion. Someone that keeps getting in accidents, but is really good at paying for the repairs.

                  In my country (Norway) it really is more about showing that you've never mismanaged you finances to begin with. That with todays income and predicted future, you will be able to continue managing your loan even throughout ups and downs in the future. A lump sum/downpayment and/or co-signers for that sum is a part of that assessment. As a way to prove that you are able to save or have someone that is able to save and thus can manage.

                  I know some people look up their credit scores with companies that offer such services, but it is really not something most people ever do, nor need to. Not even for things like mortgages for homes or cars.

                  Our "credit checks" are agains the national registry where they keep track of mismanaged bills.

                  It really is an entirely different way of seeing it.

                  1. stu2b50
                    Link Parent
                    The difference is that 3 doesn’t have experience. So there an unknown. That’s better than being known for being shit, but worse than knowing for being good. Or take a restaurant - which seems more...

                    The difference is that 3 doesn’t have experience. So there an unknown. That’s better than being known for being shit, but worse than knowing for being good.

                    Or take a restaurant - which seems more appealing, one with a 4.6 star average with 500k reviews, one with no reviews, or one with a 1.3 star review with 500k reviews?

                    To me, the entire notion of a credit score is a variant of your second suggestion. Someone that keeps getting in accidents, but is really good at paying for the repairs.

                    Taking out credit isn’t a bad thing. Again, it’s a myth that paying off your credit card will cause your credit score to go down. There’s little reason in the US not to put all your expenses on a credit card and to pay it off at the end of the month. Otherwise, you’re just paying for other people’s credit card rewards if you use cash or debit.

                    There’s a lot of anecdotes on the internet, but that’s because there’s a lot of stupid idiots on the internet.

                    6 votes
                  2. unkz
                    Link Parent
                    But they're obviously not the same thing. This is a major factor in why young drivers pay higher insurance -- because insurance companies don't yet know what they're getting into with a contract...

                    In my eyes, 1 and 3 show the same: A lack of issues that the insurance company will have to pay for. 2 shows that you incur extra expenses all the time.

                    But they're obviously not the same thing. This is a major factor in why young drivers pay higher insurance -- because insurance companies don't yet know what they're getting into with a contract with young drivers.

                    5 votes
              3. [3]
                Minori
                (edited )
                Link Parent
                Look, this is just flat out incorrect; you're repeating a common financial myth. For credit cards, all that matters is keeping the account open (referred to as "having an open line of credit")....

                Credit scores only tell banks how good you are at maintaining a level of debt. Because that means they will earn fees off of you. Paying off loans, and not having credit cards (which are also borrowed money until paid back) will see your score drop. Credit scores are only good for what is literally the name: how good you are at maintaining a credit. I.E. keeping a level of debt.

                Look, this is just flat out incorrect; you're repeating a common financial myth. For credit cards, all that matters is keeping the account open (referred to as "having an open line of credit"). All you have to do is buy a $5 gift card once a year to keep a credit card open to improve your credit score.

                Lenders want to know if you'll repay the loans they give you. It's purely a measure of "if I give you money, will I get it back?" which is the same thing I assume most humans do with people that ask them for money.

                Credit cards in particular make huge amounts of money off interchange fees which they charge merchants. Banks don't look for high credit scores purely as a proxy for wealth. It's completely legal to discriminate by wealth, and they largely don't do it. They use credit scores instead because they're looking for people that'll always pay off their loans because that's guaranteed income via licensing, merchant fees, etc.

                6 votes
                1. [2]
                  DrEvergreen
                  Link Parent
                  Even this is specifically showing them how good you are at taking on, and then managing debt. Whereas where I live, it's the exact opposite. You are assumed to be okay until you prove you can't...

                  Look, this is just flat out incorrect; you're repeating a common financial myth. For credit cards, all that matters is keeping the account open (referred to as "having an open line of credit"). All you have to do is buy a $5 gift card once a year to keep a credit card open to improve your credit score.

                  Lenders want to know if you're repay the debt they offer you. It's purely a measure of "if I give you money, will I get it back?" which is the same thing I assume most humans do with people that ask them for money.

                  Even this is specifically showing them how good you are at taking on, and then managing debt.

                  Whereas where I live, it's the exact opposite. You are assumed to be okay until you prove you can't manage your bills. Income and assets, a lump sum and/or someone co-signing, plus a lack of registered cases of failure to pay is what they look for.

                  Your final paragraph seems to me to say you kind of agree with me, just seeing it from the other side? That it is in fact about seeing who is good at staying in debt and managing it.

                  Plenty of people have seen their credit scores drop when they've paid off studen loans or stopped keeping a credit card.

                  Even if it is only a nominal sum, it is still proving that you're good at staying a little bit in debt, but not too much.

                  Not incurring debt of any kind to begin with seems much better to me. Showing that you never borrow from tomorrow to pay stupid silly things today.

                  1 vote
                  1. Minori
                    Link Parent
                    No, I'm saying it's about paying off lines of debt, not staying in debt. Credit cards are unique because they're revolving lines of credit which are effectively recalculated every month. That's...

                    That it is in fact about seeing who is good at staying in debt and managing it.

                    No, I'm saying it's about paying off lines of debt, not staying in debt. Credit cards are unique because they're revolving lines of credit which are effectively recalculated every month. That's why having an open credit card with a $0 balance is a good thing for your credit score. Staying in debt is actually bad for credit score because the lower the amount of credit you're using, the higher your score is. Debt burden is actually the second most important factor in credit score.

                    Plenty of people have seen their credit scores drop when they've paid off studen loans or stopped keeping a credit card.

                    Yes, that's expected for credit scores. This happens because they no longer have an open line of credit with a long payment history. Fully paying off a loan is not the same as consistently paying a loan every month. Even if drops their score, it won't suddenly give them a bad score unless they were extremely borderline to begin with.

                    Not incurring debt of any kind to begin with seems much better to me. Showing that you never borrow from tomorrow to pay stupid silly things today.

                    Let's say I live in Turkey where annual inflation is 50% and someone offers me a ₺600,000 loan with a 4% APR. I'd be an idiot not to take the loan because inflation is higher than the interest rate which means I can make free money by taking the loan then slowly paying it back. Some people have poor self control, but that doesn't make debt a bad tool. Not being able to use debt effectively is a sign of poor financial literacy.

                    3 votes
            3. [4]
              dangeresque
              Link Parent
              Your kneejerk defensiveness simply makes my point for me. If you're unwilling to acknowledge that there might be blind spots in your models and resort to generalization of unbanked people as...

              Your kneejerk defensiveness simply makes my point for me. If you're unwilling to acknowledge that there might be blind spots in your models and resort to generalization of unbanked people as dirty, loud, and unreliable, then I don't see a point in engaging with you further.

              1 vote
              1. [3]
                Minori
                Link Parent
                This feels uncharitable when the comment literally said:

                If you're unwilling to acknowledge that there might be blind spots in your models

                This feels uncharitable when the comment literally said:

                I've worked with some of the major bureaus to develop alternative credit models specifically to find ways to eliminate bias from models to more effectively lend to the unbanked, unemployed, retired, and disabled borrowing sectors.

                2 votes
                1. [2]
                  dangeresque
                  Link Parent
                  Honestly I stopped reading carefully after the first paragraph which painted unbanked people with a broad and kind of disgusting brush. How's that for a kneejerk reaction, I guess. It's a damn...

                  Honestly I stopped reading carefully after the first paragraph which painted unbanked people with a broad and kind of disgusting brush. How's that for a kneejerk reaction, I guess.

                  It's a damn shame that we have to discuss credit reporting models at all in a conversation about renting an apartment. Real estate investors want to own all this property, pricing regular people out of the ownership market, and mitigate their risk with these easy models that can give them a yes/no answer to determine whether somebody is worthy of having a roof over their head. They want to make it cold, quick, impersonal, and cheap (and unless prohibited by law, still impose those costs upon applicants) so they can more efficiently hoard millions of units and maintain a revenue stream. Our economy is broken in many more ways than implicit bias in credit scoring systems, and solving for bias in one small aspect of the economy only serves to alleviate a symptom of systemic bias and discrimination in society.

                  I'm frustrated by the overall situation that capitalists have put us in.

                  1 vote
                  1. Minori
                    Link Parent
                    Capitalist developers want to build more housing so they can make more money. It's NIMBY homeowners that have made it impossible to build in California and many other regions of the US. We'll...

                    I'm frustrated by the overall situation that capitalists have put us in.

                    Capitalist developers want to build more housing so they can make more money. It's NIMBY homeowners that have made it impossible to build in California and many other regions of the US.

                    We'll always have these problems as long as there's a supply shortage that allows landlords to be picky about tenants. Housing abundance is the only way out of the current mess in the Anglosphere.

                    1 vote
          2. [4]
            Minori
            Link Parent
            You're right that other parameters could have tight correlations with race, but I do not think credit score systems are intentionally designed to discriminate by race. Much like I don't believe...

            You're right that other parameters could have tight correlations with race, but I do not think credit score systems are intentionally designed to discriminate by race. Much like I don't believe standardized testing is racist, I can't see credit scores as racist. Any apparent racial biases are because of pre-existing inequalities in the system.

            Of course if discriminating by credit score effectively works as a racial proxy, then yeah critical race theory is appropriate. I think it's important to acknowledge how credit scores have improved credit access compared to the previous status quo. There are lessons from history that can help us craft better policies for a more egalitarian society.

            if they have no bank account and they pay rent in cash to some slumlord in the ghetto because that's where and how they grew up, they won't have enough of a credit history to get a nicer place even if they can afford it.

            In those cases, it's common to require a larger security deposit or proof of income statements. I can't say whether this is universally practiced, but it's the default in the places I've lived around the US.

            4 votes
            1. [3]
              DrEvergreen
              Link Parent
              Look into redlining in the US. Credit scores are literally a score for how well you manage your credit. Meaning your level of debt, and how maintained it is. Not how much you avoid it to begin...

              Look into redlining in the US. Credit scores are literally a score for how well you manage your credit. Meaning your level of debt, and how maintained it is. Not how much you avoid it to begin with.

              If you are denied loans for a home, denied credit cards, denied car loans etc, you won't be able to own. Meaning your children won't have you to lean on for help with getting started in life themselves, since most of those things that count for credit scores rely on maintaining debt and being allowed to take on debt with co-signers.

              These types of societal barriers of entry, and when the wealthy are allowed to make their own way of living the standard they measure people up against, is intimately tied to social classes of all kinds.

              3 votes
              1. [2]
                Minori
                Link Parent
                I'm extremely familiar with housing policy in the US and the racist and classist origins of zoning rules and the like. I don't understand how historical redlining relates to credit score in this...

                I'm extremely familiar with housing policy in the US and the racist and classist origins of zoning rules and the like. I don't understand how historical redlining relates to credit score in this context. While there are racial disparities in home ownership rates in the US, there are plenty of ways to build good credit even in poverty.

                I fully acknowledge there are racial gaps in banking access, but it's possible for almost everyone to use a secured credit card with a refundable deposit of $50-200 to build a credit history. Credit score is a good tool that's better than the alternatives. The least biased way to do it is to require a larger deposit from all renters to offset risk, but that's much more expensive for everyone and arguably discriminates more against the poor.

                9 votes
                1. AnthonyB
                  Link Parent
                  Those policies have generational impacts in terms of building wealth. On top of that, bias impacts almost every corner of credit building. Just because it is possible to build credit doesn't mean...

                  I don't understand how historical redlining relates to credit score in this context.

                  Those policies have generational impacts in terms of building wealth. On top of that, bias impacts almost every corner of credit building. Just because it is possible to build credit doesn't mean there aren't systemic disadvantages that perpetuate historical inequality.

                  There is a lot of information out there on this topic that can be found with a quick search.

                  This offers a decent overview but there's also a Wikipedia entry that serves as a good starting point with links to various articles that explore the topic.

                  2 votes
    2. [2]
      boxer_dogs_dance
      Link Parent
      I used the headline that came with the article but the meat of the article is not ai, I agree. However, using an algorithm with no possiblity of review can itself be a problem. The government is...

      I used the headline that came with the article but the meat of the article is not ai, I agree. However, using an algorithm with no possiblity of review can itself be a problem. The government is (at the moment) the most reliable payment source available. The housing voucher should have been accounted for in the decision. It's regular income and is not fungible. It must be spent on housing.

      24 votes
      1. Wes
        Link Parent
        I agree. If the algorithm did not factor in housing vouchers, then it wasn't a very good algorithm. An unfair algorithm is really no better than a biased person making a decision, and it's...

        I agree. If the algorithm did not factor in housing vouchers, then it wasn't a very good algorithm. An unfair algorithm is really no better than a biased person making a decision, and it's sometimes worse as it gives them the ability to deflect responsibility.

        I don't know enough about this space to say if housing vouchers would have been enough to offset the credit card debt or not. However, the ability to override such a decision should always be present. Though arguably that falls to the landlord, and not to the scoring company.

        It sounds like SafeRent has decided to stop offering a score at all when vouchers are involved. While I think it would be better to improve their product, if at all possible, they may have preferred to sidestep the legal concern altogether and simply say "unscorable". I wonder if that will help or harm those in a similar position in the future.

        18 votes
    3. [10]
      updawg
      Link Parent
      Just because it doesn't use deep learning or an LLM doesn't mean it's not AI. There are a lot of things that use AI that people consider just "technology" these days. I don't really fault a...

      Just because it doesn't use deep learning or an LLM doesn't mean it's not AI. There are a lot of things that use AI that people consider just "technology" these days. I don't really fault a newspaper for choosing to use correct wording that is also more engaging just because it's not exactly what people think of.

      16 votes
      1. [9]
        zestier
        Link Parent
        It is unfortunate how poorly "AI" is defined and is getting eroded further. My perception is that marketing people started using "AI" to get laypeople excited about machine learning because people...

        It is unfortunate how poorly "AI" is defined and is getting eroded further. My perception is that marketing people started using "AI" to get laypeople excited about machine learning because people weren't interested in "ML" as a buzzword anymore. That's all to say that we kind of need a line somewhere because calling all algorithms "AI" just doesn't fit within reality. For example, if (creditScore < 650) return "DECLINE"; is very much not "AI" but could have easily produced the above result.

        21 votes
        1. [3]
          ShroudedScribe
          Link Parent
          I agree with you, but this is one of those battles over terminology that has already been lost. So many companies wanted to appeal to buzzword-addicted executives that they rebranded their...

          I agree with you, but this is one of those battles over terminology that has already been lost. So many companies wanted to appeal to buzzword-addicted executives that they rebranded their existing logic-based programming as "AI" even though it's far from the truth.

          Journalism injecting it into clickbait titles isn't a shock either. That's how headlines have been for the last 10+ years.

          The first time I closely watched this unravel was with the term "hacking." It quickly devolved from finding exploits in code or open firewall ports to reading a password to someone's Facebook account from a sticky note on their monitor. Even legal definitions are simply "unauthorized access," regardless of how easy it may have been to obtain it.

          7 votes
          1. spicyq
            Link Parent
            A little off topic, but it's kind of funny that "hacker" is your example of a word losing it's legitimate meaning of being able to find exploits, because that meaning was also considered a...

            A little off topic, but it's kind of funny that "hacker" is your example of a word losing it's legitimate meaning of being able to find exploits, because that meaning was also considered a distortion of the original meaning of programmers and tinkerers.

            8 votes
          2. zestier
            (edited )
            Link Parent
            I do agree that it is being eroded, but I think we're far from the world really accepting "AI" to be defined as "any logic run on a computer". We will definitely still have people misusing it...

            I do agree that it is being eroded, but I think we're far from the world really accepting "AI" to be defined as "any logic run on a computer". We will definitely still have people misusing it though.

            The reason I say this is that even with the abysmal definition of "AI" it has a large number of connotations that don't fit standard algorithms. A few examples being dedicated AI chips, massive energy and water usage, and legally grey usage of copyrighted material.

            I'm currently working on the assumption that the journalist above is just wrong about AI being used, but using the buzzword pulls in a bunch of connotations that are likely misapplied. Comments like, "Do we really need to be burning down the planet to identify people that risk missing payments? We have their payment history already." or "This is the danger of using AI. If they just wrote actual logic instead they could handle housing vouchers correctly."

            My interpretation of the whole thing is that they may be using human-crafted logic, but that said logic just didn't account for housing vouchers. There could be real discussion about how or why that was missed or what the most appropriate fixes would be, but letting the "AI" connotations seep further in distracts from those.

            3 votes
        2. sparksbet
          Link Parent
          It's very possible (and imo likely) that they were using something more complicated than just the line of code you're writing here, though -- maybe they're using decision trees or some other ML...

          It's very possible (and imo likely) that they were using something more complicated than just the line of code you're writing here, though -- maybe they're using decision trees or some other ML technique. Things like that are simpler than the generative LLMs that are currently in vogue, but they've been called AI since before those things were around.

          3 votes
        3. [4]
          cdb
          Link Parent
          What would you say is a good definition of AI?

          What would you say is a good definition of AI?

          1. [3]
            zestier
            Link Parent
            As someone that just exited a job that assisted with building "GenAI" applications, it now just means ML targeted directly at non-technical users. There are some other reasonable markers that work...

            As someone that just exited a job that assisted with building "GenAI" applications, it now just means ML targeted directly at non-technical users.

            There are some other reasonable markers that work for the fuzzier definitions that importantly keep the connotations on the same page. These are things like that they use tons of power, consume massive amounts of data, and heavily leverage "AI chips". If we also call cases where none of those are true "AI" then the larger discussions about problems with AI start to not even make any sense. This is because when people see "AI" they also start to wonder about it's training data and energy usage.

            2 votes
            1. [2]
              cdb
              (edited )
              Link Parent
              I read your comment over three times, and I honestly don't feel like there is a strong definition in there that could be applied to say whether a thing is AI or not. AI definitely has a broader...

              I read your comment over three times, and I honestly don't feel like there is a strong definition in there that could be applied to say whether a thing is AI or not.

              AI definitely has a broader definition than ML targeted towards certain users. Also, this is impossible to actually pin down as a definition because it depends on who is doing the targeting.

              AI also definitely cannot be defined by the power consumption, amount of data, and type of equipment used. That's a definition based on methods we are using to accomplish a certain goal rather than defining the goal. It's like saying that the definition of "splitting wood" is "when people use an axe."

              If there is a definition that has been eroded, what did that original whole and complete definition look like?

              1 vote
              1. zestier
                Link Parent
                Fair, I didn't really provide a definition as I think it largely lacks one due to its buzzword status. There are obviously strong connections to it's usage though, which is why I was discussing...

                Fair, I didn't really provide a definition as I think it largely lacks one due to its buzzword status. There are obviously strong connections to it's usage though, which is why I was discussing connotations rather than denotations.

                I don't remember the wording, but there was a question like, "Is AI making the world better or worse?" in the demographic survey. The erosion I was referring to is related to the implied definition of questions like that. I don't think anyone answering that would've included cases like putting a threshold on maximum missed payments in a given time frame as "AI" in their consideration, unless all of that was happening inside an ML box.

                If I were to attempt to denote that use more specifically, which is still imperfect: virtual recreation of human behavior(s) within a specified domain by applying models intended to represent human behavior(s) within that domain.

                Buzzwords suck. It now unfortunately means whatever is happening when a company slaps "Powered by AI" on a product, and that seems to mean "we're now using AI chips."

                3 votes
    4. [9]
      frostycakes
      Link Parent
      Seems odd, as credit cards would be the first thing I'd let fall behind in a situation where I needed to make rent and had to choose a bill to not pay on time. Hell, I have had to make that exact...

      falling behind on credit card bills is a strong marker for a risky tennant

      Seems odd, as credit cards would be the first thing I'd let fall behind in a situation where I needed to make rent and had to choose a bill to not pay on time. Hell, I have had to make that exact choice when I was out of work and a new job ended up starting months later than it initially was slated to.

      Using credit card debt alone does not reflect the fact that people will pay rent/mortgages before almost any other bills they have. I had to default on a credit card in my situation, but my rent was paid on time and in full during that entire time. I'm fine with the fact that it took a long time to be able to qualify for a credit card again afterwards, but why should the fact that someone prioritized their housing payments in a tight situation be used to determine that they're a bad tenant? Makes zero sense, once one digs below the surface.

      11 votes
      1. [8]
        stu2b50
        Link Parent
        I mean, the goal would be to have tenants who have enough financial stability to not have to let any bills go unpaid.

        I mean, the goal would be to have tenants who have enough financial stability to not have to let any bills go unpaid.

        16 votes
        1. [7]
          frostycakes
          Link Parent
          I mean, if you're a landlord accepting subsidized housing vouchers, that seems like a criterion that should not be used. It's not as if one is going to be in a good financial situation when...

          I mean, if you're a landlord accepting subsidized housing vouchers, that seems like a criterion that should not be used. It's not as if one is going to be in a good financial situation when they're eligible for said vouchers, and the whole point is to house those in need. Defeats the entire purpose, IMO.

          It's sickening that the desires of these landlords outweighs the need to get people into housing, especially with subsidized vouchers that they choose to accept.

          6 votes
          1. [6]
            stu2b50
            Link Parent
            I think there’s a difference between being low income and having a history of being in debt and not paying. While I’m sure there’s more of the latter in the former, there are still people in the...

            I think there’s a difference between being low income and having a history of being in debt and not paying. While I’m sure there’s more of the latter in the former, there are still people in the former who are not in the latter. If someone gets by with better planning and cutting more expenses, it makes sense for them to be preferred.

            If nothing else, it’s a signal.

            7 votes
            1. [5]
              frostycakes
              Link Parent
              We're talking about people who are definitionally the former here, as they have subsidized housing vouchers. Credit scoring should be prohibited when you have a tenant whose housing voucher...

              We're talking about people who are definitionally the former here, as they have subsidized housing vouchers. Credit scoring should be prohibited when you have a tenant whose housing voucher guarantees that income will be coming in to the landlord, period.

              6 votes
              1. Minori
                Link Parent
                It is broadly illegal to discriminate against housing vouchers holders: https://www.hud.gov/program_offices/public_indian_housing/programs/hcv/source-of-income-protections Factors such as credit...

                It is broadly illegal to discriminate against housing vouchers holders: https://www.hud.gov/program_offices/public_indian_housing/programs/hcv/source-of-income-protections

                Factors such as credit history are still considered though because a voucher doesn't necessarily cover the full cost of housing.

                8 votes
              2. [3]
                stu2b50
                Link Parent
                Well, yeah, that’s what I said. But again, being low income does not necessarily mean you have a low credit score. The original discussion was whether credit scores were a strong signal or not for...

                Well, yeah, that’s what I said. But again, being low income does not necessarily mean you have a low credit score.

                The original discussion was whether credit scores were a strong signal or not for being late in rent payments. I see no reason why they wouldn’t be. The particular threshold can be set at whatever risk level is appropriate for the aim of the housing. If I were a landlord, given equal, low, incomes, I’d rather rent to a tenant with no credit history, than one with a bad credit history.

                6 votes
                1. [2]
                  frostycakes
                  Link Parent
                  And we wonder why a homelessness crisis exists. People's need for shelter should outweigh any desires on the part of landlords who are accepting state subsidies for tenants. If they don't like it?...

                  And we wonder why a homelessness crisis exists. People's need for shelter should outweigh any desires on the part of landlords who are accepting state subsidies for tenants.

                  If they don't like it? They should throw support behind state owned and operated public housing again then, and let the scummy investor class go back to playing with solely market rate apartments. It's the cost of not having a robust public housing system, and it's the landlord class who should bear the brunt of those negative externalities, not the tenants in need, regardless of why the tenants are in said financial situation.

                  4 votes
                  1. Minori
                    Link Parent
                    Because there's not enough housing due to NIMBYs. It's not big corporations driving zoning policy in major cities. If developers had the choice, they'd love to build more housing. Affordable...

                    And we wonder why a homelessness crisis exists.

                    Because there's not enough housing due to NIMBYs. It's not big corporations driving zoning policy in major cities. If developers had the choice, they'd love to build more housing.

                    Affordable public housing is a whole other can of worms. In the US, black residents of St. Louis celebrated when Pruitt-Igoo was demolished because it was awful for residents and served to isolate poor black residents even further in poverty and crime. The state isn't perfect, and public housing has a sordid history in the US. Some countries do it better, but many people rightfully treat public housing with suspicion in America.

                    3 votes
    5. DefinitelyNotAFae
      Link Parent
      Every article uses AI in talking about this. I think it's likely they changed their language. But two years ago....

      Their product page doesn't mention AI anywhere, and there's no indication they use machine learning or other AI technologies in their score. If publications continue to arbitrarily insert the term AI for clicks, it's just going to mislead people about what artificial intelligence even is and what it can do.

      Every article uses AI in talking about this. I think it's likely they changed their language. But two years ago. https://www.nclc.org/saferent-solution-accused-of-illegally-discriminating-against-black-and-hispanic-rental-applicants/
      At no point have they seemed to object to the term AI being used to describe them.

      Here's a similar product:
      https://newsroom.transunion.com/property-managers-gain-greater-predictive-power-to-help-further-decrease-future-evictions/
      All of them use the same language of AI even if they don't say the word.

      In 2022 they said "powered by machine learning" and earlier versions use similar language

      But frankly if all these AI companies want to use AI for "Algorithm" and "machine learning" it's not shocking when those two terms have become synonymous. The media didn't start this misconception.

      4 votes
  2. [11]
    DefinitelyNotAFae
    Link
    It's like clockwork. "AI" gives biased outcomes due to biased input. Passing the bias on to machines fixes nothing.

    SafeRent has settled with Louis and Douglas. In addition to making a $2.3m payment, the company has agreed to stop using a scoring system or make any kind of recommendation when it came to prospective tenants who used housing vouchers for five years. Though SafeRent legally admitted no wrongdoing, it is rare for a tech company to accept changes to its core products as part of a settlement; the more common result of such agreements would be a financial agreement.

    It's like clockwork. "AI" gives biased outcomes due to biased input. Passing the bias on to machines fixes nothing.

    17 votes
    1. [10]
      Minori
      Link Parent
      The company didn't change the algorithm in any way if I'm understanding this correctly. They simply stopped calculating scores for prospective tenants with housing vouchers. Who knows whether...

      The company didn't change the algorithm in any way if I'm understanding this correctly. They simply stopped calculating scores for prospective tenants with housing vouchers. Who knows whether that'll improve the odds of people with housing vouchers or not.

      12 votes
      1. [7]
        DefinitelyNotAFae
        Link Parent
        Oh I know. They admit no wrong doing. I don't think landlords need to know how someone's credit card payments have gone in the first place. If no matter how much they struggle the tenants pay rent...

        Oh I know. They admit no wrong doing. I don't think landlords need to know how someone's credit card payments have gone in the first place. If no matter how much they struggle the tenants pay rent on time, then the landlord should see them as good tenants not judge their credit card bills (which can come from other necessities as much as "irresponsible spending")

        It's just predictable

        5 votes
        1. [6]
          Minori
          Link Parent
          I think it makes sense to include all relevant factors. While some people may always pay their rent no matter what, the facts are that someone with crippling credit card debt is far more likely to...

          I think it makes sense to include all relevant factors. While some people may always pay their rent no matter what, the facts are that someone with crippling credit card debt is far more likely to fall behind on rent and become a freeloader that has to be evicted. It seems foolish not to consider the likelihood of paying rent each month when picking tenants (and of course the only reason landlords can be so picky is due to the supply shortage in the US).

          6 votes
          1. [5]
            DefinitelyNotAFae
            Link Parent
            But people with "crippling" (your word) credit card debt who still always pay their rent are more likely to keep paying their rent. It's all about where you make the data cut off. It seems the...

            But people with "crippling" (your word) credit card debt who still always pay their rent are more likely to keep paying their rent. It's all about where you make the data cut off. It seems the important thing is "do they pay the rent and will they keep doing so" rather than "what is overall credit card situation and how might that impact their future rent paying."

            The algorithm they used is private and they don't share what they're using, how they're weighting it or why someone wasn't approved. That to me is not better than "I don't like the vibes and I'm not renting to you" especially if both the algorithm and the "vibes" demonstrate bias.

            I had a landlord that "liked to rent to single women" because he thought they didn't cause problems and he was mad paternalistic. Guess who didn't want me to have my partner move in... because he had a kid that would visit on holidays... who was a teenager...I'm not legally allowed to share what sort of outcome happened there, but in similar situations discrimination against parents is a fair housing act violation.

            Does the algorithm avoid fair housing act violations? Who can say. Also I object to the framing of people who fall behind in their rent as "freeloaders." I'm not sitting here calling landlords "vultures" so if we could keep the framing away from that sort of language I'd appreciate it.

            5 votes
            1. [4]
              Minori
              Link Parent
              This is a nice theory, but do you have any evidence to back it up? Everything I'm familiar with says high-risk tenants with bad credit scores are far more likely to eventually become non-paying...

              But people with "crippling" (your word) credit card debt who still always pay their rent are more likely to keep paying their rent.

              This is a nice theory, but do you have any evidence to back it up? Everything I'm familiar with says high-risk tenants with bad credit scores are far more likely to eventually become non-paying residents that have to be evicted.

              If it wasn't a useful metric and the company is randomly discriminating against some tenants for no reason, I'd be shocked.

              5 votes
              1. [3]
                DefinitelyNotAFae
                Link Parent
                It's tautological. That was my point. They're not considering, apparently, whether someone's poor credit matters if they always pay rent. And companies and business owners (including landlords)...

                It's tautological. That was my point. They're not considering, apparently, whether someone's poor credit matters if they always pay rent.

                And companies and business owners (including landlords) make stupid decisions based on bias all the time. They're not some sort of ultra-rational actors. Historically they will discriminate for stupid reasons all the time.

                5 votes
                1. [2]
                  Minori
                  (edited )
                  Link Parent
                  While you're right, competing organizations are still semi-rational and tend towards better metrics to improve outcomes. If their algorithm didn't work, I don't think it'd be widely used. I'm not...

                  While you're right, competing organizations are still semi-rational and tend towards better metrics to improve outcomes. If their algorithm didn't work, I don't think it'd be widely used.

                  I'm not sure it's tautological to use a proxy metric that's associated with a group you want to filter out (like renters that may not pay rent). Credit score is consistently used as a proxy for "likelihood of making payments" which I'd generally consider to be more a function of financial literacy than poverty. Plenty of poor people have good credit scores. I'm speaking from personal and familial experience.

                  Unpaid rent can go to collections and end up on a credit report. Cedit score can thus be used as an indirect, cheap proxy for tenant history sometimes: https://www.chase.com/personal/credit-cards/education/credit-score/can-eviction-hurt-credit-score

                  1 vote
                  1. DefinitelyNotAFae
                    Link Parent
                    It's tautological that someone who always pays their rent regardless of their credit card payment history always pays their rent. Regardless, poor people will inevitably have lower credit scores...

                    It's tautological that someone who always pays their rent regardless of their credit card payment history always pays their rent. Regardless, poor people will inevitably have lower credit scores as a group due to having less to spare for emergencies, less likely to have financial professionals, and simply accessing credit less. (It's genuinely screwed up that you "have to" get a credit card or other loan to be able to get future credit and have a score to rent, but get zero benefit from paying your rent towards this. Eviction history is generally public knowledge, but only the negatives of renting being on the credit report is some bullshit.)

                    Landlords aren't really competing with each other, not when there's not a glut of housing on the market. And it's not a given that you only pay for a service if it's correct, after all they paid for this service when it was almost certainly discriminating against voucher holders. Maybe they undercut the cost of other background checks, maybe they make it easy to not have to screen people yourself, could be anything. At best SafeRent is competing with other algorithm companies - at least one of which is getting charged with price fixing IIRC - and they still don't have to be "right" especially in a landlord's market.

                    3 votes
      2. [2]
        tibpoe
        Link Parent
        But after 5 years, I imagine that they will have had time to add housing vouchers to their algorithm. From how fast they settled (about a year), it almost sounds to me like the folks at the...

        But after 5 years, I imagine that they will have had time to add housing vouchers to their algorithm. From how fast they settled (about a year), it almost sounds to me like the folks at the company didn't realize that vouchers were a major thing that they needed to consider.

        Not calculating scores seems like a reasonable stopgap to limit further harm.

        5 votes
        1. Minori
          Link Parent
          My concern is some landlords may be more likely to reject tenants with vouchers because they can't get risk scores. States with laws that make it illegal to ask for criminal history often have...

          My concern is some landlords may be more likely to reject tenants with vouchers because they can't get risk scores.

          States with laws that make it illegal to ask for criminal history often have more racial discrimination in hiring for example. There very well could be some unintended consequences.

          3 votes