I was wondering how many people were in each group. Here is the full report. It’s not easy to cut and paste excerpts, but here are some numbers: Group A had 260 people and 209 completed the...
I was wondering how many people were in each group. Here is the full report. It’s not easy to cut and paste excerpts, but here are some numbers:
Group A had 260 people and 209 completed the initial survey, and 154 the six months survey.
Group B had 260, 193, and 136.
Group C: 300, 231, 167.
For a rough comparison, an election poll of a thousand people typically has a 3% margin of error and a hundred people would be a 10% margin of error according to this article.
Weird data, group C (the "control" group) showing almost the same amount of improvement in homelessness as group A, which makes the whole study seem pretty inconclusive. I believe in some form of...
Weird data, group C (the "control" group) showing almost the same amount of improvement in homelessness as group A, which makes the whole study seem pretty inconclusive. I believe in some form of UBI system, but this study does little to convince me it works for the unsheltered
As ducc stated, they didn't go with a control group, but a "comparison group" that got some small monetary compensation. On one hand, that weakens some of the results of the study. On the other...
As ducc stated, they didn't go with a control group, but a "comparison group" that got some small monetary compensation. On one hand, that weakens some of the results of the study. On the other hand, perhaps it means that even a small-but-reliable increase in income makes a non-trivial difference to those in their position.
I don't believe it's reasonable to draw conclusions that $50 a month made any significant difference on its own. This is much more likely just an instance of correlation =/= causation.
I don't believe it's reasonable to draw conclusions that $50 a month made any significant difference on its own. This is much more likely just an instance of correlation =/= causation.
$50/mo that you can depend on recurring when you're on the street? There's a lot you can do with that that'll make a difference. That's enough to pay for a gym membership so that you can regularly...
$50/mo that you can depend on recurring when you're on the street? There's a lot you can do with that that'll make a difference. That's enough to pay for a gym membership so that you can regularly shower, or an inexpensive cell phone plan, or even simply more consistent food. Having consistent income means you can plan, and even $50/mo can make a huge difference if you have very little.
The study says the group had a 20% increase in owning/renting a home. That's the part I don't believe the 50 bucks made a big difference in. Especially considering the group who got 1000 bucks a...
The study says the group had a 20% increase in owning/renting a home. That's the part I don't believe the 50 bucks made a big difference in. Especially considering the group who got 1000 bucks a month only had a 26% increase in owning/renting a home. Those two numbers just don't add up.
The entirety of the question was asking where the people slept last night, and one of the options was "at a home that I owned/rented." This tells us something, but less than would really be...
The entirety of the question was asking where the people slept last night, and one of the options was "at a home that I owned/rented." This tells us something, but less than would really be helpful. We know it's not a hotel, because that's a separate option, and it's not for free at a friend's place, for similar reasons, but we know nothing about the duration of the rental, quality of the situation, etc. The chances of it being a place they own is likely very small or nil, but we don't know because they're lumped together.
I thought the same thing and despite being in favor of UBI, this study was very disconcerting. Even though the control group appears to improve at nearly the same rate as Group A, it’s impossible...
I thought the same thing and despite being in favor of UBI, this study was very disconcerting.
Even though the control group appears to improve at nearly the same rate as Group A, it’s impossible to trust any conclusion from the data. Based on the numbers that skybrian posted, the data they have reported is not only inconclusive but could even be wildly misleading, so no conclusions should be drawn.
The percentages of participants that receive money and did not fill out the 6 month survey for each group are as follows:
Group A: 19.6%
Group B: 47.7%
Group C: 44.3%
It’s impossible to know what the housing, employment situations, or even well being is for each of the participants after receiving these monies, however what if the reason that nearly HALF of Group B didn’t show up is because they were murdered for having $6500 of cash while experiencing homelessness?
I’m not suggesting that is what happened but rather stating it’s irresponsible to lose half of your study’s subject population and still think the resulting data is meaningful.
On the contrast, perhaps you could think of it as a relatively cheap scheme ($50/monthly) has a marked improvement, and that would be much easier to implement for the entire homeless population if...
On the contrast, perhaps you could think of it as a relatively cheap scheme ($50/monthly) has a marked improvement, and that would be much easier to implement for the entire homeless population if further studies show even a small amount monthly leads to significant improvements over nothing.
I support further testing, but that's not a conclusion I would make based on this data. I don't believe such a small sum of money would have any significant impact on people's lives, especially...
I support further testing, but that's not a conclusion I would make based on this data. I don't believe such a small sum of money would have any significant impact on people's lives, especially compared to the amount that gets spent on shelters, food banks, etc which homeless already have access to.
I know this is in the US, but the UK did some experimentation in London some years ago and had massive unticks to the homeless managing to get back on their feet and live really productive lives...
I know this is in the US, but the UK did some experimentation in London some years ago and had massive unticks to the homeless managing to get back on their feet and live really productive lives again. Just by getting cash into the bank, rather than it being targetted.
London has had a rough time with homelessness thanks to government policy in the past 13 years, so that was seen as a breakthrough. Naturally the results are buried into academia and not put forward in the papers, but it was a huge deal that showed it works.
It also gets a lot of the administration cost away from managing food banks, homeless shelters, benefits / welfare and all sorts.
I agree with you, and I think most likely this is a function of the attrition of responses (people who get back on their feet would be more likely to continue to respond). That's why I added the...
I agree with you, and I think most likely this is a function of the attrition of responses (people who get back on their feet would be more likely to continue to respond). That's why I added the caveat of further studies (ideally with a proper negative control of no money).
I've read that there are a lot of people for whom homelessness is transient. It seems like we should expect some people's lives to improve from their own efforts, but not everyone? The study also...
I've read that there are a lot of people for whom homelessness is transient. It seems like we should expect some people's lives to improve from their own efforts, but not everyone?
The study also excluded people with mental health problems.
I have been technically homeless, in that I was couch surfing, but I had a solid social network that gave me space to exist while I figured out my finances. Drug addicts, the mentally ill, the...
I have been technically homeless, in that I was couch surfing, but I had a solid social network that gave me space to exist while I figured out my finances.
Drug addicts, the mentally ill, the socially inept and people who are unlucky don't have a social network to support them in that way.
I think this is something that isn’t discussed as much in popular talk of homelessness. There is a distinction between street homeless (the most visible, and IMO what is most discussed) and...
I think this is something that isn’t discussed as much in popular talk of homelessness. There is a distinction between street homeless (the most visible, and IMO what is most discussed) and sheltered homeless. They often don’t have quite the same causes, and may require different policy measures to deal with.
I'm pretty interested in the idea of UBI, but not having a control group which wasn't given anything seems like a large problem to me. They explicitly state this was done on purpose: While I get...
I'm pretty interested in the idea of UBI, but not having a control group which wasn't given anything seems like a large problem to me. They explicitly state this was done on purpose:
Additionally, this study uses an active comparison group rather than a control group. A control
group typically receives “treatment as usual,” meaning they would not receive any monthly
stipend from DBIP. However, due to ethical and research engagement concerns, the DBIP
program and research design team decided to employ an active comparison group,
hypothesizing that $50 a month would be enough money to incentivize and compensate
participants to engage in DBIP, but a significantly smaller sum of money than that given to
participants in Groups A and B such that the research design would still be able to capture the
impact of different cash payments for outcomes of concern.
While I get the reasoning, I think the approach doesn't do a great job at isolating confounding factors (such as changing economic conditions as a whole). The differences in between comparison groups are promising, but it still doesn't tell us what UBI does compared to doing nothing.
But, like ethically and such, are you gonna be the one who chooses who's in the "Gets nothing group"? Yeah, the data could be cleaner but at least some people didn't suffer more as a result.
But, like ethically and such, are you gonna be the one who chooses who's in the "Gets nothing group"?
Yeah, the data could be cleaner but at least some people didn't suffer more as a result.
Aren't there routinely medical studies where patients are sorted into control/treatment groups, including when such treatment might theoretically benefit the control group? I'm not seeing what...
Aren't there routinely medical studies where patients are sorted into control/treatment groups, including when such treatment might theoretically benefit the control group? I'm not seeing what specifically is different in these situations. Or perhaps you are objecting to those as well.
But, since the results of this study are not fully conclusive, isn't there an argument that if a "gets nothing" group gave us more info, we would be able to take better action in the future to help more people? That is, how much do we value helping ~200 people right now by $50 each, and how much do we value the chance of better policy to help some presumably larger population in the future?
Besides, I think the point was that we don't know how much giving people $50 helps them. So it's a little weird to place a high value on making sure people "get at least something".
With medical trials, the control group will typically receive current standard of care to avoid ethical issues of not treating people with a disease when there is an available treatment.
With medical trials, the control group will typically receive current standard of care to avoid ethical issues of not treating people with a disease when there is an available treatment.
( this is also meant as a reply to @Wish_for_a_dragon and @Sodliddesu ) Wouldn't the "current standard of care" here be just not giving out any money, since access to shelters is not changed by...
Wouldn't the "current standard of care" here be just not giving out any money, since access to shelters is not changed by the study? Giving a smaller amount is more like giving a smaller dose of the treatment, not the current standard of care.
To be clear, I'm not objecting against giving the $50 per month. I think it's helpful for getting a good follow-up rate, for example. And I suppose it's possible to do some kind of regression on the results to guess what $0 would do.
The issue I have is that the results of this study seem to indicate that either: (1) there is very little benefit to giving UBI at all, OR (2) most of the benefit of UBI is in the first $50. This seems like a pretty important question to answer to figure out the feasibility of UBI. But if it is really unethical to give $0, then such a follow-up study can't be done. Is that a good thing for UBI policy?
Together with the fact that any such study's population likely only involves a fraction of the total population experiencing homelessness in that city (i.e., there will be an unstudied group of people "chosen" to receive $0 anyway), I feel that the claim "having a $0 group is automatically unethical" is not a self-consistent position.
As mentioned by @tannhauser, the control group will receive the current standard of care to avoid ethical concerns. However, in a medical setting there is a situation where, when the drug is so...
As mentioned by @tannhauser, the control group will receive the current standard of care to avoid ethical concerns. However, in a medical setting there is a situation where, when the drug is so obviously beneficial that it would be unethical to withhold it as treatment, it can be administered to the control group. Whether during or after the study, I can’t recall, and my Google-fu is failing me to find the name of this situation.
It's called Early Stopping. An independent body reviews draft data at milestones during the drug trial (like 1, 3, and 6 months), and they may decide to end the study early if there's an extremely...
It's called Early Stopping. An independent body reviews draft data at milestones during the drug trial (like 1, 3, and 6 months), and they may decide to end the study early if there's an extremely obvious benefit. Study participants are usually still followed; sans control group.
This is still somewhat controversial because drug makers are only incentivized to find enough evidence of efficacy to get the drug to market. There's debate in the literature over whether stopping trials early leads to an over-estimation of benefit. It's hard to conclusively say since there's no randomized double-blind control group with early stopping.
$50 a month (along with paying people to complete the surveys) isn't nothing, though. They got substantially less but still more than people who didn't participate (who got nothing).
$50 a month (along with paying people to complete the surveys) isn't nothing, though. They got substantially less but still more than people who didn't participate (who got nothing).
While the study isn't perfect, I think some of the criticism here is a bit misplaced. If the study was exactly the same, and suggested a greater benefit to UBI, would you be as critical of the...
While the study isn't perfect, I think some of the criticism here is a bit misplaced. If the study was exactly the same, and suggested a greater benefit to UBI, would you be as critical of the design? You need humility and the capacity to question your own beliefs, or you can keep repeating a study or tweaking the data until you see what you want to see.
What's bothering me is I didn't see any statistical analysis reported in the article or in the linked infographic.
Edit: Ah I see there is data in the DBIP Interim Quantitative Report.
They did the stats in a way where they don't even compare between groups, they do an ANOVA to assess differences at enrollment, but then only t-tests for before and after. They literally don't even compare if UBI is more effective than vs the comparison group, really disappointing, this wouldn't get published in any real journal.
I think your edit basically reaffirms the concerns. Unless the effect size was absolutely massive (which it doesn't seem to be), their analysis and study design is too weak to prove anything...
I think your edit basically reaffirms the concerns. Unless the effect size was absolutely massive (which it doesn't seem to be), their analysis and study design is too weak to prove anything...
I'm talking more about the comments saying "$50 isn't a real control", if they didn't offer the $50 there's no way they would have comparable retention. Also, I'm not a stats guy, but I'm pretty...
I'm talking more about the comments saying "$50 isn't a real control", if they didn't offer the $50 there's no way they would have comparable retention.
Also, I'm not a stats guy, but I'm pretty sure you can compare between groups with this study design, they just chose not to, (probably they did and it was insignificant).
It’s an “interim report.” Hopefully they’ll do more analysis in the final report? Ruling out large bad effects might be useful for showing the money isn’t wasted. (Some people fear it would be...
It’s an “interim report.” Hopefully they’ll do more analysis in the final report?
Ruling out large bad effects might be useful for showing the money isn’t wasted. (Some people fear it would be used on drugs or alcohol.)
I was wondering how many people were in each group. Here is the full report. It’s not easy to cut and paste excerpts, but here are some numbers:
Group A had 260 people and 209 completed the initial survey, and 154 the six months survey.
Group B had 260, 193, and 136.
Group C: 300, 231, 167.
For a rough comparison, an election poll of a thousand people typically has a 3% margin of error and a hundred people would be a 10% margin of error according to this article.
Weird data, group C (the "control" group) showing almost the same amount of improvement in homelessness as group A, which makes the whole study seem pretty inconclusive. I believe in some form of UBI system, but this study does little to convince me it works for the unsheltered
As ducc stated, they didn't go with a control group, but a "comparison group" that got some small monetary compensation. On one hand, that weakens some of the results of the study. On the other hand, perhaps it means that even a small-but-reliable increase in income makes a non-trivial difference to those in their position.
I don't believe it's reasonable to draw conclusions that $50 a month made any significant difference on its own. This is much more likely just an instance of correlation =/= causation.
$50/mo that you can depend on recurring when you're on the street? There's a lot you can do with that that'll make a difference. That's enough to pay for a gym membership so that you can regularly shower, or an inexpensive cell phone plan, or even simply more consistent food. Having consistent income means you can plan, and even $50/mo can make a huge difference if you have very little.
You could shower and even wash a change of clothes. Smelling and looking relatively normal is a huge boost if you’re looking for any kind of job.
The study says the group had a 20% increase in owning/renting a home. That's the part I don't believe the 50 bucks made a big difference in. Especially considering the group who got 1000 bucks a month only had a 26% increase in owning/renting a home. Those two numbers just don't add up.
The entirety of the question was asking where the people slept last night, and one of the options was "at a home that I owned/rented." This tells us something, but less than would really be helpful. We know it's not a hotel, because that's a separate option, and it's not for free at a friend's place, for similar reasons, but we know nothing about the duration of the rental, quality of the situation, etc. The chances of it being a place they own is likely very small or nil, but we don't know because they're lumped together.
I thought the same thing and despite being in favor of UBI, this study was very disconcerting.
Even though the control group appears to improve at nearly the same rate as Group A, it’s impossible to trust any conclusion from the data. Based on the numbers that skybrian posted, the data they have reported is not only inconclusive but could even be wildly misleading, so no conclusions should be drawn.
The percentages of participants that receive money and did not fill out the 6 month survey for each group are as follows:
Group A: 19.6%
Group B: 47.7%
Group C: 44.3%
It’s impossible to know what the housing, employment situations, or even well being is for each of the participants after receiving these monies, however what if the reason that nearly HALF of Group B didn’t show up is because they were murdered for having $6500 of cash while experiencing homelessness?
I’m not suggesting that is what happened but rather stating it’s irresponsible to lose half of your study’s subject population and still think the resulting data is meaningful.
On the contrast, perhaps you could think of it as a relatively cheap scheme ($50/monthly) has a marked improvement, and that would be much easier to implement for the entire homeless population if further studies show even a small amount monthly leads to significant improvements over nothing.
I support further testing, but that's not a conclusion I would make based on this data. I don't believe such a small sum of money would have any significant impact on people's lives, especially compared to the amount that gets spent on shelters, food banks, etc which homeless already have access to.
I know this is in the US, but the UK did some experimentation in London some years ago and had massive unticks to the homeless managing to get back on their feet and live really productive lives again. Just by getting cash into the bank, rather than it being targetted.
London has had a rough time with homelessness thanks to government policy in the past 13 years, so that was seen as a breakthrough. Naturally the results are buried into academia and not put forward in the papers, but it was a huge deal that showed it works.
It also gets a lot of the administration cost away from managing food banks, homeless shelters, benefits / welfare and all sorts.
I agree with you, and I think most likely this is a function of the attrition of responses (people who get back on their feet would be more likely to continue to respond). That's why I added the caveat of further studies (ideally with a proper negative control of no money).
I've read that there are a lot of people for whom homelessness is transient. It seems like we should expect some people's lives to improve from their own efforts, but not everyone?
The study also excluded people with mental health problems.
I have been technically homeless, in that I was couch surfing, but I had a solid social network that gave me space to exist while I figured out my finances.
Drug addicts, the mentally ill, the socially inept and people who are unlucky don't have a social network to support them in that way.
I think this is something that isn’t discussed as much in popular talk of homelessness. There is a distinction between street homeless (the most visible, and IMO what is most discussed) and sheltered homeless. They often don’t have quite the same causes, and may require different policy measures to deal with.
I'm pretty interested in the idea of UBI, but not having a control group which wasn't given anything seems like a large problem to me. They explicitly state this was done on purpose:
While I get the reasoning, I think the approach doesn't do a great job at isolating confounding factors (such as changing economic conditions as a whole). The differences in between comparison groups are promising, but it still doesn't tell us what UBI does compared to doing nothing.
But, like ethically and such, are you gonna be the one who chooses who's in the "Gets nothing group"?
Yeah, the data could be cleaner but at least some people didn't suffer more as a result.
Aren't there routinely medical studies where patients are sorted into control/treatment groups, including when such treatment might theoretically benefit the control group? I'm not seeing what specifically is different in these situations. Or perhaps you are objecting to those as well.
But, since the results of this study are not fully conclusive, isn't there an argument that if a "gets nothing" group gave us more info, we would be able to take better action in the future to help more people? That is, how much do we value helping ~200 people right now by $50 each, and how much do we value the chance of better policy to help some presumably larger population in the future?
Besides, I think the point was that we don't know how much giving people $50 helps them. So it's a little weird to place a high value on making sure people "get at least something".
With medical trials, the control group will typically receive current standard of care to avoid ethical issues of not treating people with a disease when there is an available treatment.
( this is also meant as a reply to @Wish_for_a_dragon and @Sodliddesu )
Wouldn't the "current standard of care" here be just not giving out any money, since access to shelters is not changed by the study? Giving a smaller amount is more like giving a smaller dose of the treatment, not the current standard of care.
To be clear, I'm not objecting against giving the $50 per month. I think it's helpful for getting a good follow-up rate, for example. And I suppose it's possible to do some kind of regression on the results to guess what $0 would do.
The issue I have is that the results of this study seem to indicate that either: (1) there is very little benefit to giving UBI at all, OR (2) most of the benefit of UBI is in the first $50. This seems like a pretty important question to answer to figure out the feasibility of UBI. But if it is really unethical to give $0, then such a follow-up study can't be done. Is that a good thing for UBI policy?
Together with the fact that any such study's population likely only involves a fraction of the total population experiencing homelessness in that city (i.e., there will be an unstudied group of people "chosen" to receive $0 anyway), I feel that the claim "having a $0 group is automatically unethical" is not a self-consistent position.
As mentioned by @tannhauser, the control group will receive the current standard of care to avoid ethical concerns. However, in a medical setting there is a situation where, when the drug is so obviously beneficial that it would be unethical to withhold it as treatment, it can be administered to the control group. Whether during or after the study, I can’t recall, and my Google-fu is failing me to find the name of this situation.
Compassionate use, maybe?
It's called Early Stopping. An independent body reviews draft data at milestones during the drug trial (like 1, 3, and 6 months), and they may decide to end the study early if there's an extremely obvious benefit. Study participants are usually still followed; sans control group.
This is still somewhat controversial because drug makers are only incentivized to find enough evidence of efficacy to get the drug to market. There's debate in the literature over whether stopping trials early leads to an over-estimation of benefit. It's hard to conclusively say since there's no randomized double-blind control group with early stopping.
$50 a month (along with paying people to complete the surveys) isn't nothing, though. They got substantially less but still more than people who didn't participate (who got nothing).
For reference looks like the line breaks are off in your quote
While the study isn't perfect, I think some of the criticism here is a bit misplaced. If the study was exactly the same, and suggested a greater benefit to UBI, would you be as critical of the design? You need humility and the capacity to question your own beliefs, or you can keep repeating a study or tweaking the data until you see what you want to see.
What's bothering me is I didn't see any statistical analysis reported in the article or in the linked infographic.
Edit: Ah I see there is data in the DBIP Interim Quantitative Report.
They did the stats in a way where they don't even compare between groups, they do an ANOVA to assess differences at enrollment, but then only t-tests for before and after. They literally don't even compare if UBI is more effective than vs the comparison group, really disappointing, this wouldn't get published in any real journal.
I think your edit basically reaffirms the concerns. Unless the effect size was absolutely massive (which it doesn't seem to be), their analysis and study design is too weak to prove anything...
I'm talking more about the comments saying "$50 isn't a real control", if they didn't offer the $50 there's no way they would have comparable retention.
Also, I'm not a stats guy, but I'm pretty sure you can compare between groups with this study design, they just chose not to, (probably they did and it was insignificant).
It’s an “interim report.” Hopefully they’ll do more analysis in the final report?
Ruling out large bad effects might be useful for showing the money isn’t wasted. (Some people fear it would be used on drugs or alcohol.)