6 votes

The polls weren't great this year and that was always a possibility

12 comments

  1. [4]
    nacho
    Link
    The idea that someone will be able to forecast weeks or even months ahead of time how an election will end up when millions of people decide if they're going to vote and what they're going to vote...

    The idea that someone will be able to forecast weeks or even months ahead of time how an election will end up when millions of people decide if they're going to vote and what they're going to vote the day before or the day of the election -- That's not a reasonable ask.

    The polls are what they are. They have limitations. When people lie to themselves, they're also going lie to pollsters or researchers getting self-reported data. Generally polls don't misrepresent themselves.

    Irresponsible use of polls is my gripe. We're talking media use of polls, people viewing swings in individual polls that are within margins of error as shifts in attitude.

    We're in a time of change. During the last several terms, things have changed substantially in 4 years. That means polling also faces challenges. Are assumptions and corrections the same as society changes and different people are the ones who go to the polls?

    12 votes
    1. [3]
      NoblePath
      Link Parent
      Edward Bernays would beg to differ.

      The idea that someone will be able to forecast weeks or even months ahead of time how an election will end up when millions of people decide if they're going to vote and what they're going to vote the day before or the day of the election -- That's not a reasonable ask

      Edward Bernays would beg to differ.

      1 vote
      1. [2]
        Icarus
        Link Parent
        Could you elaborate a little more for someone who doesn't know who this is and why you bring them up in this context?

        Could you elaborate a little more for someone who doesn't know who this is and why you bring them up in this context?

        2 votes
        1. NoblePath
          Link Parent
          Edward bernays is the nephew(?) of freud who developed the focus group and really developed us political propaganda to the next level. There is a great bbc documentary entitled “century of the...

          Edward bernays is the nephew(?) of freud who developed the focus group and really developed us political propaganda to the next level. There is a great bbc documentary entitled “century of the self” that is available on youtube.

          The point is that it is not sonfar fetched toquery a group of people amd usefully determine from that what millions if people will decide to do in the future. He would probably say pollster have simply lost their ability to ask the right people, the right questions in the right way. They just need to adapt beyond trying to account for more fundamental shifts through mere refinements in statistical analysis.

          3 votes
  2. [7]
    twisterghost
    (edited )
    Link
    Nate Silver wrote and pushed - daily - article about why we should trust the polls this time around. Walking that back now rubs me pretty wrong. I get that its all probabilities and the nature of...

    Nate Silver wrote and pushed - daily - article about why we should trust the polls this time around. Walking that back now rubs me pretty wrong. I get that its all probabilities and the nature of this work is that all outcomes are possible, but at the same time they make big claims about what we should expect and their business model is for us to trust their forecast, so when they are wrong, turning around and saying "well see we also said that us being wrong was possible" and shifting blame to the pollsters feels like some CYA in action.

    They specifically grade pollsters to make their model more accurate per their view. People use fivethirtyeight's grade for pollsters in all discourse I've seen. Why should we even trust those grades anymore.

    In the end it wasn't massively off, but damn does it not instill confidence.

    10 votes
    1. [6]
      stu2b50
      Link Parent
      Where are those articles? I looked back and the articles Nate published had the exact same kind of hedging. https://fivethirtyeight.com/features/final-2020-presidential-election-forecast/...

      Where are those articles? I looked back and the articles Nate published had the exact same kind of hedging.

      Biden’s Favored In Our Final Presidential Forecast, But It’s A Fine Line Between A Landslide And A Nail-Biter

      This year … I’m not really sure what I’m trying to convince you of. If you think that polling is irrevocably broken because of 2016 — well, that’s not really correct. On the other hand, if it weren’t for 2016, people might look at Joe Biden’s large lead in national polls — the largest of any candidate on the eve of the election since Bill Clinton in 1996 — and conclude that Trump was certain to be a one-term president. If you do think that, please read my story from earlier this week about how Trump can win and why a 10 percent chance needs to be taken seriously.

      https://fivethirtyeight.com/features/final-2020-presidential-election-forecast/

      I’m Here To Remind You That Trump Can Still Win

      And indeed — although nobody needs any reminders of this after 2016 — Trump can win. All the election models are bullish on Biden, but they are united in that a Trump win is still plausible despite his seemingly steep deficit in polls.

      https://fivethirtyeight.com/features/final-2020-presidential-election-forecast/

      9 votes
      1. [5]
        twisterghost
        Link Parent
        Having trouble finding the exact article. I recall it being linked in the preamble text for weeks on the election forecast page. I wonder if they changed the headline to be less strongly worded....

        Having trouble finding the exact article. I recall it being linked in the preamble text for weeks on the election forecast page. I wonder if they changed the headline to be less strongly worded. They often linked to it saying "you can trust the polls this year".

        I guess I don't see what the point is of all of this is if all he can say is "Well maybe, but also maybe not, but trust me, but also if I am wrong, it's not my fault."

        I'm stuck on his pollster rating setup. He claims the authority to rate pollsters, but when several elections come out with pollsters being way off, he's just like "well y'know polls aren't always great" - but yet fivethirtyeight rates them?? Idk, I just don't have the patience for so much walking back, two major presidential elections in a row.

        Biden won, which was his prediction, but still, it ruins my view of their work. The massive asterisks on all their articles makes it harder.

        2 votes
        1. [4]
          stu2b50
          Link Parent
          What do you mean? This isn't a binary "polls are 100% incorrect and have zero correlation with reality" vs "polls are 100% and completely predictive of the final result". If you went by polling...

          I guess I don't see what the point is of all of this is if all he can say is "Well maybe, but also maybe not, but trust me, but also if I am wrong, it's not my fault."

          What do you mean? This isn't a binary "polls are 100% incorrect and have zero correlation with reality" vs "polls are 100% and completely predictive of the final result".

          If you went by polling averages, you would call 48/50 of the Presidential races correctly. Only Florida and North Carolina were wrong, and only Florida was a significantly wrong swing (+3 D to +3 R).

          Polls can simultaneously be useful (and they are, after all they're the only form of live update we have in this political system; neither you, nor politicians want to be running blind for 4 years before we have data) and have uncertainty.

          I'm stuck on his pollster rating setup. He claims the authority to rate pollsters, but when several elections come out with pollsters being way off

          All polls have uncertainty. The ratings come about by running the pollsters through their historical performance and grading them based on how well they correlated with reality.

          Idk, I just don't have the patience for so much walking back, two major presidential elections in a row.

          They also walked back in 2012? When the national polling error was +3 D, approximately the same as the Trump polling error now, just for Obama instead? What changed now?

          9 votes
          1. [3]
            twisterghost
            (edited )
            Link Parent
            Polls are useful, I don't believe otherwise. I think Fivethirtyeight and Nate Silver are overblown in their perceived authority on the matter, given that they make it their business to rate and...

            Polls are useful, I don't believe otherwise. I think Fivethirtyeight and Nate Silver are overblown in their perceived authority on the matter, given that they make it their business to rate and rank pollsters then continuously put out articles that boil down to "we might be right or we might be wrong in our assessment of the quality of polls".

            I don't think they have zero value. I don't think polls are useless. I see no reason to lift 538 above anything else. Hell, some of the closer polls were their lower ranked ones. Especially in 2016.

            I find the positioning poor. They headline their prediction model and interactive apps based on their model, then publish articles saying "but this might be totally wrong". Which like, of course it could be, but they don't drive traffic and make money by being in the business of being "maybe wrong" often. All of that can be true, but that doesn't make it better.

            It just all feels gross. Sports stats is one thing, but this is politics, in which policy which impact peoples lives is at stake.

            They also walked back in 2012? When the national polling error was +3 D, approximately the same as the Trump polling error now, just for Obama instead? What changed now?

            Nothing. Idk what your point is here. I wasn't following them during that election and have no opinion. This feels like a weird "gotcha" and I don't know what you're getting at. Sounds like more of the same

            1 vote
            1. [2]
              stu2b50
              Link Parent
              A broken clock is still right twice a day. That's not evidence of anything in particular. They have actual metrics that they use to evaluate effectiveness. Fact of the matter is, they still have...

              I see no reason to lift 538 above anything else. Hell, some of the closer polls were their lower ranked ones. Especially in 2016.

              A broken clock is still right twice a day. That's not evidence of anything in particular. They have actual metrics that they use to evaluate effectiveness. Fact of the matter is, they still have the best polling averages, and the best polling ratings.

              I find the positioning poor. They headline their prediction model and interactive apps based on their model, then publish articles saying "but this might be totally wrong". Which like, of course it could be, but they don't drive traffic and make money by being in the business of being "maybe wrong" often. All of that can be true, but that doesn't make it better.

              I don't understand what you're saying, though. Yeah, they highlight their prediction model. Prediction model. And on the page for that prediction model, they tried extremely hard to get people to understand how it works. And how it works is that probabilistic events... are probabilistic. When someone has a 90% to win, that's 90%, not 100%.

              It just all feels gross. Sports stats is one thing, but this is politics, in which policy which impact peoples lives is at stake.

              I seriously don't understand how it's gross. There is not a single page on 538 that tries to mislead anyone; their frontpage election model says, and said (if you want to backdate with internet archive). What do you want them to do?

              The overall electoral environment favors Democrats, which is one reason they have decent odds of controlling the presidency, Senate and House (a 72 percent chance, according to our forecast). Of course, there’s always the chance of a polling error, which tends to be correlated from state to state when it happens. Trump needs a bigger-than-normal error in his favor, but the real possibility that polls are underestimating Trump’s support is why he still has a path to win reelection. A 10 percent chance of winning is not a zero percent chance. In fact, that is roughly the same odds that it’s raining in downtown Los Angeles. And it does rain there. (Downtown L.A. has about 36 rainy days per year, or about a 1-in-10 shot of a rainy day.)

              If people choose to ignore the words they write and play numbers, that's really their problem.

              6 votes
              1. cfabbro
                (edited )
                Link Parent
                And as anyone who has played XCOM can tell you, a 90% chance can often feel more like 10% when you only get one go at it, and is actually not that great of odds when the outcome going your way is...

                When someone has a 90% to win, that's 90%, not 100%.

                And as anyone who has played XCOM can tell you, a 90% chance can often feel more like 10% when you only get one go at it, and is actually not that great of odds when the outcome going your way is critical, and failure will be catastrophic.

                90% chances can feel like an absolutely sure thing to most people, but anyone who plays the odds frequently enough comes to understand pretty quickly that the 1/10 outcome is actually a lot less rare than most assume it to be. And hell, even getting that 1/10 several times in a row isn't all that rare when you play those same odds repeatedly.

                6 votes
  3. Kuromantis
    (edited )
    Link
    "Historical standards" for reference, according to the article linked above Presidential CYCLE PRIMARY GENERAL COMBINED 2015-16 10.1 4.8 6.8 2011-12 8.9 3.6 5.1 2007-08 7.4 3.6 5.4 2003-04 7.1 3.2...

    The margins by which the polls missed — underestimating President Trump by what will likely end up being 3 to 4 percentage points in national and swing state polls — is actually pretty normal by historical standards.

    "Historical standards" for reference, according to the article linked above
    Presidential
    CYCLE PRIMARY GENERAL COMBINED
    2015-16 10.1 4.8 6.8
    2011-12 8.9 3.6 5.1
    2007-08 7.4 3.6 5.4
    2003-04 7.1 3.2 4.8
    1999-2000 7.6 4.4 5.5
    All years 8.7 4.0 5.9
    State/Congressional level
    CYCLE GOVERNOR US SENATE US HOUSE COMBINED
    2017-18 5.2 6.0 4.1 5.1
    2015-16 5.4 5.0 5.5 6.8
    2013-14 4.4 5.4 6.7 5.4
    2011-12 4.8 4.7 4.7 5.1
    2009-10 4.9 4.8 6.9 5.7
    2007-08 4.1 4.7 5.7 5.4
    2005-06 5.0 4.2 6.5 5.3
    2003-04 6.1 5.6 5.4 4.8
    2001-02 5.2 4.9 5.4 5.2
    1999-2000 4.9 6.1 4.4 5.5
    1998 8.1 7.4 6.8 7.5
    All years 5.4 5.4 6.2 5.9

    Voters and the media need to recalibrate their expectations around polls — not necessarily because anything’s changed, but because those expectations demanded an unrealistic level of precision — while simultaneously resisting the urge to “throw all the polls out.”

    The polls aren’t getting more inaccurate

    Polling error in the FiveThirtyEight national polling average compared to the national popular vote margin, 1972 to 2020

    YEAR FINAL AVERAGE RESULT ERROR
    1972 R+24 R+23 1
    1976 D+1 D+2 1
    1980 R+2 R+10 8
    1984 R+18 R+18 0
    1988 R+10 R+8 2
    1992 D+7 D+6 1
    1996 D+13 D+9 4
    2000 R+4 D+1 5
    2004 R+2 R+2 0
    2008 D+7 D+7 0
    2012 D+3 D+4 4
    2016 D+4 D+2 2
    2020 D+8 D+4* 4

    Finally, there’s the fact that the election comes at a time of exceptionally high anxiety for the country. Between the pandemic and the election — and in an era when the media and other forms of expertise are constantly being challenged in both constructive and unconstructive ways — there’s not a lot to feel certain about.

    On that front, I’m afraid I have some bad news. If you want certainty about election outcomes, polls aren’t going to give you that — at least, not most of the time.

    It’s not because the polls are bad. On the contrary, I’m amazed that polls are as good as they are. Given that response rates to polls are in the low single digits and that there are so many other things that can go wrong, from voters changing their minds after you poll them to guessing wrong about which voters will turn out — plus the unavoidable issue of sampling error — it’s astonishing that polls get within a couple of points the large majority of the time. And yet, if a poll projects the outcome at 53-47 and it winds up being 51-49 (a 4-point miss), it will probably receive a lot of criticism — even, as we’ve seen this year, if it “calls” the winner right. It’s a fairly thankless task.

    The main reason that polls aren’t going to provide you with the certitude you might desire is because polls have always come with a degree of uncertainty. In a highly polarized era, most elections are going to be close — close enough as to exceed the ability of polls to provide you with a definitive answer. Say the final polling averages miss by a bit more than 3 points on average, as our forecast assumes. That means the margin of error is closer to 7 or 8 points. And every presidential election so far this century has fallen within that range.1 So if you’re coming to the polls for strong probabilistic hints of what is going to happen, they can provide those — and the hints will usually lead you in roughly the right direction, as they did this year. But if you’re looking for certainty, you’ll have to look elsewhere.

    2 votes