Theresa May, Britain’s prime minister, has joined a long line of politicians who have gambled that they understood the populist wave overtaking Western politics and lost.
Thursday’s election capped a year in which the latest theory of politics in the populist era perpetually seemed to prove incorrect, as did many predictions of election outcomes. What explains this seeming inexplicability?
Populism’s individual effects, after all, have become well known. Voters oppose party establishments, scramble demographic coalitions and are more motivated by what they oppose than by what they support.
The problem is that, even among leading scholars, how these factors interact in any given election is still poorly understood.
The changes are simply too complex and too new.
Everyone knows that populism has fundamentally altered the rules of Western politics. But no one has deduced what the new rules are.
The result is that politicians and observers enter each election, whether they know it or not, merely guessing. Miscalculations and surprises have become the new normal.
That has been most pronounced in Britain, where politics are playing out along an issue, “Brexit,” that has nearly 50-50 support, so that even slight miscalculations can swing elections, with global consequences. But the phenomenon extends across Western democracies.
Voters are not waiting for scholars to understand their changing behavior, and neither can politicians. As leaders try to reckon with forces they cannot fully comprehend, uncertainty and volatility are, for now, built into the system.
A Year of Failed Gambles
Mrs. May was acting on the best and most current understanding of populist voters and how to galvanize them, which shows how little populism is understood.
The previous year of elections had seemed to demonstrate that championing Brexit and promising to clamp down on terrorism by curtailing human rights laws would lead her Conservative Party to victory.
That voters did not simply hand her a defeat, but shifted to the center-left Labour Party when it is led by a left-wing populist, Jeremy Corbyn, once again confounds today’s understanding of the populist moment.
She was hardly the first politician to succumb to a false belief that she understood populism. David Cameron, her predecessor, held last June’s referendum on leaving the European Union believing it would co-opt nativist sentiment that was seen as too marginal to prevail.
In the United States, Republican Party leaders, and by some accounts even Donald J. Trump’s own campaign, all underestimated the anti-establishment swell that pushed him to the presidency in the 2016 election.
And in the Netherlands this year, the center-right prime minister, Mark Rutte, sought to contain populism by co–opting some of its harsh attitudes toward immigrants, but went on to lose one-fifth of his party’s seats in Parliament.
Some surprises have bolstered establishments, underscoring that the only certainty is uncertainty.
In Dutch elections, the far-right leader Geert Wilders fell significantly short of expectations, suggesting that even populists are struggling to harness populism.
In France, after the far-right candidate, Marine Le Pen, won more votes in the presidential race than her party ever had before, many expected the party would gain in coming legislative elections. But polls suggest it is heading for catastrophic defeat.
Instead, French voters, in a move anticipated by no known models of politics in the populist era, appear to be flocking to Emmanuel Macron’s party, which is somehow both pro- and anti-establishment: newly founded, but embracing centrist, globalist politics.
In Germany, the biggest shift in voter support, though short-lived, was from the center-right party, led by Chancellor Angela Merkel, to the establishment center-left.
In nearly every case, expectation and reality have clashed, often in totally new ways.
This matters beyond elections. Leaders must now govern societies reshaped by forces that are barely understood.
Finding a New Theory
Politicians can hardly be blamed for their uncertainty. While scholars have advanced in understanding populism, they remain far from a theory that can fully explain or predict an election.
Matthew Goodwin, a University of Kent professor who studies radical politics in Britain, wrote last month on Twitter: “I do not believe that Labour, under Jeremy Corbyn, will poll 38%. I will happily eat my new Brexit book if they do.”
On Thursday, as the exit polls showed Labour winning 40 percent of the vote, Mr. Goodwin tweeted, “Anyone got ketchup?”
Pollsters have not fared much better. While they know they must adjust their models, they can hardly devise a formula to account for dynamics that are rife with unknowns.
In the British election, most polls predicted a Conservative victory. YouGov drew mockery for predicting a hung Parliament — accurately, it turned out.
Ben Lauderdale, who helped design YouGov’s model, acknowledged that his colleagues were feeling their way through uncertain terrain, saying even he had been surprised at its accuracy.
“Party loyalty really was scrambled to a greater degree than usual,” Mr. Lauderdale said, citing one of several factors that have confounded pollsters. “There were some people switching one way and other people switching another way.”
Conventional wisdom has also not kept up with changes, Mr. Lauderdale said, pointing to the belief in both parties that Mr. Corbyn was a fringe figure who had little hope of appealing to mainstream voters.
Political establishments, which have their own stubborn subcultures, have held on to conventional wisdom even as it has led them astray — for instance, by discounting Mr. Corbyn’s potential.
But those establishments have also been reactive, drastically recalibrating their expectations after each surprising new election, imposing a new theory intended to make sense of the world. Those theories inevitably survive only until the next election, but in the meantime shape polling models, news media coverage and party behavior.
Few systems in the world are more complex than the one between a person’s ears, particularly when he or she is choosing how to vote. It is both a political and social act, governed by a host of factors that are broadly known but interact in profoundly complicated ways.
It does not always seem that way. When issues and candidates fit a broadly similar mold year after year, voting tends to follow predictable patterns. But populism has introduced a new, or at least newly powerful, set of factors, forcing anyone who works in or studies politics to throw out long-held models.
For one, anti-establishment sentiment is scrambling parties and their agendas.
Britain’s two leading parties each ran against the establishment from within it, Mrs. May by presenting herself as the champion of Brexit, and Mr. Corbyn by rejecting Labour’s decades of centrism.
It remains unclear whether anti-establishment voters, in this election or others, tend to choose the most anti-establishment platform, the most outside-the-system leader or the opposite of whoever is in power.
Another factor: Parties are weakening, but polarization is strengthening. Voters increasingly see themselves as voting against the party or person they dislike, rather than for one they do like.
Sometimes that leads people to abandon mainstream parties in droves, as happened in the Dutch election.
Other times, polarization leads voters to support one mainstream party because that is the best way to oppose another. In Britain, Labour and the Conservatives have grown more dominant than they have been in years.
These changes have been exacerbated by tectonic shifts in demographic coalitions. Working-class whites are shifting from left- to right-wing parties, driven by a backlash against economic and demographic change. Voters are becoming likelier to split by age and education. Such changes are forcing parties to drastically alter their platforms and strategies.
But scholars are only beginning to understand these new coalitions, which are motivated by crosscutting issues.
For instance, after the British and American elections, it appeared that older voters had shifted significantly to the right. Yet in France, Ms. Le Pen underperformed with those voters relative to other age groups. It is unclear whether the new conventional wisdom about older voters was wrong, or if some other factor influenced them in France.
This is just a sampling of the factors altering politics. Any one of them would take years to fully understand, but together, they have confounded even the most seasoned professional’s ability to anticipate politics.