Rebecca Huntley’s Australia Fair has two striking virtues. It reminds us of how important the mobilisation of belief is in politics. And it is a tonic for those of us who might think the absence of any productive action on worrying social problems over the past decade represents not only political incompetence but also public indifference. Yet the Coalition’s win in the 2019 federal election raises profound questions about how to reconcile Huntley’s evidence of a persisting public commitment to social democracy and the apparent repudiation of such a program by the electorate.

Huntley sets out to clear the ground, exploring public opinion to show that it is not resistance to reform or withdrawal from democratic engagement that are at issue, but a perception that political institutions and elites are failing to heed the clear opinion of majorities on our signal challenges: climate change, housing, immigration and the treatment of asylum seekers. Underlying this is an argument that, if only the people at large were listened to, the Australian commitment to social democracy – government that intervenes when necessary to ensure services are delivered, fairness and relative equality are sustained and market failure is addressed – would be acted upon.

Hers is an optimistic essay, acknowledging in the closing pages persisting hurdles, but clearly framed both in relation to the strength of opinion on these key issues, and a conviction that the time for a progressive renaissance is at hand – and, she hoped, almost certain to be delivered at the 2019 federal election. She does not oversimplify, recognising that the task can only be undertaken by a Labor government, and noting that though majority opinion is moving in the right direction on climate change, for instance, the public is “only inching” towards realising the scale of the threat it represents. In fact, a Lowy poll published since the essay was released, during the 2019 campaign, registered for the first time that climate change had reached the top of the list of public concerns: had a tipping point been reached? Apparently not, if the recent election outcome is taken into account.

I wanted to believe that Huntley was right, and still hope her essay is widely read. Yet reading it in the midst of the 2019 election campaign – as the polls tightened – prompted me to confront some questions that it begs, to do with how beliefs are mobilised in politics.

The first is this: if one accepts Huntley’s analysis of the progressive zeitgeist, then why is it that the Coalition government was so deaf to public demand? Well, of course it came to power on the back of Tony Abbott’s war on “the great big tax” supposedly represented by emissions trading. It’s not easy to backtrack after that, even as public attitudes move on, so thereafter the Coalition vigorously defended the status quo. It was also undoubtedly influenced by coal industry donations, lobbying and the strategic placement of industry insiders in ministers’ offices, as has been well documented. But more important than all of that was that it was hobbled by wars over belief within the party. Huntley’s focus on public opinion – polls and focus groups – does not sufficiently attend to the beliefs of party insiders and the dynamics among party activists and supporters.

Three research studies are indicative of the problem. Huntley cites CSIRO research that supports her contention about the growing support for government action on climate change. That research also hints at the divergence of conservatives from the mainstream, since it notes that conservative voters are less likely to believe that climate change is caused by human behaviour and less likely to think government should do more to address the issue. An earlier study, in 2012, by Kelly Fielding and others at the University of Queensland, sharpens this differentiation. It shows that among politicians, political party affiliation and ideology have a powerful influence on climate change beliefs, since centre-left and progressive parties exhibit beliefs more consistent with scientific consensus about climate change than non-aligned or conservative leaders, and that motivated social cognition (that is, accepting only information that accords with existing views) is a powerful factor among conservatives. Finally, new research published by Anika Gauja (University of Sydney) and Max Grömping (Heidelberg University) in 2019 demonstrates that there are not only differences between party supporters and the rest of us, but also between party supporters themselves. The stronger party identification becomes, the lower the congruence between the views of supporters and the broader public. Parties now can be conceptualised as a series of concentric circles of increasing engagement but declining representativeness.

This clarifies the Coalition’s predicament over recent years. In an age when the shared commitments that sustained mass parties have evaporated, leaders are relied on to stand in for the party, to speak for what it represents. Their success in doing so is evaluated by constant polling. But as party membership gets ever smaller, and residual true believers increasingly diverge from the mainstream, leaders are trapped. Abbott, arguably, faithfully represented the views of his party’s most intense identifiers, and fatally lost public support. Turnbull instead spoke for what the public wanted – a more progressive program in general, including action on climate change – but each attempt to respond to popular demand provoked insurgency in the party room from those claiming (with some justification) to represent the beliefs of the party base. Has the task of satisfying both public expectations and party demands become impossible?

One might think that this is a problem only for the Coalition parties; that a change of government and the cold reality of Opposition might have forced the Liberals to adapt and to reform the party to recapture a broader, small “l” liberal constituency. That is to assume that the more progressive parties are immune from that dynamic of concentric circles, where party central turns out to be least in tune with the public at large. The research does not support that assumption.

Bill Shorten, as his equivocation during the campaign revealed, had his own difficulties in balancing the demands of some of those workers the ALP represents (in mining areas) with the public demand for climate action. Labor’s lead in the polls tightened as, on one side, Queensland unionists worried about their jobs and, on the other, progressive climate activists rued his inability to go as far as some wanted. And while progressive parties now might be more attuned to the general directions of public opinion, the intense identifiers among them are prone to characteristic mistakes. Among the Greens, for instance, self-righteousness and the conviction that “most people” agree with them, at least on climate change, renders them blind to something Huntley also identifies: the pragmatic temperament of the Australian electorate.

Thus, at a time when the millennium drought had made the public receptive to a climate change message, the sainted Bob Brown refused to accept any pragmatic compromise and helped to spike Labor’s first attempt at legislating emissions reductions in 2009 because it was not “good enough.” It set the stage for a decade of climate wars. And there he was again this year, leading the Adani protest convoy, apparently oblivious that a tactic that plays well in St Kilda is completely counter-productive in Clermont, where, as one resident said, “Up here, coal is our economy. It is … everything!” Yes, Adani must be stopped, but to ride into town with an injunction, yet offer no suggestion for how to manage the transition to an alternative economic future, simply provoked derision: “Those guys have taken time off from their barista jobs and unemployment to drive up here in fuel-guzzling cars. I just think it’s an insult, a slap in the face.” It was a gift to the hard right struggling to hang on in those areas, encouraging some to bet that the egregious George Christensen would hold his marginal seat of Dawson. In the event, the election saw a swing of 11.26 per cent in favour of Christensen, with similar swings in adjoining coal-belt seats.

One other thing niggled away at my wish to share Huntley’s optimism: reliance on what seem to be solid majority trends underestimates the way shifts at the margins can now be manipulated to destabilise “common sense.” We have seen in Donald Trump’s campaign and in Brexit how marginal and diverse minority opinion groups can be influenced through social media into aggregate coalitions of resentment and fear, leaching support away from commonly held views. The hired guns of opinion analysis have become experts at nudging belief to these ends. The Cambridge Analytica scandal was a stark instance. But our homegrown outfits working the same vein are no slouches. Crosby Textor’s role in British elections and the Brexit “leave” campaign was notable, and Scott Morrison mentioned Lynton Crosby as among the “experts” he “listened to” while campaigning. More worryingly, it is clear that the big spending and disruptive tactics of Clive Palmer’s UAP campaign, an overt instance of targeted messaging, were significant in influencing voting preferences to destroy Labor’s chances in Queensland, as Laura Tingle persistently reminded us on election night. He won no seats, not even a seat in the Senate, but note what he gained: leverage over a government that will likely facilitate exploitation of coal reserves in the Galilee Basin in which not only Adani, but Gina Rinehart and Palmer himself hold mining tenements.

Despite mulling over these concerns, I, like most, persisted in believing that the consistency of Labor’s apparent lead would ensure a win for progressives in 2019, but doubted there would be a landslide. It was not to be. And my concern about the disparity between public opinion and party insider belief underestimated the scale of the upset that eventuated. Labor’s dreams were smashed. There will now be many debates about why. The immediate question for Huntley is whether the disjunction between the election outcome (the repudiation of a progressive reform program) and her prior exposition of the Australian penchant for social democracy fatally undermines her argument. My conclusion is: not entirely.

Others will parse this question by looking closely at the differential clustering of opinion by age, geography and demography, to ask whether segmentation manifest in different regional voting patterns can explain how a very close election result can run counter to “national” opinion (as reported by Huntley). For my part, in trying to fathom the wreck of my own hopes, three things seem pertinent. First, one of my early mentors, the late Alan Davies, long ago explained the incoherence and inconsistency of our political outlooks, describing them as like a DIY project where we fashion a response “good enough” to satisfy a particular need, then put that aside until another challenge arises, when we might adopt something different, building up an assemblage of contradictory elements that we can draw on when prompted, but that we never squarely address (see his Skills, Outlooks and Passions, 1980). Different events will then elicit disparate responses (the question of a survey researcher on climate change, versus the task of deciding a vote, for instance). So it becomes possible for an individual to believe that action on climate change is needed, yet to vote for a party that shows little potential for action because a supervening belief (on sound economic management, for instance) is called forth.

Second, Chris Achen and Larry Bartels (Democracy for Realists, 2016) have shown convincingly that voting decisions are driven not by assessments of evidence and policy, but by group identity, emotion and a search for cues from those one regards as “people like us.” Thus, even Liberal supporters, well removed from the inner circles of party activism, closer to mainstream opinion and inclined to support climate action (Huntley notes that 60 per cent of Coalition voters are in this category), once in the voting booth will nevertheless succumb to the emotional pull of party identity and bridle at voting against “people like us.”

Third, one of the pioneers of opinion research, Walter Lippmann, nearly a century ago, warned us to be wary of “the phantom public” in a book of that title (1925). There is, he argued, no “public” out there waiting to be tapped; rather “publics” are created by political mobilisation, triggered by insiders for their own ends. They are emergent rather than stable entities, continually evolving in response to political action and representation. Thus Tony Abbott, always ready with simplifying binaries, articulates the crucial factor in how belief around climate change was mobilised in the 2019 campaign: “Where climate change is a moral issue, we Liberals do it tough. But where climate change is an economic issue the Liberals do well.” The responses Huntley records might well be construed as answers to a normative question: “What should we do?” But the actions of voters on the day can be thought of as an evaluation of economic interests. The Coalition, in successfully mobilising climate action as an economic issue, created a countervailing “public” to that which Huntley and others thought representative of the zeitgeist.

James Walter is emeritus professor of politics at Monash University. His latest book is The Pivot of Power: Australian Prime Ministers and Political Leadership, 1949–2016 (with Paul Strangio and Paul ’t Hart).