Here’s a trick question on our national identity: how long after the arrival of the First Fleet did it take for the Australian population to become more white than black? Twenty, fifty or 100 years?

The answer can be all of the above, or none. It depends on the people and the settlement. And it depends on what is meant by the words “white” and “black.”

Europeans achieved their first majority in Van Diemen’s Land in the 1810s. But they did not do so on the continent until the late 1830s, once the boats began carrying more free migrants than convicts to the colonies. Even then, the European majority was restricted to the southeast. To the north and west, they were still outnumbered. Queensland remained predominantly Indigenous until the 1860s; Western Australia until the 1880s; and the Northern Territory until the 1950s.

This probably overstates the speed of the “white” ascendancy. I used a conservative estimate of the Indigenous population in 1788: 314,500. Others have put the figure at closer to a million. We don’t really know the starting point for the comparison. Nor is there any way to measure the people in between, those born locally with both migrant and Indigenous heritage.

These missing pieces in our colonial demography came to mind as I read Stan Grant’s powerful essay. As he points out, most Indigenous people today are descendants of immigrants: “By the post-war era, many people identifying as Aboriginal had mixed heritage: for me, Irish convict stock and a white grandmother of English/German background.” This got me thinking: how many “white” Australians with convict or free settler ancestors have “black” branches in their families of which they are not aware? The number need not be that large to make the point of a shared history. And the number is almost certainly much larger than most people imagine.

For the first century of Australia’s short European history, the “white” population was excessively male. Consider the NSW census of 1833. The total population was 60,794, of which 43 per cent were free men and women, 40 per cent were convicts, and 17 per cent were children under twelve. The children split almost 50/50 between boys and girls. But 67 per cent of the free adults and 89 per cent of the convicts were male. After almost half a century of “settlement,” the gender gap remained as in the First Fleet: three males for every female.

It doesn’t take much imagination to suppose that European males took local Indigenous women as sexual partners, whether by consent or rape. The alternative is to assume that the surplus “white” men took a vow of celibacy or took to one another, or that the secret history of “white” Australia was a culture of female control, with each woman having two or three blokes.

When Queensland separated from New South Wales in 1859, it did so as a free settlement, but the gender imbalance persisted. There is a telling passage on this score in the 1861 census. “As a general rule, it may be safely laid down that a great disproportion of the sexes is a great evil, particularly where the population is dense and collected in large masses,” the registrar-general, F.O. Darvall, wrote. “It however admits of some question whether this assertion is not weakened, and the evils considerably modified, by the peculiarly isolated condition of most of the inhabitants of the country portion of this Colony.” The evils to which he referred were those that had troubled every governor since Arthur Phillip: grog, violence and prostitution. Colonial authorities were on a perpetual mission to civilise the “white” male through the importation of single women. So why would the Queensland bush be seen as the exception to this problem, when European men outnumbered European women by around three to one, and Indigenous people were still the majority?

Stan’s essay reminds us that the White Australia Policy sought to erase that shared history. The Initial Conference of Commonwealth and State Aboriginal Authorities in Canberra in 1937 drew a colour line between the “half-caste” and other Indigenous Australians. “The destiny of the natives of aboriginal origin, but not the full blood, lies in their ultimate absorption by the people of the Commonwealth and it is therefore recommended that all efforts be directed in to that end,” the conference resolved.

Stan has dug out a remarkable quote to explain the official thinking. The “half-castes” were to be absorbed on the same terms as “the Greek and Italian migrants,” through the economy. That suggestion, from M.T. McLean, the Chief Protector of Aboriginals for South Australia, was probably meant as a compliment to the southern Europeans. Yet in the first half of the twentieth century, the Greeks and Italians were not quite “white” enough for Australia. Their numbers were restricted by conservative governments, and the Labor Party hounded those who were allowed to come. In the final week of the 1929 election campaign, Prime Minister Stanley Bruce addressed 3000 rowdy voters at the Prahran Town Hall in Melbourne. “When Mr. Bruce rose to speak there were shouts of ‘Importer of black-fellows’ and ‘The Dagoes’ friend,’” according to a report in the Hobart Mercury. My people only became “white” when the Vietnamese came in the 1970s. But we did it on our terms, as Greek and Italian Australians. Our newly acquired whiteness wasn’t burdened with the demand that we disown our parents. We became “white” at the same time that multiculturalism replaced assimilation in immigration policy.

This is a key point that is misunderstood by conservatives obsessed with empowering bigotry in the name of free speech. Multiculturalism didn’t divide the country; it saved it from tribalism. Imagine the division today if the postwar migrants and their local-born children had been told by Whitlam, Fraser, Hawke and Keating that they would continue to be treated as second-class citizens unless they renounced their heritage; if the groups with the highest rates of home ownership, with the best results in the classroom, and who were dominating the professions and industry were told they weren’t really Australian.

This is the paradox I kept returning to as I read Stan’s essay. He traces the rise of an Indigenous middle class that echoes the journey of the migrant. Yet there has been no equivalent change in Indigenous policy. Assimilation remains the default setting. “Black” means disadvantage; “white” means privilege. If an Indigenous Australian enters the mainstream, the question of colour becomes an accusation. You can’t call yourself “black” if you’ve made it in the “white” world. You certainly can’t ask government to change policy, let alone surrender the levers of paternalism to allow Indigenous Australians to decide what’s best for their communities.

Then again, if an Indigenous Australian makes the footballer’s journey from outcast to elite, they have to remain a certain type of “black” – humble, exceptional. They have to stand out as individuals in a team sport, their talent unsullied by training or game plans. When they kick a goal, they can’t raise a defiant fist to the stand to rally their supporters and humiliate the other side; that is the domain of the white warrior.

Adam Goodes was a victim of this double standard. He had every word he uttered, every step he took on the sporting field, censured. His accusers claimed the right to free speech, and insisted he was obliged to listen. The booing increased in 2014, after he became Australian of the Year, and he was hounded through his final season for the Sydney Swans in 2015. Goodes had won two Brownlow Medals and played in two premierships. But those achievements had raised no ire in the stands. Footy fans only took offence after he spoke.

When they booed, Stan heard “a howl of humiliation” aimed at his people. We’ve talked about this often, reversing the normal discussion that occurs between league and footy fans. Not which game is better (sorry, Stan, it’s Aussie Rules) but which spectator base is the less tolerant – the supposedly redneck fans in league’s northern states, or the supposedly cultured folk who watch footy in the southern states? In other words, why did this happen at a game of Aussie Rules, and not rugby league? The answer lies, in part, in our colonial demography.

Here’s a dirty secret of the tolerant southern states: they have little shared history with Indigenous Australia. The free migrants of Victoria and South Australia overran the “black” population within a decade of arrival, whereas it took half a century in New South Wales and Queensland. While Indigenous blood was spilled across the continent, in the rugby league states it was also decidedly more mixed.

Political leadership also matters. Twenty seasons before they started booing Adam Goodes, another shy Indigenous Australian footballer spoke out against racism and was greeted as a hero. Why was Michael Long able to do so in 1995? I suspect Paul Keating had something to do with it. When Long demanded action from AFL officials after an opponent called him a “black c—,” he did so in an era that offered the promise of reconciliation. Keating had given his Redfern address at the end of 1992, and parliament had passed the Native Title Act twelve months later. Between these two landmarks, another Indigenous footballer, Nicky Winmar, had already made his name. They booed Winmar in 1993, but he lifted his jumper and pointed to his chest to say, “I’m black and I’m proud.” Keating praised Winmar; the prime minister could have avoided the topic, but then he would have been just another politician.

Goodes was caught in the moral vacuum of Tony Abbott’s government. This prime minister said nothing when it counted, giving license to conservative commentators to tell Goodes that he had brought the booing on himself.

With hindsight, Keating, not Abbott, was the exception. Keating was the last prime minister with an uncomplicated view of racial politics. He saw no point in appeasing the fringe; on the contrary, he understood that if he looked the other way when someone played the race card, whether on his own turf of politics, or against someone like Long on the footy field, he would lose his authority and Australia would become a little less cohesive.

Racism did not die with multiculturalism. What changed under the governments of Whitlam, Fraser, Hawke and Keating was the way the outsiders viewed themselves. They were told they belonged.

In the twenty years since Keating lost office, politicians have tried to accommodate racism, not out of any shared belief, but out of electoral panic. Yet the proportion of voters that can be motivated by calls to restrict immigration or to deny basic rights to Indigenous people has never been that great. As a rough rule of thumb, it sits somewhere between the 9 per cent who voted “No” at the 1967 referendum to allow Aborigines to be counted as part of the national census and the 4 per cent who voted for Pauline Hanson’s One Nation in the Senate in 2016. 

What is striking about Hanson’s second coming is the Australians who trouble her. Her beef with multiculturalism is no longer with the Asians. We are now in danger of being swamped by Muslims, she says. But she hasn’t shifted her position on Indigenous Australians. She still thinks they are claiming special benefits not available to ordinary Australians. It is too easy, she says, to call yourself an Aboriginal. She wants a debate on what defines an Aboriginal.

What was shocking about those particular comments was the absence of a comeback from Malcolm Turnbull or Bill Shorten. Then again, Hanson was merely reinforcing the status quo – a “white” politician wanting the final word on what it means to be “black.” But I can’t help wondering how the Queensland senator would feel if she found an Aboriginal ancestor in her family tree.

George Megalogenis’s book The Australian Moment won the 2013 Prime Minister’s Award for non-fiction and formed the basis for his ABC TV series Making Australia Great. His most recent book is Australia’s Second Chance. He is the author of two Quarterly Essays, Balancing Act and Trivial Pursuit, recently re-published in a single book.