The NSW Independent Planning Commission said "the site is not suitable for the development" because the Ravensworth Homestead complex, comprised of colonial buildings and historic gardens in an agricultural setting, would fall in the middle of the expansion project.
The commission said the mine expansion would not be in the public interest because "the removal of the complex and associated heritage from the site would constitute a significant loss to future generations".
Glencore, which operates 17 coal mines in Australia making it the largest coal producer in the country, had proposed to continue extracting 135 million tonnes of coal by 2045, near Singleton in the Hunter Valley.
The company said it would have provided job opportunities for up to 690 people and add $282 million in royalties to the state government.
The commission conducted a site inspection, held a public hearing in March 2022 and received nearly 900 written submissions, including over 280 objections.
In its 51-page decision, which has been celebrated by the Greens and environmental groups, the commission noted the Ravensworth complex dating back to 1832 is of particular significance because of "its association with frontier conflict between European and Aboriginal people".
The Plains Clan of the Wonnarua People (PCWP) said massacres of Aboriginal people at the hands of European colonists in the 19th century had taken place in the area near the complex.
But Glencore disputes the Plains Clan claims saying in a September statement that a massacre of Aboriginal people in 1826 had occurred 20 kilometres away from the Homestead and six years before it was built.
AAP contacted Glencore and the Wonnarua Aboriginal Corporation for comments on Friday's decision.
NSW Greens MP Sue Higginson hailed the decision as a "powerful victory for the Wonnarua people" and the fight against climate change.
"While communities across NSW continue to suffer under climate disasters, it's shocking that any new coal or gas mines are even under consideration," she said.
The Environmental Defenders Office representing Wonnarua Traditional Owners Scott Franks and Robert Lester in lodging complaints with corporate watchdogs against Glencore described the IPC ruling as an "incredible result... for all Australians who value First Nations cultural heritage".
"It would have desecrated Wonnarua lands and destroyed a place of profound significance," the group said.
"Our clients are very relieved the IPC has rejected this proposal."
Even though the commission acknowledged the project would provide"economic and social benefits" to the region and that greenhouse gas emissions and biodiversity concerns could have been worked around, the overriding factor in their refusal was intangible cultural heritage loss.