The Commonwealth Games Federation (CGF) struggled to find a host before Victoria volunteered in April 2022.
But the premier said the projected cost had now tripled and become “well and truly too much” for the state to bear.
CGF called the decision “hugely disappointing” and said it is “committed to finding a solution”.
The Commonwealth Games are a multi-sport tournament that take place every four years. They have only ever been cancelled during World War Two.
To be eligible to participate in the games, competitors must be from one of the Commonwealth’s 56 members. Most of the countries in the Commonwealth were once part of the British Empire.
Organisers had originally estimated the event – hosted across cities including Geelong, Bendigo and Ballarat – would cost A$2.6 billion (£1.4bn; $1.8bn) and the state government had billed it as a boost for the regions.
But now the 12-day tournament was expected to cost more than A$6 billion, Mr Andrews said, adding that the new figure was “more than twice the estimated economic benefit” it would bring to Victoria.
“I’ve made a lot of difficult calls, a lot of very difficult decisions in this job. This is not one of them,” he told a press conference.
