Labor claimed the eastern Melbourne electorate of Aston - the first time a federal government has won a seat from the opposition at a by-election in 103 years.
Mr Dutton says the party will go through some soul-searching.
"We have a lot of work to do, I accept responsibility as I'm the leader of the party," the coalition chief told the ABC's Insiders program on Sunday.
"Now, the question is how we rebuild from here, the policies that we have, the brand rebuilding that we need to do in Victoria.
"I can tell you it makes me more determined to rebuild this party and be in a winning position by 2025."
Mr Dutton argued state issues also played into the 6.5 per cent swing to federal Labor, given the dominance of the state Labor government, while noting that Melbourne seats had always been tough for the Liberals to win.
"Obviously, the difficulties for us in Victoria haven't germinated in Aston over the course of the last five weeks," he said.
"Even back to 2013, with all of my predecessors, Victoria is the one state that we have never held a majority of seats in and there are huge issues at a state level as well."
Senior Liberal frontbencher Jane Hume said cost of living pressures remained the number one issue with voters but other factors meant the party didn't cut through at the by-election.
Liberals have chastised Labor for running a "dirty" campaign on the ground and are also arguing voters blamed them for having to go to the polls for the third time in a number of months.
"You will never put it down to one thing, otherwise, we would have fixed it," Senator Hume told AAP.
"There was a confluence of events and timing. Even with a great candidate, we couldn't overcome that confluence."
The by-election was called after controversial former Morrison government minister Alan Tudge, who held the seat on a margin of 2.8 per cent, resigned from politics.
Federal National leader David Littleproud said Mr Dutton remained the best person to lead the opposition.
He also blamed the by-election loss on Liberal party infighting in Victoria over the fate of a state MP who attended an anti-transgender rally where neo-Nazis were also present. Moira Deeming was suspended from the party room for nine months.
"The Victorian Liberal party did everything they could locally there to trash their own brand," Mr Littleproud said.
Aston Liberal candidate Roshena Campbell was also criticised for not living in the electorate.
Mr Littleproud said there were lessons for the Liberals.
"The candidate probably not being a local shows that we have got to respect the local electorate and understand that they want someone local," he added.
Senator Hume also backed Mr Dutton as leader.
"I cannot imagine there would be any mood in the party room for (a change of leader)," she said.
Mr Dutton said he led a united team, which was hard from opposition after a federal election loss.
"Parties always tear themselves apart in opposition. We haven't gone through that period of self-destruction," he said.
"I have a leadership style which I believe they appreciate which is why people very strongly are expressing their support to me."
Independent MP Zali Steggall said the Liberals needed to resist the urge to go further to the right after the loss.
"It's a sign that small 'l' liberals and moderate liberals are rejecting the idea of this extreme right conservative ideology," she told Sky News on Sunday.
"I don't think the opposition is coming to grasp with that."
22°C