LONDON (AP) — A defiant British Prime Minister Boris Johnson was in a stubborn standoff with his frustrated Conservative Party on Saturday, after two thumping special election defeats renewed calls for a major change of government direction.

Johnson dismissed the defeats as predictable mid-term blues, and said people expecting “some sort of psychological transformation” from him should know “that is not going to happen.”

The prime minister told the BBC said he believed “voters are heartily sick of hearing about me and the things I’m alleged to have done wrong. What they want to hear is what we’re doing for them.”

“What I want to do is to get on with changing and reforming and improving our systems and our economy,” he said.

Despite the bullish tone, this is a perilous moment for Johnson, who is due to be out of the U.K. until Thursday at a Commonwealth meeting in Rwanda, a Group of Seven leaders’ gathering in Germany and a NATO summit in Spain.

Simmering concerns about Johnson’s ethics and judgment have been brought to a boil by Conservative defeats in two special elections this week. The party lost the rural southwest England seat of Tiverton and Honiton to the centrist Liberal Democrats, and was defeated by the left-of-center Labour Party in the post-industrial northern town of Wakefield.

The results alarmed many in the party, who fear Johnson’s electoral magic — his rare ability to appeal both to affluent traditional Conservatives and working-class former Labour supporters — has been erased by ethics scandals and a cost-of-living crisis.

Conservative Party chairman Oliver Dowden quit after the election results Friday, saying the party “cannot carry on with business as usual,” and two former Conservative leaders said it was time for Johnson to go.

Michael Howard, who led the Conservatives between 2003 and 2005, said “the party, and more importantly the country, would be better off under new leadership.” William Hague, who was Tory leader from 1997 to 2001, said the results showed the party was “potentially heading towards a disaster” unless it changed its leader.

“Cabinet members need to steel themselves to do that,” he said.

Johnson led the Conservatives to a big election victory in December 2019 on a promise to “get Brexit done” after years of rancor about Britain’s decision to leave the European Union.

His government has been beset by difficult post-Brexit relations with the EU, and now by the worst cost-of-living crisis in a generation, with Russia’s war in Ukraine squeezing supplies of energy and food staples at a time of soaring consumer demand while the coronavirus pandemic recedes.

Johnson’s popularity also has been eroded by months of ethics allegations, culminating in a scandal over parties held in government buildings while millions of others were banned from meeting friends and family during coronavirus lockdowns.

Johnson was one of 83 people fined by police for attending the parties, making him the first prime minister found to have broken the law while in office. A civil servant’s report on the “partygate” scandal said Johnson must bear responsibility for “failures of leadership and judgment.”

Johnson survived a no-confidence vote by his own party this month but was left weakened after 41% of Conservative lawmakers voted to remove him. Under party rules, Johnson can’t face another such vote for a year.

That could change. No-confidence votes are overseen by a powerful party committee, and Johnson opponents say they will push to alter the rules. The Conservatives have a history of ousting leaders they consider liabilities. Johnson’s predecessor, Theresa May, also won a no-confidence vote in December 2018 — but internal party pressure forced her to step down six months later.