The 72-year-old Suga, long-time right-hand man of former premier Shinzo Abe, has seen his support sink to just over 30% - traditionally considered a danger line for Japanese leaders - from around 70% when he took office last September.
Suga took over after Abe, citing ill health, ended a tenure that lasted nearly eight years and made him Japan's longest-serving premier. Before that, Japan went through six prime ministers in as many years, including Abe's own troubled first one-year tenure.
Suga's dream scenario was to contain the virus outbreak, preside over a successful Games and call a general election. That has been upended after a recent surge in COVID-19 infections led to a fourth state of emergency in Tokyo and forced Olympic organisers to ban spectators from almost all venues.
"He's not doing a good job at handling the party and policy, and nobody likes him being in power," said Steven Reed, a professor emeritus at Chuo University. "All they need is an alternative."
New infections in Tokyo surged to a nearly six-month high of 1,308 on Thursday and medical experts have sounded the alarm. Japan's mostly voluntary restrictions have failed to curb the movement of people that can spread contagion.
Efforts by Economy Minister Yasutoshi Nishimura to get bars and restaurants to stop serving alcohol as part of anti-COVID-19 measures backfired and caused a public outcry.
Nishimura, Suga's point man on pandemic response, was forced to apologise and withdraw requests for banks to pressure establishments that failed to comply with the alcohol request and for liquor wholesalers not to supply such eateries.
Japan's vaccination campaign was also initially slow and is now facing supply bottlenecks, adding to dissatisfaction.
Japan has not suffered the explosive outbreaks seen elsewhere but has recorded nearly 830,000 COVID-19 cases and about 15,000 deaths. Only 31% of the public have had at least one shot.
FACE OF ELECTIONS
For Suga's Liberal Democratic Party, his biggest failing is an inability to win elections.
The LDP lost three parliamentary by-elections in April, and this month the party and its allies fell short of a majority in the Tokyo assembly. The LDP gained far fewer seats than it expected in that vote, seen as a bellwether for national elections.
"The consensus among the LDP had been that unless the LDP saw significant losses (in the Tokyo election), Suga would be prime minister until next spring," said a senior bureaucrat with deep knowledge of the party.
Now, "people in the party are mulling how to replace him," the bureaucrat said on condition of anonymity.
No incumbent LDP heavyweight has publicly called for Suga to be replaced.
Suga won the LDP leadership after all major factions united around him. But he lacks his own strong base, and party divisions have deepened since he took office.
His term as LDP president expires in September, although there has been talk of postponing the party vote until after an election for parliament's powerful lower house. The general election must be held by November.
Ditching an incumbent premier is tough and the absence of an obvious successor makes it harder. Nor are there signs Japan's powerful business lobbies are unhappy with Suga.
"If it were obvious who would replace him, he'd probably be in more trouble, but who is the party going to coalesce around?" said Tobias Harris, a senior fellow at the Washington-based think-tank American Progress. A divided opposition and low turnout could also limit LDP losses in the lower house vote.
"Is there any reason to think turnout will be better than the last couple elections? So how bad will LDP losses be?," Harris said. "This seems to be the 'new normal'."