In a report, they warned there was “a risk the movement of people” would spread Covid infections and “strain the [country’s] medical system”.
But other Japanese officials have indicated they want domestic fans to attend if possible. Foreign spectators have already been banned.
The Olympics, originally meant to happen last year, will open on 23 July.
The report by the Japanese medical experts, led by top health adviser Shigeru Omi, was issued on Friday.
It said that holding the Games without spectators was the “least risky” option and the desirable one.
The president of Japan’s organising committee, Seiko Hashimoto, agreed that a “no fans” approach would reduce infections.
But she added that she would still explore ways to allow spectators.
The final decision is expected on Monday.
In a separate development on Friday, officials and volunteers working on the Games started receiving vaccinations.
A state of emergency in Tokyo is set to be lifted on Sunday, but polls in local media suggest public scepticism to the Games going ahead remains high amid a slow vaccine roll-out.
Only about 16% of the country’s population have received one Covid-19 dose so far, according to Reuters news agency.
The vaccine doses allocated for Games officials have been supplied by Pfizer as part of a deal agreed with the International Olympic Committee and the Japanese government.
Those being vaccinated include volunteers and staff working at the athletes village and members of the media covering the Games.
“Now that I will be vaccinated, I will feel a little more reassured doing my job,” Chika Hirai, director of doping control for Tokyo 2020, was quoted by the AFP news agency as saying.