Asian stocks extended a global selloff as the outbreak in China, which has killed 106 people and spread to many countries, fuelled concern over the damage to the world's second largest economy - an engine of global growth.
"There are concerns over the impact to the global economy from the spread of infection in China, transportation disruptions, cancellation of group tours from China and an extention in the Lunar Holiday," Nishimura told a news conference after a regular cabinet meeting.
"If the situation takes longer to subside, we're concerned it could hurt Japanese exports, output and corporate profits via the impact on Chinese consumption and production," he said.
China is Japan's second largest export destination and a huge market for its retailers. The Chinese make up 30% of all tourists visiting Japan and spent nearly 40% of the total sum foreign tourists used last year, an industry survey showed.
The outbreak could hit Japanese department stores, retailers and hotels, which count on a boost to sales from an inflow of Chinese tourists visiting during the Lunar Holiday.
Automaker Honda Motor, which has three plants in Wuhan, the capital of Hubei province and the epicentre of the outbreak, plans to evacuate some staff. Aeon will close its shopping malls in the city until Thursday.
Economists at SMBC Nikko Securities estimate that if a ban China has imposed on overseas group tours lasts another six months, it could hurt Japan's economic growth by 0.05%.
Some expect the potential damage could be much worse.
Hideo Kumano, chief economist at Dai-ichi Life Research Institute, said the decline in tourists from China could hurt Japan's GDP growth by up to 0.2%.
"The biggest worry is the risk the negative impact from the outbreak persists and hits (the economy) during the Tokyo Olympic Games," when a huge number of Chinese tourists are expected to visit Japan, he said.
"If the number of visitors decrease rather than increase, the hit to Japan's consumer industry will be quite large."
Japan will host the 2020 Olympics in July and August.