Panetta also urged Pakistan to go after the Haqqani militant network, one of the United States' most feared enemies in Afghanistan, and said Washington would exert diplomatic pressure and take any other steps needed to protect its forces.
"It is an increasing concern that safe havens exist and those like the Haqqanis make use of that to attack our forces," he said.
"We are reaching the limits of our patience for that reason. It is extremely important for Pakistan to take action to prevent (giving) the Haqqanis safe havens, and for terrorists to use their country as a safety net to conduct attacks on our forces."
Panetta, who left Kabul shortly afterwards, blamed the group for an attack last week on a U.S. base in the east in which several insurgents, including some wearing suicide vests, attacked it with rocket propelled grenades.
The attack was foiled, but it underlined the challenge facing Western and Afghan forces in the east where insurgents take advantage of the steep, often forested terrain and the Pakistani border to launch attacks and then slip across the border.
"What happened the other day in Salerno is an indication that they are going to continue to come at us and let me be clear anybody who attacks US soldiers is our enemy and we are going to take them on. We have got to be able to defend ourselves," he told U.S. troops earlier at Kabul airport.
The comments came as Washington appears to be looking to other allies in the region for help in the face of Pakistan's foot-dragging. Panetta arrived in Kabul after a visit to India, Pakistan's old enemy, where he urged New Delhi to take a more active role in Afghanistan.
NATO has signed an agreement with three countries to the north of Afghanistan for land routes as the U.S.-led alliance begins a withdrawal of its forces from the country next year.
NATO chief Anders Fogh Rasmussen said earlier this week the "reverse transit" deal was signed with Uzbekistan, Kazakhstan and Kyrgyzstan.
Pakistan closed the shorter and cheaper routes through its territory last year to protest a cross-border NATO air attack that killed 24 Pakistani soldiers. Discussions to reopen the Pakistan routes have stalled.
Other irritants in U.S.-Pakistan ties include drone attacks in the lawless areas of Pakistan near the Afghan border where several militant groups operate but seen by many Pakistanis as a violation of sovereignty.
Washington says the attacks are crucial to rooting out militancy and four days ago, a U.S. drone strike in northwest Pakistan killed al Qaeda's second-ranking leader, Abu Yahya al-Libi. It was the biggest blow to the militant group since the killing of bin Laden.
(Writing by Michael Georgy and Sanjeev Miglani; Editing by Raju Gopalakrishnan and Ed Lane)