Deputy Chief David Carter blamed poor record keeping over the past 15 years for the missing 850 keys but said there's no indication they have been used for criminal purposes.
"Our accountability isn't the best here," he said.
Some of the missing keys may have been damaged and replaced and others may have been changed hands when firefighters retired, Carter said.
"We've probably not done a good job in tracking whether that key was returned back to us," he said.
The Fire Department keeps the keys in engines and other vehicles and gives them to certain personnel so they don't have to break into stores, office buildings, apartment complexes and other sites to gain entry.
The lost keys open lock boxes. The lock boxes contain other keys that open doors of individual businesses or other buildings.
Carter said the department will replace the locking mechanisms of the 9,000 lock boxes at establishments using the system in Phoenix. Department spokeswoman Shelly Jamison said Phoenix set aside $50,000 for the work and that the lock-box company, Phoenix-based Knox Co., will pay for any additional costs.
Jamison did not have a total figure, and Knox spokesman Keith Wolski referred all cost questions to Phoenix fire officials.
Carter said the department recently took stock of its keys after learning of a 2013 criminal case in Seattle in which a man was accused of breaking into similar lock boxes in that city to burglarize apartment complexes.
Prosecutors in that case said in court papers that the Seattle Fire Department was rekeying boxes at a cost of over $500,000, KIRO-TV reported in October.
Carter said the Phoenix Fire Department will have employees install new locking mechanisms in the boxes.
"We're the custodians of these keys. We understand the businesses place a lot of trust in us," he said. "We need to maintain that trust with the business community."
Todd Sanders, president of the Greater Phoenix Chamber of Commerce, said the Fire Department's inability to account for the keys was a concern but that it was encouraging that officials had discovered the problem and were taking action.
KPNX-TV initially reported Monday that the keys were missing.
Christopher Laski / AP