"Some people were teary. Many people didn't know it ahead of time," a source tells PEOPLE. "People are really in shock. People can't believe it. We always thought it was gonna happen – just like that."
The show's star guest also got caught up in the moment, the source said of the scene before Thursday's Late Show taping, when Letterman, 66, said he will leave his late-night talk show in 2015.
"Johnny Depp was there and he told Dave, 'You have some weepers back there.' Some people were upset."
But they were also rooting for their boss, who marked three decades on air and announced his exit with gratitude and his trademark humor.
"I just want to reiterate my thanks for the support from the network, all of the people who have worked here, all of the people in the theater, all the people on the staff, everybody at home, thank you very much," he told stunned viewers. "We don't have the timetable for this precisely down – I think it will be at least a year or so, but sometime in the not too distant future, 2015 for the love of God."
Said the source: "It was sad but also sweet and honorable the way he did. Everyone there wants the best for Dave."
Reporting by LIZ McNEIL