At the Television Critics Association press tour on Monday, Colbert opened up about his debut and shared that ten days before Letterman's exit, he asked if he could come to the show and hang out.
"We talked for an hour about how he approached every day for production and what the relationship with the audience was," Colbert said.
Colbert, 51, moved into his new space at the Ed Sullivan Theater, which is in the middle of renovations. The former Colbert Report host added that the theater is now a "very intimate space."
David Letterman Says Goodbye, Nikki and Ian's Bonus Cannes Honeymoon & More!
Along with the renovations comes another change audiences are sure to notice – the position of the host desk, a suggestion from Letterman.
One thing that won't change? The manual freight elevator Letterman used to get to set each night.
"[Letterman] said, 'Oh, it's the best thing in the building.' And he showed me how to run the elevator and then he showed me how to open the door so the elevator would be right there," Colbert said. "And then he said, 'There, now it's waiting for you.'"
While Colbert has big shoes to fill after Letterman's departure, he's excited and honored at the opportunity to takeover for the legendary host.
"He started his show in 1982 and I started college in 1982," Colbert recalled. "Dave was everything to us. I'm incredibly grateful and I had a chance to tell him how grateful I was for his example, for what he did for comedians for my generation, no matter what they were going to do."
Reporting by AILI NAHAS