Fortunately, he made up for lost time, putting together an extended tribute to their 38-year friendship.
Letterman, 67, recalled how the pair met at the Comedy Store in West Hollywood – Letterman said he stuck around one night to catch the very first set Williams ever did at the club, to "make fun of the new guy."
But as Letterman says, if he and the other comics there that night were like morning dew, Williams was a hurricane.
Williams's funeral is reportedly set to be held in San Francisco, the actor's hometown, although a date has yet to be set.
Watch the full clip of Letterman's tribute below.
The Williams family is asking well-wishers to send contributions to charities close to the actor's heart in lieu of flowers. Suggested organizations include St. Jude Children's Research Hospital, Challenged Athletes, USO, the Muhammad Ali Parkinson Center, the Christopher and Dana Reeve Foundation and Glide Memorial Church in San Francisco.
For more on Robin Williams's tragic death and his legacy of comic genius, pick up the latest issue of PEOPLE