"I had a great time," the star, 77, told PEOPLE just after announcing his decision to depart. "I had a blast. I wish I could continue."
Despite coming in with the lowest score just before the first elimination, the fans voted to bring Williams back for another week of competition.
"I'm sorry that I'm not going to be around, but my back just can't handle it," he said. "My health is more important."
Williams's partner Emma Slater is in full support of the actor's early exit from ABC's hit show.
"I knew the back was a problem for Billy," she told PEOPLE. "We were already kind of taking it slowly, but for me, surprisingly, this decision wasn't as difficult as it could have been because it really was the right thing to do. We made peace with that. We more directed it as a celebration of the time we've had together."
Williams, who attributes his back problems with years of film stunts, doesn't have regrets.
"I can't say it's not worth it," he said. "I did what I did and I had a good time and made a lot of money."
Although he complained mid-interview that his "back is really hurting," he explained that he's "been advised not to do surgery," adding, "I don't know what I'm going to do about it."