It was all for a good reason: He's lost over 120 lbs.!
"I'd never worn a Speedo in my life!" Smith tells PEOPLE. "When you feel fit, you don't want to wear any clothes and let it all hang out."
Smith says that he was in the best shape of his life while filming Masters in Las Vegas – a fact that helped him cope with the extreme climate.
"Filming in 110-degree heat, I never could've done it," he said emphatically. "I'd run 16 miles [a day] in the desert in Las Vegas."
Smith attributes his success to trainer Az Ferguson, who helped him prep for two marathons in 2010. Smith no longer does "crazy" long runs.
"I'm 52, I'm no young chicken," he said. "I need my legs and my knees for my restaurant."
Instead, he's shifted his focus to weight training and boxing with Chicago-based trainer Joey Thurman.
It's still challenging for Smith to maintain his healthy weight, and he admits that he doesn't always enjoy his signature dishes, including fried chicken, as much as he used to – though he still occasionally eats chicken and beef.
"You can change your life with a salad," He says.
His motivational motto: "I found that no food tastes as good as what it feels like to be healthy."