"I'm a straight-up sort of person," Williams told New Zealand's TV3. "If I had known something was going on, the whistle would have been blown."
Though Woods hasn't spoken to Williams since his accident – and subsequent scandal – in November 2009, Williams has been personally affected by his employer's indiscretions.
"Of course I'm mad at him, why would you not be?" Williams said. "I'm close with his wife and he's got two lovely children and he's let them down."
But that doesn't mean Williams doesn't still stand by his boss. "When a guy's having a tough time, it's not up to me to beat him with a stick right now," the caddy said. "He's getting enough grilling from everybody else."
"Tiger's one of my closest friends and he needs my support right now," Williams continued, "and I'd never think of walking away."
Woods, 34, who is reportedly home from his Arizona rehab, has been spotted hitting golf balls at a range in Isleworth, Fla., near his home. But Williams says Woods has informed him that he is ready to return to competition.
For his part, Williams says he doesn't want to dwell on the past. "When I [will] talk to him, I [won't] talk to him about what's happened," he said. "I [will] talk to him about the future and about what we're going to try to accomplish and how we're going to get over it."