Throughout the day "he was very quiet," a source tells PEOPLE. "He seems very upset but also very happy to be back at work."
While tattoo model Michelle McGee claimed earlier this week that she had a months-long affair with James, the daredevil TV personality called the "vast majority" of her allegations "untrue and unfounded" in a statement to PEOPLE.
"Everyone is just trying to be understanding and supportive," the source says, adding that business at West Coast Choppers has even picked up since the reports of James's alleged infidelity first surfaced on Wednesday."He is actually a really nice guy and a good friend."
On Monday, Bullock left the Southern California home she shares with James and subsequently canceled plans to appear at the London premiere of The Blind Side.
Reporting by Pernilla Cedenheim