Affleck, 42, has been spending time on serene Hampton Island, an ultra-private waterfront community about 35 miles south of Savannah where he first purchased property in 2003.
A source tells PEOPLE that this past weekend, the Gone Girl star hosted a party for nearly three-dozen guests on the island, catered by The Smokin Pig BBQ Joint in nearby Richmond Hill.
"They prepared barbecue and all the fixings for 35 people," said the source.
The conservation-minded area has been a beloved retreat for Affleck for more than a decade – he previously found sanctuary there following his split from Jennifer Lopez in January 2004.
"Mostly when I want to go and be at peace by myself, you know, I live down this long dirt road, and people are very respectful of people's privacy down there and [when I want to] be my calmest and happiest, that's where I go," he said on Larry King Live in March 2004.
Affleck has been in Georgia while Garner continues to film Miracles from Heaven in Atlanta. Initially spending time on her rental property with their three children, he has not been spotted on the property since news broke last week of the alleged affair.
A source close to Affleck contends there was "no romantic relationship" between the actor and nanny Christine Ouzounian, although sources close to Garner told PEOPLE the actress learned the pair were intimate.
"To Jen, this is the ultimate betrayal," one friend tells PEOPLE.
Meanwhile, Affleck has been leaning on his mother for support.
He also will continue spending time in Georgia when he begins filming his latest project, Live by Night, in November.
In February, he was spotted scouting locations for the film in Brunswick, Georgia, not far from Hampton Island.
"I am looking between Savannah and the end of the Georgia coast," Affleck said, according to The Brunswick News. "This area has always been so hospitable to me."
Reporting by POLLY POWERS STRAMM