"They are really two different people," a source tells PEOPLE. "Problems got bigger in their marriage after a few years."
When they fell for each other on the set of Daredevil in 2003, Affleck, 42, was already an Oscar winner (for 1997's Good Will Hunting) and Garner, 43, was a rising TV star best known for her roles on Alias and Felicity.
A West Virginia-reared former band geek who often spoke of her affection for Martha Stewart and all things domestic, Garner has long had a warm maternal side.
"She's always loved kids," a childhood friend told PEOPLE in 2012. "We used to babysit as kids. And she was always the best babysitter."
The Boston-bred Affleck had a less conventional childhood, having worked as a kid actor on PBS shows. With the smash success of Good Will Hunting, his film career took off – and after marrying Garner, he channeled his professional ambition in even bigger ways, including directing.
One issue: the separations caused by Affleck's career, which frequently left Garner at home to care for the couple's three children: Violet, 9, Seraphina, 6, and Samuel, 3, while also continuing to act in movies such as Dallas Buyers Club and Men, Women & Children.
"Ben's frequent absences didn't work for this marriage," an entertainment source says.
A different source adds, "Ben is a good dad when he is around, but he has so many outside interests and personal issues to get beyond. It had become a bigger problem than either had imagined."
Another source close to the couple counters that Affleck "was a good husband and loves and cares for his wife and family. He was juggling that as well as his career — everyone knows that juggling a family and career is hard."