Berry became alarmed, say the sources, when the couple's 2½-year-old daughter Nahla became hysterical after Aubry came to pick her up for a visit a few days before the holiday.
"She was crying and kicking and screaming," claims one source. "She was hanging onto her mother's legs so she didn't have to go."
According to the source, when Berry tried to calm down her daughter, Aubry grew impatient. "He got Nahla and forced her into the car," adds the source, who says that Berry was extremely upset by the incident and that things between the exes – who had at one point been on friendly terms – immediately disintegrated.
Other sources close to Berry, 44, say a similar episode played out several subsequent times, including one incident in which Nahla appeared "traumatized," according to a Berry source, after a visit with her father. They claim such episodes led Berry to fear for Nahla's well-being.
A rep for Berry had no comment on the claims.
Public FightIn subsequent days, Aubry, 35, has rebutted his ex's claims, saying that he was "disappointed in Halle's decision to falsely malign him publicly."
Adds a source close to Aubry: "Those who know Gabriel well fortunately don't believe a thing about what is being said."
The drama appears far from over. On Wednesday, Berry – who earlier in the day was spotted with her boyfriend Olivier Martinez and Nahla – paid a visit to her family law attorney Neal Raymond Hersh in Beverly Hills.
"This is a very difficult time for Gabriel," says the Aubry source. "He doesn't need this right now."
For much more on Halle and Gabriel's custody battle, pick up this week's issue of PEOPLE, on newsstands Friday
Reporting by Ken Lee, Elizabeth Leonard, Aili Nahas and Oliver Jones