The Russian-born singer, 40, alleged in an emergency restraining order filed Monday against Gibson, 54, that he struck her during an argument at his Malibu home – an allegation that a Gibson source says is distorted.
"Oksana was completely out of control that day, screaming, and shaking Lucia like a ragdoll," says the source. "Mel stepped in to stop her from injuring the baby."
The source further alleges Grigorieva then withheld 7-month-old Lucia from the actor for Father's Day last weekend despite a custody agreement allowing the visit, which resulted in a face-off in a Los Angeles court on Tuesday.
"He was as frustrated as any father would be when they're kept away from their child," says a Gibson friend. "He was with all his kids and family and the one person missing was Lucia. Oksana is not putting the interest of their daughter first."
Adds the first Gibson source: "The allegation that Oksana is in fear of Mel is ridiculous. From January to March, she traveled with Mel to London and Paris at her insistence. This is all an attempt to gain financial advantage over him."
Oksana: Custody Agreement UnfairA source close to Grigorieva wouldn't comment on the alleged physical altercation, but denied the drama was escalated by the singer withholding their daughter.
"That is absolutely not what caused all this," the Grigorieva source says. "There's other reasons in this case that brought it to where it is today. Oksana doesn't feel the custody agreement is fair."
Grigorieva voiced her frustration Friday to reporters outside her San Fernando Valley home. "He's playing dirty. The truth will come out," she told Radar Online. Later, a somber-looking Grigorieva told PEOPLE: "I'm doing okay. I really am. There's nothing more I can add."
Gibson, who is temporarily forbidden from contacting Grigorieva, was granted access to Lucia after his attorneys presented the judge with the custody agreement which the couple hammered out in May after they split.
The actor also filed his own restraining order against Grigorieva on Tuesday, but it only prevents her from speaking about certain sensitive aspects about the case.
A hearing on whether to extend the restraining orders is set for July 20.
Additional reporting by ELIZABETH LEONARD and JOHNNY DODD