"Did you get my last message about me being a bad father and Tim being a great dad?" the actor-filmmaker, 54, allegedly asks Grigorieva, 40, on the tape released Monday by RadarOnline.
"You didn't hear that one? Well, you should go [expletive] him … because I don't care," he says.
The Russian musician has been leaning on Dalton, 64, with whom she has a 12-year-old son, for support during her bitter custody battle with Gibson.
On the brief tape, which appears to be a voicemail message, an outraged Gibson also yells that "the game's over" before telling the mother of their 9-month-old daughter Lucia, "I'm so [expletive] sorry I had a child with you."