The Gossip Girl star's daughter Helena, 6, and son Hermes, 8, will leave the United States on Thursday to return to the home they share with their father, German businessman Daniel Giersch.
The kids spent their summer in New York with Rutherford, 46, arriving July 3, but have lived with Giersch, 41, since 2012, when a California judge gave the estranged parents joint custody yet ordered Helena and Hermes to live with their dad temporarily because his U.S. visa had been revoked. (The State Department has said he never reapplied for a visa.)
She's tried to repeal the ruling ever since. California and New York family courts declined to hear her case this month, claiming they no longer have jurisdiction.
"What Daniel I'm sure would argue is that it's in the best interest of the kids for them to continue to remain where they've lived for the last three years, where their friends are, where they go to school and where they have family and all of these things," Robert Wallack, who represented Rutherford in federal court last year but is no longer on her legal team, tells PEOPLE.
"Kelly obviously disagrees, and she would argue that it's in the kids' best interest for them to be with their mother, which is what the California order originally intended, for them to live in New York because they're American citizens, for them to go to school here and be with their friends here and all of that," he says. "So then you get into a fight over what's in the kids' best interests. The question now is where is that fight going to be held, and it seems like it's going to be in Monaco, right now. I think it's tough because who knows what's going to happen there."
Rutherford has been focusing on her precious time with her family, taking the children to museums, art classes and events from Manhattan to the Hamptons.
The actress brought both kids to the Ovarian Cancer Research Fund's Super Saturday fundraiser – where they mingled with Hilaria Baldwin and her older daughter with Alec Baldwin, Carmen, almost 2 – and Helena accompanied her mom for some girl time at the Blushington New York City Grand Opening Party.
"I don't know how it happened so fast, they're like teenagers," she told PEOPLE in April. "'m like, 'Wait a minute, you guys.' It's incredible how articulate they are and kind and loving. They're just amazing people, and I'm really proud to be their mom, and I'm proud of how they've handled all of this. I tell them that."
"I think she's a great mother. I think she is entirely dedicated to her kids and to her children's happiness, and I think it's devastating for her to not have them living here with her and to visit with them and then have to say goodbye to them," Wallack says. "But I think that that's also what's fueling the drive to get them back here, is her love for them and her dedication as a mom."
Through his attorney Fahi Takesh Hallin, Giersch has declined to comment on the ongoing custody battle.
"Daniel will continue to promote Kelly's relationship with the children," Hallin said in a statement July 23. "He believes that the children deserve to love both parents and has never nor does he intend to ever participate in any negative press directed at Kelly. As always, Daniel will continue to guard the privacy of the children, in their best interests and for their safety."