From PEOPLE Magazine Click to enlarge
In the month since he fired the first public shots in a nasty legal battle with three former employees, Rob Lowe has gotten on with business as usual, filming This Side of the Truth with Jennifer Garner in Boston and shooting the third season of Brothers and Sisters in L.A. But privately the actor, 44, has been deeply affected by the brewing storm over sexual-harassment claims he vehemently denies. "This thing has consumed his life," says a source close to the Lowe family. "He's holding up. It's [wife] Sheryl and the kids who are taking it the hardest. She's really upset, and the kids are old enough to read about it all."

The latest lawsuit in the case was filed April 30 by Laura Boyce, the second former nanny to accuse the Lowes of sexual harassment. Boyce claims Sheryl, 46, harassed her during the seven months in 2007 that Boyce cared for the Lowes' sons Matthew, now 14, and John Owen, 12. Among Boyce's claims (see box, next page): Sheryl walked around the house naked and made crude and racially derogatory comments about Boyce's sex life. At a May 1 press conference, a tearful Boyce, 37, said she left her job "because I could no longer put up with Mrs. Lowe's conduct towards me."

Responds the Lowes' attorney Larry Stein: "The claims against Sheryl are not only false but come seven months after she left the job and Gloria Allred [who also represents nanny Jessica Gibson] became her lawyer." Boyce's suit does not allege any sexual misconduct by Rob, but Allred says that both Lowes are responsible for the harassment since both employed Boyce. Boyce "is devastated; those weren't crocodile tears," says a relative, adding that Boyce has lost 20 lbs. from stress. "But she's trying to be strong. She's like, 'I'm not gonna let them knock me down.'"

Gibson, 24, filed a suit April 14 claiming Rob Lowe groped her and exposed himself. On April 7 the Lowes filed preemptive suits against both ex-nannies, along with former chef Peter Clements (who has not yet filed a legal response), accusing them of conspiring to spread lies about the Lowes. The Lowes say Gibson threatened to sue if they didn't pay her $1.5 million, Gibson has denied any intent to blackmail them, and Allred calls the suit against Boyce an attempt "to intimidate her from providing information" that could support Gibson's suit.

For Lyndhurst, N.J., native Boyce, the spotlight is an uncomfortable place to be, says her relative. "She's a regular, normal, down-to-earth person." After stints in the fashion industry and nightclub promotions, Boyce dealt with financial problems—including a bankruptcy—and found "her calling," the source says, as a nanny. That included babysitting for TV personality Shanna Moakler. Boyce appeared on an episode of the MTV reality show Meet the Barkers when Moakler was married to drummer Travis Barker. But she's not a fame-seeker, says her relative: "She was the one behind closed doors taking care of the kids."

Moakler, who now employs Boyce, tells PEOPLE, "I've been friends with Laura for the last 10 years—before she was my nanny—and I completely stand by her."

How strong is Boyce's case against the Lowes? Much depends on whether Allred can prove that Sheryl made the comments and that they were unwelcome, severe and pervasive, says Jon Miller, a Los Angeles attorney specializing in sexual-harassment cases (but not involved with this one). "If Boyce willingly participated in such conversations, then it's not illegal harassment."

For now, both sides are standing their ground. Boyce's relative calls her "an honest person, going up against people with a lot of money and fame." The Lowes, meanwhile, plan to defend themselves at any cost. Says their friend: "They've worked so hard to make a good home for themselves and a happy family."