Just don't expect her to be fazed by it.
"We got more comfortable in telling our opinions without being afraid of the booing," Demi Lovato told PEOPLE Tuesday of herself and fellow judge Spears. "When Britney and I first started on this show, we were looking at each other like, "What do we do? We're so nervous!' "
A confident Spears added of prepping for the live shows later in the season, "There's no way of prepping, really. You just get ready and you go out there and you do it."
Simon Cowell agrees that the longtime pop star is ready for her new role. "She's fearless," he told reporters of Spears, 30. "When she says 'no' or disagrees, she couldn't care less."
But that wasn't always the case. "The first day, the second she said 'no,' everyone booed her and that happened a lot and I think she found it very intimidating," says Cowell, 52.
But now, Cowell says of Spears, "I think she's really enjoying doing something other than singing. I think she likes being part of a team."
And while Spears has gotten lots of attention for being a tough judge on the reality show, she thinks there's someone else at the judges' table who's even tougher. "I think Demi [is], I just do," says Spears. "She's a little younger than me so she's a spitfire."
"That wouldn't be the first time I've been called that," joked 20-year-old Lovato.
"I don't have any rivalry with [Britney], friendly or not," added Lovato. "I have a rivalry with Simon for sure. And you know what? It's not so friendly."
The X Factor premieres Wednesday at 8 p.m. on Fox.