But now Lovato can say that it's "really cool to hang out with" the pop superstar.
"What surprised me the most about Britney is that she's actually really quirky," Lovato, 20, told PEOPLE of Spears, 30, at a party for the season 2 finalists at the SLS Hotel in Beverly Hills on Monday.
"You kind of expect a legendary pop star [to] sometimes [be], like, a diva or something, but she's really not. She's a sweet, down-to-earth, Southern girl."
Lovato says The X Factor judges get along because they're united in a common goal – even if they're competing against each other to find the winner.
"At the end of the day, it's about the contestants," she explains. "It's not about us as judges. I think other shows maybe forget that the entire purpose of the competition is to find a superstar, not to get publicity for the wrong [reasons]."
Inside the party, Lovato, whose newly brunette hair was in its signature top-bun, smiled as she mingled with some of the show's female contestants, who seemed to enjoy the "Give Your Heart a Break" singer's company.
"I'm having a lot of fun," Lovato said. "It's such a blessing to be on this show. It came at a time in my life where I really needed something to look forward to in my career. It's been an amazing opportunity, and getting to sit next to Britney and Simon [Cowell] and L.A. [Reid] has been an honor. [And] Just being able to mentor girls that I believe in is so – I can't explain it. It feels so great."