Demi Lovato and Simon Cowell
12/07/2012 at 03:00 PM EST
with Simon Cowell
and, after Thursday's elimination
of CeCe Frey, she became the first judge this season to lose all her contestants on The X Factor
So, should freshman judge Demi Lovato
be worried about going the way of season 1's female judges
, Paula Abdul
and Nicole Scherzinger?
"It's that time of year [when] you're always going to get asked that question," Cowell told PEOPLE after Thursday's show, "and I'm not going to answer it."
If Demi does leave the show it won't be due to any personal issues Simon may have with the "Give Your Heart a Break" singer. Despite their near-constant bickering on the panel, Simon admits he has a soft spot for her spitfire nature.
"She's fun," Simon, who still has two acts in the competition, said. "She really, really is a tough little thing. I know she plays the 'I've been bullied' [card] – trust me, this one gives as good as she gets."
Still, that doesn't mean Simon agrees with the harsh criticisms Demi has been dishing out to his groups, boy banders Emblem3 and girl group Fifth Harmony.
"It did get a bit bitchy and unpleasant," he said of their recent spats. "I've got to look after my artists. Luckily, they didn't listen to her."
Reporting by PATRICK GOMEZ