TV Watch

The X Factor

Simon Cowell: 'We Have to Find a Role' for Mariah Carey on The X Factor

By Lorenzo Benet and Dahvi Shira

08/07/2011 at 06:30 AM EDT

Whatever Mariah Carey wants, Mariah Carey gets.

The X Factor judge Simon Cowell, whose new talent competition debuts this fall, is willing to give into the demands of the pop diva, whom he says "wanted to be involved in the show from day one."

"You can't say no to Mariah," Cowell, 51, tells PEOPLE. "We have to find a role for her. I'm a huge fan of hers – not just as an artist, but as a person, and I love her to death."

RELATED: Simon Cowell Has 'Horrible Nightmare' He's Nice on The X Factor

But Cowell also says that having Carey on the show poses a slight problem.

"We just can't shut Paula [Abdul] and Nicole [Scherzinger] up," he joked. "Add Mariah into the mix as well and you won't even have anyone singing."