Simon Cowell and Mariah Carey
Ferdaus Shamim/WireImage; Jeff Vespa/WireImage
08/06/2011 at 07:10 AM EDT
Whatever Mariah Carey
wants, Mariah Carey gets.
The X Factor
judge Simon Cowell
, whose new talent competition debuts this fall, is willing to give into the demands of the pop diva, whom he says "wanted to be involved in the show from day one."
"You can't say no to Mariah," Cowell, 51, tells PEOPLE. "We have to find a role for her. I'm a huge fan of hers – not just as an artist, but as a person, and I love her to death."
But Cowell also says that having Carey on the show
poses a slight problem.
"We just can't shut Paula [Abdul]
and Nicole [Scherzinger]
up," he joked. "Add Mariah into the mix as well and you won't even have anyone singing."