Simon Cowell: 'We Have to Find a Role' for Mariah Carey on The X Factor

By Lorenzo Benet and Dahvi Shira

Sunday August 07, 2011 06:30 AM EDT

Simon Cowell: 'We Have to Find a Role' for Mariah Carey on The X Factor | Mariah Carey, Simon Cowell

Simon Cowell and Mariah Carey

Ferdaus Shamim/WireImage; Jeff Vespa/WireImage

Whatever Mariah Carey wants, Mariah Carey gets.

The X Factor judge Simon Cowell, whose new talent competition debuts this fall, is willing to give into the demands of the pop diva, whom he says "wanted to be involved in the show from day one."

"You can't say no to Mariah," Cowell, 51, tells PEOPLE. "We have to find a role for her. I'm a huge fan of hers – not just as an artist, but as a person, and I love her to death."

But Cowell also says that having Carey on the show poses a slight problem.

"We just can't shut Paula [Abdul] and Nicole [Scherzinger] up," he joked. "Add Mariah into the mix as well and you won't even have anyone singing."