Last year's Idol runner-up, Jessica Sanchez, told PEOPLE "I would pee my pants" if she had to sing in front of Carey, but other hopefuls went for it with gusto.
"There was this one time a girl sang one of my songs [and] I thought she was really good," Carey said during a panel last week to promote the Fox show's upcoming 12th season, which premieres Wednesday. "I thought, 'This girl has a chance.' I was proud that I had written a song that she was inspired to sing."
But the Grammy winner admits to cringing during some lackluster performances of her hits.
"Sometimes I'm, like, 'Is this for laughs? Are we doing this for fun?' " explains Carey.
Aside from contestants who could nail one of her own songs, Carey was intrigued by contestants who wrote their own material.
"To me, that adds to them as an artist. Every time someone would [sing a good original song] I was really impressed because that is my thing," she said. "I love to write songs."
Fellow new judge Keith Urban commented that it was more risky to sing Carey's material than original music – as long as the music was good.
"There was a girl in Hollywood week, she's 15 and she played the piano and sang [an original song]. I told her, 'If you release that song tomorrow, people will buy it by the boatloads,' " said the country singer, who described the new judging panel as a "dysfunctional family."
The final new judge, Nicki Minaj, had simpler advice for auditioning hopefuls, "Flash me!"
Joking aside, Minaj said she was looking for "an all around entertainer, someone who is going to captivate people with or without music."
Not that there wasn't a little flashing during the audition rounds. "There was one boy who came in and pulled off his shirt trying to be sexy," Minaj recalled. "And I signed a [girl's] boob!"