During an appearance on The Ellen DeGeneres Show, Jones said the hardest part of his job is listening to "the producers screaming in my ear," telling him to cut off the sometimes verbose judges, including Simon Cowell and Paula Abdul.
"I'm wearing an earpiece and there's about five people screaming, 'Shut Paula up. Shut her up now. Move on,' " Jones, 34, told Ellen. "I'm going to have hearing damage in this ear by the end of series."
"She just talks and talks and talks," he joked about Abdul. "Where is the off button on this woman?"
All teasing aside, Jones said he often feels guilty about having to keep the Fox show on its tight schedule while allowing all four judges to comment on every performance.
"I feel terrible because I feel on occasions I'm coming across as rude and abrupt and that's really not me," he added. "I cherish every word that comes out of Paula's ... mouth. You know, it's a two-hour ... show. We're in limited time. We need a conclusion, otherwise it will just kind of end and nothing will have happened."