Island Def Jam chairman Antonio "L.A." Reid will sit alongside Cowell on the panel, a source confirms to PEOPLE.
While he doesn't mention X Factor, Reid sent out an email to his staff on Thursday announcing, "I have decided to leave my position as chairman of the Island Def Jam Music Group. I have always thrived on growth and the next great challenge, and I look forward with much enthusiasm to what the future holds."
"We always wanted to work with L.A., whether it was now or in the future," Cowell recently told PEOPLE when asked who's in the running for the judging slots. "Because in terms of his credibility and what he’s achieved, he’s always somebody who we would love to work with ... He’s had so much success. And he’s also a good friend of mine."
The X Factor, which is set to premiere on Fox in the U.S. in the fall, is holding auditions in Miami, New York, Seattle, Los Angeles, Dallas and Chicago. There's a $5 million recording contract up for grabs for the winner of the show.
Go here to sign up.
In addition to Cowell and Reid, two more celebrity judges are expected to be named. While there's been a lot of speculation about who will join the show, Cowell told PEOPLE he's still meeting with candidates.
"We've met with Paula [Abdul]," he says. "What I've always loved about Paula is that she actually has got good taste in music."
Cowell also reveals that Jessica Simpson is a frontrunner for one of the spots: "She's very into the mentoring side."
Fox had no comment.
For more on Simon Cowell's thoughts on American Idol and The X Factor, pick up this week's issue of PEOPLE on newsstands Friday
On Newsstands Now
- Brad's Devotion: The Inside Story
- Oklahoma Tornado: Heroic Rescues
- Michael Douglas on Catherine's Health
Pick up your copy on newsstands
Click here for instant access to the Digital Magazine