12/13/2012 AT 07:10 PM EST
The latest season of The X Factor
is down to the wire
– but L.A. Reid is already looking ahead to next season. And he's not planning to come back.
"I have decided that I will not return to The X Factor
next year," Reid, who is chairman and CEO of Epic Records, tells Access Hollywood
. "I have a company to run that I've kind of neglected."
Of his decision to leave the singing competition, he says, "It saddens me a little bit, but only a little bit."
One of the things he's sure to miss is spending so much time with his pal and fellow judge Simon Cowell
. "I love Simon. We have a great relationship," says Reid. "We have fun together, we talk, we laugh, we do bad things, we're friends."
Reid says being on the show, which also currently stars judges Britney Spears
and Demi Lovato
, was "a nice break" from his day job, "but now I've got to go back to work."
Reid has been a judge on the show for two seasons with Cowell. Nicole Scherzinger and Paula Abdul
lasted only one season as judges before leaving.