Simon Cowell: We Got Enough Baby Shower Gifts for '100 Kids'
12/13/2013 AT 11:30 AM EST
"I wasn't there. I wasn't invited, literally," the music producer, 54, said Thursday at an X Factor press conference in Los Angeles.
But he did have a peek at the stacks of presents afterwards. "There were a lot of them," he admitted. "There were a lot of clothes. It looks like I'm having 100 kids right now, there are so many."
Cowell and Silverman are expecting a boy in February. He'll be well-dressed, that's for sure – but will he follow his father into the music business?
The X Factor judge said he would approve of his son becoming a musician – on one condition. "As long as he signed a lifelong recording contract with me, I'd be all right with that. No options," Cowell joked.
The parents-to-be are planning a Caribbean getaway for Christmas. And a source has told PEOPLE that Cowell is likely to propose to Silverman now that her divorce from Andrew Silverman has been finalized.