Simon Cowell Is Having a Baby with Lauren Silverman

Wednesday July 31, 2013 11:15 AM EDT

It's true. Simon Cowell is finally going to be a dad!

The X Factor judge, 53, is expecting his first child with New York socialite Lauren Silverman, 36, multiple sources confirm to PEOPLE. The news has been hot gossip among friends recently because Silverman is married to Cowell's close friend, real estate mogul Andrew Silverman.

The Silvermans have filed for divorce, a source tells PEOPLE.

When reached by PEOPLE, an assistant to Cowell's U.K. rep says, "This is a very sensitive issue and all media enquiries are being dealt with by his U.S. attorney." No other calls to reps were returned.

So how does Cowell, who has shattered the dreams of many a kid on Idol and X Factor, feel about having kids? It depends when you ask him.

"God, no. I couldn't have children. If I had them here drawing on the walls I'd go nuts," he said in 2009. "You've got to be up at a certain time. Got to listen. When all you want to do is sit in a corner thinking."

But in an interview with AOL Music in March 2012, Cowell seemed more open to the idea.

"I do really, really like kids, because I can talk to them and listen to them," he said. "The problem has always been how much time you need to devote to bringing up kids and I've always dedicated all my life to work, and at 52, I've probably missed the opportunity."

Prior to meeting Silverman, Cowell was engaged to Idol makeup artist Mezhgan Hussainy, but the couple broke off their 18-month engagement in September 2011.

He also dated Carmen Electra from December 2012 to February 2013. "I'm now a single man," Cowell told PEOPLE after the split. "We spent time together and she is so much fun."