Simon Cowell Previously Denied Adultery Rumors
08/01/2013 AT 01:00 PM EDT
"Any implication or statement that my client is engaged in adultery and has allegedly broken up the marriage of Ms. Silverman is not only false but reckless," Martin Singer told the New York Post at the time.
The high-profile Hollywood attorney also branded as "false and defamatory" any suggestion that the X-Factor host's relationship with Silverman "has become more serious such that she is now getting a divorce so that Simon and she can be together."
Silverman, 36, is now pregnant with Cowell's child and the TV host is reportedly named in divorce papers filed July 15 by her husband, real estate magnate Andrew Silverman, alleging adultery.
Neither Cowell nor any of his reps, including Singer, have commented since the pregnancy news broke.
A source tells PEOPLE that Lauren Silverman was "definitely unhappy" in her marriage at the time she met Cowell, 53, and that their relationship wasn't the only factor in her split with her husband.
"It was probably not the best relationship when she first met Simon," the source says of the Silvermans' marriage. "It wasn't like they were blissfully in love at the time."