"There were some good times and some not so good times, but it's very hard to live your life out on the screen," the source says, referencing her divorce from Kelsey Grammer, which played out on the show's first season.
Grammer's decision comes at a time when there is great uncertainty regarding the upcoming season's cast.
"Adrienne Maloof is also considering leaving the show, but she has not made a definitive decision as of yet," the source tells PEOPLE.
But a source close to Maloof tells PEOPLE that she is not appearing on the reunion special as well – for legal reasons.
"She's painted into a legal corner," the source says. "Because of the judgement in her divorce proceedings she is limited in what she can say – and she will be unable to defend herself. She's very unhappy about how the show tried to paint her and made her into their evil person for simply trying to protect her kids and family's privacy." (TMZ.com first reported the news that Maloof was not appearing on the reunion episode.)
Adds the source: "She thinks [the show has] gotten silly and petty, and she's frankly ready to move on. ... Don't expect her back for season 4."
As for Grammer, however, she has no regrets about the show says the source close to her: "She is definitely a much stronger person now than when she started."